LT-SNN: Self-Adaptive Spiking Neural Network for Event-based Classification and Object Detection
Authors: Ahmed Hasssan
Jian Meng
Jae-sun Seo
Venue: Submitted to ICLR 2023 (2023)
Keywords: Spiking neural networks
efficient neuromorphic computing
spatial-temporal adjustment
separate surrogate gradient path
output regularization and self-adaptive and learnable potential threshold.

Abstract: Spiking neural networks (SNNs) have received increasing attention due to its high biological plausibility and energy efficiency. The binary spike-based information propagation enables efficient sparse computation with event-based computer vision applications. Prior works investigated direct SNN training algorithm to overcome the non-differentiability of spike generation. However, most of the existing works employ a fixed threshold value for the membrane potential throughout the entire training process, which limits the dynamics of SNNs towards further optimizing the performance. The adaptiveness in the membrane potential threshold and the mismatched mechanism between SNN and biological nervous system remain under-explored in prior works. In this work, we propose LT-SNN, a novel SNN training algorithm with self-adaptive learnable potential threshold to improve SNN performance. LT-SNN optimizes the layer-wise threshold value throughout SNN training, imitating the self-adaptiveness of the biological nervous system. To stabilize the SNN training even further, we propose separate surrogate gradient path (SGP), a simple-yet-effective method that enables the smooth learning process of SNN training. We validate the proposed LT-SNN algorithm on multiple event-based datasets, including both image classification and object detection tasks. Equipped with high adaptiveness that fully captures the dynamics of SNNs, LT-SNN achieves state-of-the-art performance with compact models. The proposed LT-SNN based classification network surpasses SoTA methods where we achieved 2.71% higher accuracy together with 10.48× smaller model size. Additionally, our LT-SNN-YOLOv2 object detection model demonstrates 0.11 mAP improvement compared to the SoTA SNN-based object detection.

Under review as a conference paper at ICLR 2023
LT-SNN: SELF-ADAPTIVE SPIKING NEURAL NETWORK FOR EVENT-BASED CLASSIFICATION AND OBJECT DETECTION
Anonymous authors Paper under double-biee blind review
ABSTRACT
Spiking neural networks (SNNs) have received increasing attention due to its high biological plausibility and energy efficiency. The binary spike-based information propagation enables efficient sparse computation with event-based computer vi- sion applications. Prior works investigated direct SNN training algorithm to over- come the non-differente oon-ifferentiabbiity of spike generation. However, most of the exist- ing works employ a fixed threshold value for the membrane potential throughout the entire training process, which limits the dynamics of SNNs towards further optimizing the performance. The adaptiveness in the membrane potential thresh- old and the mismatched mechanism between SNN and biological nervous sys- tem remain under-explored in prior works. In this work, we propose LT-SNN, novel SNN training algorithm with self-adaptive learnable potential threshold to improve SNN performance. LT-SNN optimizes the layer-wise threshold value throughout SNN training, imitating the self-ade addttidapti of the biological nervous system. To stabilize the SNN training even further, we propose separate surrogate gradient path (SGP), a simple-yet-effective method that enables the smooth learn- ing process of SNN training. We validate the proposed LT-SNN algorithm on multiple event-based datasets, including both image classification and object de- tection tasks. Equipped with high adaptiveness that fully captures the dynamics of SNNs, LT-SNN achieves state-of-te offtth--rrt performance with compact models. The proposed LT-SNN based classification network surpasses SoTA methods where we achieved 2.71% % higher accuracy together with 10.48 x smaller model size. Ad- ditionally, our LT-SNN-YOLOV' object detection model demonstrates 0.1 0. mAP improvement compared to the SoTA SNN- object detection.
INTRODUCTION
In the biological nervous system, cortical neurons process information by encoding spatial-temporal inputs into action potentials for spike generation. Inspired by that, spiking neural networks (SNNs) accumulate the membrane potential by extracting information from the input features each time step, and the resultant binary spikes (0 and I) provides a sparse and succinct information repre- sentation. Such spatial-temporal computation promotes SNN an attractive AI solution with both biological plausibility and energy efficiency in comparison to the conventional artificial neural net- works (ANNs) (He et al., 2016). Furthermore. layer-wise wise processing with binary spikes elevates the computation efficiency, which benefits the energy-constrained applications such as edge computing.
Under the context of energy-efficien AI applications, the event-based camera or dynamic vision sensors (DVS) have emerged as attractive and feasible solution for computer vision applications. Compared to the conventional frame-based camera, event cameras independently capture the abso- lute illumination changes of pixels, resulting in the asynchronous binary stream of events (Gallego et al., 2020). The captured event characterized by binary pixels and temporal resolutions, lead- ing to highly sparse and energy-efficiert visual representations. Such binarized pratial-temporal information naturally fits the computation mechanism of SNNs, bridging the gap between computer vision and neuromorphic computing.

---

Under review as a conference paper at ICLR 2023
82
LT-SNN-VGG9 (4bit)
DVS- CIFAR10 Accuracy (%)
80
LT-SNN-VGG7
78
LT-SNN-Mobilen (Ligh)
76
DSR-VGG11 (CVPR, 22)
TET-VGGSNN (ICLR, 22
74
SEW-7B-Net (NeuriPS. 21)
Learnable Threshold (Limited)
Fixed Threshoid
72
Fixed Threshold
70
68
1dBN ReeN-RRaNet--1 19 (AAAI, 21)
66
Fixed Threshold
64
62
ASF-BR BP (AAAI, 20)
60
Fixed Threshold
30
50
Model Size (MB)
classification accuracy of different SNN training methods. The proposed LT-SNN training algorithm achieves the state-of-art accuracy with the compact VGG models.
Figure 1:
One of the major bottlenecks of SNN training is the non-differentiability of the spike generation. The infinite gradient of the step function impedes the gradient propagation during backward pass in training. Early research works relied on the ANN-to-SNN conversion (Diehl al., 2015; Han et al., 2020) to obtain trained SNN models. However, additional training is required and sufficiently high accuracy was not achievable. To that end, various direct training methods have been proposed to obtain a SNN model with one-time training. Empowered by various surrogate gradient (SG) functions (Lee et al., 2016; Wu et al., 2019; Deng et al., 2021), the inaccessible gradient of the spike function is approximated and propagated during learning. However, the inaceurate approximation and heuristic SG selection hurts the training stability with deep models (e.g., ResNet (He et al., 2016)), which further motivated the temporal normalization method (Zheng et al., 2021) and output regularization techniques (Guo et al., 2022: Deng et al., 2021) to smooth the loss.
As the major inspiration of deep learning, the intricate nervous system achieves remarkable per- formance with high degree of dynamics. Previous neuroscience works observed the location- dependent potential threshold (Kole & Stuart, 2008) in nervous systems, implying the adaptive fir- ing procedure within the mechanism of spike generation. Inspired by this, some recent works on SNN training introduced the learning dynamics into the training process, albeit limited degree. (Fang et al., 202 lb) optimized the membrane time constant throughout training, with the require- ments of large-sized models. DSR (Meng et al., 2022) proposed the threshold-associited spikes with learnable potential threshold. However, the deterministic ratio between firing range and po- tential threhsold of DSR limits the adaptiveness of SNN learning. (Sun et al., 2022) removes such constraints by directly passing the gradient of potential threshold through the SG function. Neverthe- less, the instability of the straight-through surrogate gradient still results in sub-optimal performance compared to state-of-the-art (SoTA) SNN training with fixed threshold (Deng al., 2021). Although (Deng et al., 2021) achieved the SoTA performance among prior works, the fixed threshold made the membrane potential often overshoot, limiting the dynamics of SNNs. The limitations of all prior works motivate us to investigate the following question: How can we optimize the potential threshold of SNNs with high stability and superior accuracy?
To answer this question, we propose LT-SNN, a novel self- SNN training algorithm with Learnable Threshold. Starting the training from scratch, LT-SNN fully optimizes the potential threshold without introducing any additional scaling or firing constraints. To achieve highly-stable training, we propose simple-yet-effective technique, namely Separate Gradient Path (SGP). Com- pared to prior works, the proposed LT-SNN algorithm fully unleashes the advantage of wise adaptive potential threshold, leading to superior performance compared to all prior SNN algorithms. We validate LT-SNN multiple event-based computer vision datasets with various model architec- tures, LT-SNN achieves the new state-of the-art performance with weight or quantized models, shown in Figure! 1.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
ANN-to-SNN conversion Given the non-differentiabiliy of SNN training, early research works converted high-performance non-spiking ANN model into spiking version (Diehl et al., 2015; Rueckauer et al., 2016). The major drawbacks of the conversion-basee latency (high number of time-steps) and the additional efforts for the overall training. Several meth- ods have been proposed to improve the lateney of the converted model (Han et al., 2020), but still did not reach the high accuracy of direct SNN training methods.
method is high computation
Direct SNN training The expensive training effort and high latency of the conversion-based schemes promotes the direct SNN training to be attractive solution. BNTT (Neftci et al., 2019) treats the spatial-temporal computation of SNN as a special version a recurrent neural net- work (RNN). The gradient of the spiking process is estimated by surrogate gradient (SG) functions in backpropagation. Driven by accuracy and different model architectures, various SG functions have been proposed, including but not limited to rectangle function (Wu et al., 2019), aretangent (Fang et al., 2021b;a) or triangle functions (Deng et al., 2021). However, the difference between the ap- proximated gradient and the exact gradient largely limits the training stability of SNN, especially for the large-sized models. Motivated by that, tdBN (Zheng et al., 2021) introduces the batch-tempone normalization for deep SNN training, and SEW-ResNet (Fang al., 2021a) directly passes the gra- dient via designed residual architecture. In addition to the architecture design, various regularization techniques were presented stabilize SNN training by rectification of the membrane potential dis- tribution (Guo et al., 2022) and backpropagation with spatio- tentiol adjustment (Shen et al., 2022).
Biological dynamics in SNN The biology- -inppiee SNN training methods are also highlighted in recent research works. (Fang et al., 2021b) introduces the learnable time constant for direct SNN training, but the employed large-sized SNN model and extensive training efforts are computation- ally expensive. Recently proposed DSR (Meng et al., 2022) optimizes the potential threshold during training by multiplying the binary output spike with the threshold value, where the relationship between the firing range and threshold value is constrained by deterministic ratio. However, op- timizing the potential threshold value with such additional constraints limits the learnability of the SNN model. Furthermore, the chreshold-dependent binary spikes require layer-wise scaling opera- tion, which deteriorates the hardware compatibility with expansive high-precision multiplication.
3 BASICS OF SPIKING NEURAL NETWORKS
Inspired by the biological nervous system, spiking neural networks process the peatial-temporal information throughout layers. Mathematically, the widely-adopted Leaky Integrate-and -ire (LIF) model characterizes the membrane potential accumulation with the following ordinary differential equation (ODE):
du(t) (u((t) ureset) T(I), dt
T
where utf) represents the membrane potential at time t, Vreset is the reset potential after the spike operation, and Ie is the synapse current time and T is the time constant. The ODE in Eq. can be resolved as:
M++1 (1 dl
dl I,
where / is the pre-synaptic "current", characterized by the layer-wise output features of SNN model. Given the iteratively accumulated membrane potential, the binary spikes are generated by the step function:
S. = 0(, ViA) = {
if u4 2 Vin otherwise
(3)
Where represents the Heaviside step function, and Voa represents the membrane potential threshold for spiking neurons. In the forward pass SNN, the binary post-synaptic spikes are generated when the membrane potential exceeds the potential threshold Vih-
3

---

Under review as a conference paper at ICLR 2023
80
1.2
DVS CiFARRRO (N)
Average Potentaal Threshold
Average Potentaa Threshold
STSG Failed Training X
SGP Stabilized Training (Proposed)
1.0-
+
0.8
DVS CIFAR11
8.0
0.6
Learnable Accuracy
Leamable Thesshold Accurasy
0.8
20
d0
80
100
60
00
100
120
140
160
180
200
Training Epocha
Training
(b)
Figure 2: Validation accuracy and potential threshold of direct training SNN on DVS-CIFARIO dataset with (a) STSG and (b) the proposed separate gradient path (SGP) for LT-SNN.
As demonstrated by spatial-temporal backpropagation (STBP) (Wu et al., 2019), the backward prop- agation of SNN is characterized as:
aL aL as, du, 01,
(4)
OW as, du, Òl, OW
As mentioned earlier, the non-differentiable spike function 0 impedes the gradient computation of aS,/Ou, The directly-trainee SNN incorporates the surrogate gradient function approximate the intangible Dirac function. In this work, we choose triangle function for gradient approximation:
asc 0' (we- V max(0, Ou
|
VeA
(5)
4 CHALLENGES OF LEARNING THE THRESHOLD OF MEMBRANE POTENTIAL
As deseribed Eq. 5, the non-differentiability between the output spike S and the membrane po- tential u can be alleviated by the surrogate gradient approximation. Mathematically. the gradient of the potential threshold can be approximated with a similar gradient surrogation:
aL aV.. as, oV. as,
aL 0L as;
8' d'(ue -(n)
(6)
The straight-through approximation Eq. 6 is built upon the following hypothesis:
Hypothesis 1: The surrogate gradient of OS. and as, is transferrable, and the identical gradient du, aV., surrogation is suitable for separate loss landscape with respect u, and Va..
In this work, we falsify Hypothesis by empirically demonstrating the learning instability caused by the straight-through surrogate gradient (STSG). Figure 2(a) depicts the validation accuracy and the layer-wise averaged potential threshold of the VGG-9 model which is trained from scratch with STSG. The drastic change in potential threshold and exploding validation accuracy represents the incompatibility of the STSG learning scheme. In summary, we have the following observation:
The gradient approximation with respect membrane potential and potential threshold is not interchangeable.
Observation 1:
Different from the STSG scheme, the recent DSR (Meng et al., 2022) scheme computes the gradient of potential threshold with the following threshoid-assoccitee firing procedure:
S, = Vah A(u aVv.)
Where the fixed parameter E [o. 1] controls the threshold with respect to the firing range [0, Vin)- In other words, regardless the threshold value, the membrane potential has to be a deterministic portion of the total firing range. However, such constraints in the training process largely limits the learnability unleash the optimization constraints of the potential threshold by setting that, keeping the threshold and spike as identical values is commonly adopted in prior works (Deng al., 2021). As shown in Table 1, the freely-learned potential threshold (a 1.0) exhibits large
SNN. Based on the open-sourced implementation of DSR (Meng et al., 2022),
1.0. Please note
4

---

Under review as a conference paper at ICLR 2023
Table 1: The performance of DSR (Meng et al., 2022) is largely impacted by the value of a. The
proposed LT-SNN outperforms DSR with 2.45% higher accuracy on DVS- DVS-CIFARIO dataset.
Method
Model
Accuracy (%)
a
True Binary Spike
DSR (Meng et al., 2022)
VGG-11
77.19
0.3
DSR (Meng et al., 2022)
VGG-11
75.29
1.0
X
This work
VGG-9
80.04
accuracy degradation compared to the constrained leanring (a = 0.3), which implies the sub-optimal performance of the DSR (Meng et al., 2022) when the potential threshold is unconstrained.
On the other hand, associating the threshold value with the neuron output disables the true binary spikes (0 and 1) of SNN. Specifically, the output generated by Eq. 7 becomes (0, Via) instead of pure binary value (0, 1). In the context of hardware computation, layer-wise varied spikes equivalent to the layer-wise scaling on top of the true binary output, which often requires high- precision multipliers (Jacob et al., 2018) to maintain the accuracy. Unlike the conventional ANN with one-time scaling operation, the temporal computation of SNN requires step-wise high-precision multiplication, which could lead to increased energy and hardware resource consumption. The instability of STSG and the imperfections of DSR (Meng et al., 2022) motivate us to investigate the following question:
How can we maximize the performance of SNN with the freely-optimized potential threshold, while maintaining the training stability and hardware compatibility?
Question 1:
PROPOSED METHOD
To answer the question above, we propose L7-SNN, a novel SNN training algorithm that successfully resolves the contradiction between threshold optimization, training stability, and hardware compati- bility. LT-SNN optimizes the layer-wise potential threshold during training, maximizing the training performance of SNNs without introducing any learning constraints. In the meantime, LT-SNN em- braces the advantages of the adaptive potential threshold while maintaining the true binary spikes, bridging the research gap between biological inspiration and practical Al applications.
5.1 SEPARATE GRADIENT PATH
As summarized in Observation 1, the incompatibility of STSG implies the distinct loss landscape with respect to the gradient descent of the membrane potential and potential threshold. Motivated by that, we propose Separate Gradient Path (SGP), which treats the gradient computation of we and Via with dedicated gradient approximations. Specifically, SGP trains SNNs by introducing the Gradient Penalty Window (GPW), a simple-yet-effective method that optimizes the potential threshold without losing training stability. On top of the gradient approximation in Eq. 5, GPW is characterized as a non-linear function (-),, which reshapes the surrogate gradient of the layer-wise potential threshold VnA Mathematically, the GPW-aided separate gradient path is characterized as:
as. 0' d'(u - Ven) = max(0, [ue- Vva
Ou:
as. =
0l(1 Vinlo - ViA) = max (0, 1- - l44 Villo V(x)
(9)
@Vi In this work, we choose the Sigmoid function as the gradient penalty window for potential threshold:
ou
Vin) -
(10)
The choice of Sigmoid function is empirical as it produces the best results among different surrogate functions. Performance comparison of different surrogate functions with LT-SNN is presented in Appendix C, Table 11. For gradient computation of Via, we accumulate the gradient computed in Eq. avoid the dimensionality mismatch:
OL
ØL aS. aL
Σ (N.C.H,W)
(1 2 Vin. x O'(u. Vin)a(u -Ve)) (11)
V
as, VVm
as
5

---

Under review as a conference paper at ICLR 2023
Data Preprocessing
Input Spikes
LT-SNN Forward
LT-SNN Backward Pass
0V9 Exinttyll
Danala
Laspurt
Layertr1
d0g
US
Lapert
Layer
Leurrai
Wy
Learrablle
OV
BOL
FC
125 Time-darpeed Dats
0.5
Ewiremmet
Heavy mide step
Figure 3: Implementation flow of LT-SNN.
Table 2: Training results of VGG-9 model on DVS-CIFARI0 dataset with different SG schemes.
Method
Learnable Thre.
SG Func.
Top-1 I Accuracy (%)
Fixed threshold Train.
X
Triangle
78.02
Penalty for all
V GPW for all
76.40
This work
V
SGP
80.04
Since the unfired neurons have contribution to the final loss, the indicator function 1 1fu, 2 Vaa only keeps the gradient with respect to the active neurons in the forward pass.
We empirically prove the superiority of the proposed SGP algorithm based on different surrogate gradient scenarios. Table summarizes the performance of the directly trained VGG9-SNN on the DVS-CIFARIU 10 dataset with differrntt gradient computation paths. On top of that, compare the performance of SGP with non-unified learning rate based LT-SNN training. Table Appendix demonstrates the superiority of SGP over non-unified learning rate for LT-SNN. As shown in Ta- ble 2, penalizing all the surrogate gradient with GPW leads to sub-optimal performance. On the contrary, the proposed SGP algorithm achieves the best performance with distinct gradient compu- tations. SGP exploits the optimal layer-wise potential threshold with the stabilized training process and fast convergence. Empowered by the proposed SGP scheme, LT-SNN achieves >80% valida- tion accuracy within 50 training epochs, as depicted in Figure 2(b). Compared to the SoTA SNN training with fixed threshold (Deng al., 2021), SGP embraces the advantage of the adaptive poten- tial threshold learning and achieves superior accuracy. Furthermore, SGP preserves the true binary spikes (0 and 1) in the resultant model, maximizing the hardware compatibility without introduc- ing any high- precision scaling for spike generation (Meng et al., 2022). The complete flow of the proposed SGP with GPW and output regularization illustrated in Figure
Convi
Conv5
Surrogate Gradient 15,ddu,
1.0
Conv2
Conv6
Potential Threahold
0.8
Conv4
Conv8
0.8
0.6
0.4
0.6
0.2
0.0
0.5
2
1.0
2.0
Layers
Mambrane Potential
(a)
(b)
Figure 4: (a) Layer-wise adaptive potential threshold V. leads to (b) layer- adaptive surrogate gradient gradient of the membrane potential u4.

---

Under review as a conference paper at ICLR 2023
5.2 SELF-ADAPTIVE SNN LEARNING
The proposed SGP algorithm improves the performance of SNN with the unconstrained optimiza- tion of the potential threshold. However, the rationale behind the adaptiveness -induced optimal performance remains unclear, which motivates us to investigate the following question:
Question 2: Why the adaptive potential threshold optimizes the performance of SNN model?
As shown in Eq. 5, the gradient with respect to the membrane potential is approximated based on the surrogate function 0', which is characterized by the non-linearity of the function and the shifting induced by the threshold Vu.- With the proposed LT-SNN scheme, the adaptive potential threshold leads to the distinct surrogate gradient for membrane potential u, of each layer, as shown in Figure 4. Compared to SNN training with fixed potential threshold, L.T-NNN adjusts the layer-wise gradient information with the adaptive potential threshold.
5.3 TEMPORAL AVERAGED LOss FUNCTION AND OUTPUT REGULARIZATION
We also adopt and customize the temporal efficient training (TET) loss function for output regular- ization (Deng et al., 2021) to further improve the performance. We adopt the MSE regularization loss from TET to improve the layer-wise spiking activity. On top of that, We have used the aver- age learnable potential threshold of all the layers to compute the MSE loss between output spikes and Vin. The output of the LT-SNN will be regularized towards the averaged layer-wise potential threshold ?::
1 T L = T Lcn(O(),,) +1 B) 7) Σ MSE(O(), [=1
T
(12)
TET Loss
MSE regularizer
Where
Ω
(13)
L
LOE is the cross-entropy loss between the predicted and target class, MSE is the mean square difference between the network output and the regularization target Ω Ot)) represents the output spikes at the final layer, and T represents the simulation time steps of the input sample. While 3 and Ω are the weights of the moving average and target value of regularization, respectively. We choose Ω as the mean learnable threshold of the forward pass regularize the output based the layer- wise learnable threshold, where l represents the total number of layers in our SNN architecture.
6 EXPERIMENTAL RESULTS
In this section, we validate the proposed LT-SNN algorithm with multiple event-based com- puter vision datasets, including DVS-CIFAR10 (Li et al., 2017), N-Cars (Sironi et al., 2018), N- Caltech101 (Orchard et al., 2015) and Prophesee Automotive Genl (de Tournemire et al., 2020). Unlike prior works that only employ large-sized VGG or ResNet models (>1IM parameters), the proposed algorithm is validated on compact VGG models (2.4-7.1M) parameters) and light-weight MobileNet-V V1 model (1,5M parameters) (Howard et al., 2017). As mentioned earlier, our objective to achieve maximum SNN accuracy with the least model parameters. Table 3 summarizes the architectures of the experimented models.
Table 3: Convolutional SNN model architectures for LT-SNN training. "C3", "DW", "MP2", "AP2" and "FC" represent 3x3 convolution layer, depth-wise separable block (Howard et al., 2017), 2x2 max-pooling, 2x2 average-pooling, and fully connected layer, respectively.
Model
Architecture
MobileNet-Light
BDC3-61DW-6HDW-PPP2 www.://www---WWS-. WOW.DWWAPP-2-6000- FC2-F
VGG-7
32C3-32C5-44 64C3364C3--P2P2 AP2.122231331100 AA-CC-AAP2220603555000
VGG-9
32C3-64C3-644
32C3-MP2
128C3 64C1 128C3 MP2 256C1. 1tt.:://122112222....
Yoln-V2
312C3-25601-5 1-1-----1----22 216C1-2861111 102-110443---11111 AAA.CAAAA--RC12--CCCC

---

Under review as a conference paper at ICLR 2023
Table 4: Experimental results of the proposed LT-SNN on DVS-CIFARIO and CIFAR-10 datasets.
Except "VGG-9 (4-bit)", bit precision is used for all results of prior works and this work.
Simulation Length
Dataset
Method
Architecture
Parameters
Accuracy %%
ASF-BP 2021)
VGG-Like
15.2M
50
62.50
tdBN (Zheng et ol., 2021)
ResNet-19
11.1MM
10
67.80
ParamLII- (Fang al., 2021b)
VGG-Like
17.4M
20
RecDis (Guo et 2022)
ResNet-19
11.18M
10
72.42
TET (Deng et al., 2021)
VGG.Likke
9.27M
10
77.33
DVS-CIFAR-10
DSR (Meng et al., 2022)
VGG-11
9.34M
30
75.70
This
VGG-11
9.34M
30
This work
MobileNet-Vi (light)
1.28M
30
75.70
This work
VGG-7
191M
30
78.93
This work
VGG-9 (4-bit)
7.07M
30
80.04
This work
VGG-9
7.07M
10
79,10
This work
VGG-9
7.07M
78,30 8
Hybrid (Rathi & Roy, 2020)
ResNet-20
12.3M
10
92.54
CIFAR-10
IBBN (Zheng et 2021)
ResNet-19
11.18M
93.16
DSR (Meng 2022)) This work
ResNet-18
11.18M
6
91,89
ResNet-19 19
11.18M
93.97 6
The experiment is rigorously performed based on the open-sourced DSR implementation with 6 time steps.
Table 5: Experimental results of the proposed LT-SNN on N-Caltech101 and N-Cars datasets.
Dataset
Method
Representation
Accuracy( %)
Pretrain
YOLE (Cannici et al., 2019)
Histogram
70.02
False
EST (Gehrig et al., 2019)
Histogram
81.70
Truc
N-Caltech101
AsyNet (Messikommer al., 2020)
Histogram
76.10
False
AEGNN (Schaefer et 2022)
Graph
66.80
False
This work
Binary Spike
77.71
False
NDA (Li et al., 2022)
Binary Spike
91,9
False
AsyNet (Messikommer al., 2020)
Histogram
94.4
False
YOLE (Cannici et al., 2019)
Histogram
92.7
False
N-Cars
AEGNN (Schaefer al., 2022)
Graph
94.5
Fulse
Object (Cordone al., 2022)
Binary Spikes
92.4
False
This work
Binary Spike
95.02
False
SNN Training for Full-/Low-Prreciion Inference We perform LT-SNN training execute in- ference with both full full-precision and 4-bit precision. To train SNNs for low-precision inference, we adopt the Statistic-Aware Weight Binning (SAWB) quantizer (Choi al., 2019) to compress the layer-wise weights of the LT-SNN-VGG-9 network down to 4-bit precision. We separately perform and compare the full-precision and quantized aware training of our SNN architectures using hyper- parameters mentioned in Appendix A. Using our LT-SNN scheme, we achieve SoTA results for both full-precision and low-precision SNNs, as reported in Table 4 and Table 5. Proposed LT-SNN and DSR train the threshold in a layer- wise manner and the potential threshold for each layer is different. The rest of the results reported from other SoTA employ fixed Vtn for all the layers.
Image Classification Task We validate the proposed LT-SNN algorithm on both event- event-based im- age classification datasets and conventional computer vision dataset with RGB images. The LT-SNN models are directly trained from scratch, and the detailed input preprocessing and experimental setup are summarized in Appendix A. Table summarizes the performance of LT-SNN on DVS-CIFAR10 and CIFAR-IO (with RGB images) datasets. For DVS-CIFARI0 dataset, compared to the current SoTA method (Deng et al., 2021), the 4-bit VGG-9 model trained by proposed LT-SNN algorithm achieves 2.71% % accuracy improvement with 1.31 x less parameters and 10.48x model size reduc- tion (MB). Furthermore, the proposed LT-SNN demonstrates consistently superior performance with different simulation length (from time steps to 30 time steps). For the conventional RGB CIFAR- 10 dataset, I.T-NNN also achieves new SoTA performance with ResNet-19 architecture, where we adopted the tdBN (Zheng et al., 2021) scheme to stabilize the direct SNN training process. Based on the official implementation of DSR (Meng al., 2022), we rigorously conduct the experiment of DSR on CIFAR-10 dataset with simulated time steps. Additionally, to demonstrate the effee- tiveness of learnable threshold, we conduct more experiments using SGP with and without learnable threshold. Proposed LT-SNN with SGP surpassed SNN implementations with the fixed threshold for different compact networks reported in Appendix C, Table 9.
In addition to the CIFAR datasets (Li et al., 2017), we evaluate the proposed LT-SNN algorithm on N-CalTech101 (Orchard et al., 2015) and N-Cars (Sironi al., 2018) datasets. As shown in
8

---

Under review as a conference paper at ICLR 2023
Table 5, the proposed LT-SNN achieves new SoTA performance both datasets with 1.61% (N- Caltech101, without pre-training) and 0.52% (N-Cars) accuracy improvements. Compared to the conventional histogram- based computation (Messikommer et al., 2020; Cannici et al., 2019) that use non-binary activations, the proposed L.T-NNN enables end-to-end binary computation with binary spikes throughout the entire SNN,, achieving superior performance with high hardware compatibility. Additional results comparing the baseline of LT-SNN with TET and DSR using DVS-CIFARIO dataset are summarized in Appendix C, Table 7.
Impact of the Event Time Steps SNNs require iterative computation and membrane potential accumulation, which motivated prior works to exploit the computation reduction with less number of time steps represent the incoming event. We also validated LT-SNN with different simulated time steps on the DVS-CIFARIO dataset. As shown in Table 4, the proposed LT-SNN manages to achieve SoTA performance with reduced time steps and compact models, compared to the prior works. With the same 10 time steps as TET Deng et al. (2021), LT-SNN achieves 1.77% accuracy improvement, without using any data augmentation. With the extended 30 time steps, LT-SNN achieves 80.04% SoTA accuracy on DVS-CIFAR 10 dataset.
Object Detection Task In addition to the classification datasets, we validate the proposed LT-SNN on large-sized Prophesee Automotive Genl dataset (de Tournemire al., 2020) for object detection task. With 228, 123 bounding boxes for cars and 27,658 for pedestrians, the Genl dataset (de Tourne- mire al., 2020) considered as the most complex event-based computer vision task. Unlike the prior works use accumulated histogram (Mes- sikommer et al., 2020; Cannici et al., 2020; Cordone et al., 2022) graph-basee basph-based input representation (Schae- fer et al., 2022), we translate DVS event to pure binary frames and synchronize them with artificial actual ground truths from (Perot et al., 2020) for network training. The detailed data preprocessing setup is summarized in Ap- pendix A. The binary input events and intermediate spikes enables the end-to-end binarized computing for LT-SNN, elevates the computation efficiency with simplified data format. We develop LT-SNN- YoloV2 feature extraction network followed by Yolo loss to localize the objects and compute mean average precision (mAP) for detected bounding boxes against the ground truths shown in Fig- ure 5. Table and Table 10 Appendix C cluccidate that, in comparison to the current SNN-based SoTA with fixed potential threshold, our LT-SNN-custom- Yolov2 records SoTA mAP of 0.298 on Prophesee Automotive Gen1 dataset.
50-
150
250
300
100
350
200
250
300
350
400
Figure 5: Inference results of LT-SNN
on Prophesee Automotive Genl dataset.
Table 6: Experimental results of the proposed LT-SNN on Prophesee Automotive Genl dataset.
Method
Model Architecture
Spiking Model
mAP
Asynet (Messikommer et al., 2020)
FB- TB-Sparse
No
0.145
MatrixLSTM (Cannici et al., 2020)
ResNet- RetinaNet
No
0.3
RED (Perot et al., 2020)
No
0.41
VGG-I1+SSD (Cordone et al., 2022)
VGG+SSD-SNI Custom-Yokk VoloNN
Yes
0.187
This work
Yes
0.298
7 CONCLUSION
In this work, we propose novel SNN training algorithm with learnable threshold (LT-SNN), which optimizes the layer-wise threshold with direct SNN training. As one of the first studies on this topic, the proposed LT-SNN unleashes the firing constraints that were imposed in prior works, without in- troducing any additional high- sech-precision scaling factor to the spike generation. The proposed method with full and low-precision training has been verified multiple event-based image classification and object detection datasets. LT-SNN improves the of accuracy for DVS-CIFARIO dataset by 2.8% together with 10.48x smaller model size. For object detection Prophesee Au- tomotive Genl dataset, the LT-SNN outperforms SNN-based SoTA mAP by 0.11 with end-to-end binary computation.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Marco Cannici, Marco Ciccone, Andrea Romanoni, and Matteo Matteucci. Asynchronous Convo- lutional Networks for Object Detection in Neuromorphic Cameras. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 0-0, 2019.
Marco Cannici, Marco Ciccone, Andrea Romanoni, and Matteo Matteucci. A Differentiable Recur- rent Surface for Asynchronous Event-based Data. In European Conference on Computer Vision (ECCV), pp. 152- 2020.
Jungwook Choi, Swagath Venkataramani, Vijayalaksmii Viji Srinivasan, Kailash Gopalakrishnan, Zhuo Wang, and Pierce Chuang. Accurate and efficient 2-bit quantized neural networks. Confer- ence on Machine Learning and Systems (SysML), 1:348-359, 2019.
Loic Cordone, Benoît Miramond, and Philippe Thierion. Object Detection with Spiking Neural Networks on Automotive Event Data. arXiv preprint arXiv:2055004333 2022.
Pierre de Tournemire, Davide Nitti, Etienne Perot, Davide Migliore, and Amos Sironi. Large Scale Event-based Detection Dataset for Automotive. arXiv preprini arXiv:200.004999, 2020.
Shikuang Deng. Yuhang Li, Shanghang Zhang. and Shi Gu. Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting. In International Conference on Learning Represen- tations (ICLR), 2021.
Peter U Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. Fast-classifying, High-accuracy Spiking Deep Networks Through Weight and Threshold Balanc- ing. In International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2015.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep Residual Learning in Spiking Neural Networks. Advances in Neural Information Processing Systems (NeurIPS), 34:21056-21069, 2021a.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incor- porating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2661-2671, 2021b.
Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew Davison, Jörg Conradt, Kostas Daniilidis, et al. Event-based Vi- sion: A Survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(1):154-180, 2020.
Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. End-to-end Learning of Representations for Asynchronous Event-based Data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5633-5643, 2019.
Yufei Guo, Xinyi Tong, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Zhe Ma, and Xuhui Huang. RecDis-SNN: Rectifying Membrane Potential Distribution for Directly Training Spiking Neural Networks. In /EEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 326-335, 2022.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. RMP-SNN: Residual Membrane Poten- tial Neuron for Enabling Deeper High-accuracy and Low-latency Spiking Neural Network. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv preprint arXiv: 1704.0486, 2017.
10

---

Under review as a conference paper at ICLR 2023
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and Training of Neural Networks for Efficient aateger-arithmeticoon Inference. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2704-2713, 2018.
Maarten H. P. Kole and Greg J. Stuart. Is Action Potential Threshold Lowest in the Axon? Nature Neuroscience, ((11):1253-1255) 2008.
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training Deep Spiking Neural Networks Using Backpropagation. Frontiers in Neuroscience, 10:508, 2016.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar 10-DVS: An Event-stream Dataset for Object Classification. Frontiers Neuroscience, 11:309, 2017.
Yuhang Li, Youngeun Kim, Hyoungseob Park, Tamar Geller, and Priyadarshini Panda. Neu- romorphic Data Augmentation for Training Spiking Neural Networks. arXiv preprint arXiv:2203.06145, 2022.
Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Thi-Quan Luo. Train- ing High-Performance Low-Latency Spiking Neural Networks by Differentiation Spike Rep- resentation. In IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 12453, 2022.
Nico Messikommer, Daniel Gehrig, Antonio Loquercio, and Davide Scaramuzza. Event-based Asynchronous Sparse Convolutional Networks. European Conference on Computer Vision (ECCV), pp. 415-431, 2020.
Emre Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based Optimization Spiking Neural Net- works. IEEE Signal Processing Magazine, 36(6):51-63, 2019.
Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor. Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades. Frontiers in Neuroscience, 9:437, 2015.
Etienne Perot, Pierre de Tournemire, Davide Nitti, Jonathan Masci, and Amos Sironi. Learning to Detect Objects with a I Megapixel Event Camera. Advances in Neural Information Processing Systems (NeurIPS), 33: 33:1669911665, 2020.
Nitin Rathi and Kaushik Roy. DIET-SNN: Direct Input Encoding with Leakage and Threshold
Optimization Deep Spiking Neural Networks. arXiv preprint arXiv:2008. 03658, 2020.
Bodo Rucckauer, Alexa-Alexandra Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and Tools for the Conversion of Analog to Spiking Convolutional Neural Networks. arXiv preprint arXiv:1612.04003 2016.
Simon Schaefer, Daniel Gehrig, and Davide Scaramuzza. AEGNN: Asynchronous event-based graph neural networks. JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 12371-12381, 2022.
Guobin Shen, Dongcheng Zhao, and Yi Zeng. Backpropagation with Biologically Plausible Spa- tiotemporal Adjustment for Training Deep Spiking Neural Networks. Patterns, pp. 100522, 2022.
Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1731-1740, 2018.
Hongze Sun, Wuque Cai, Baoxin Yang, Yan Cui, Yang Xia, Dezhong Yao, and Daqing Guo. A Synapse- Synapse-Threshole Threshold Synergistic Learning Approach for Spiking Neural Networks, arXiv preprint arXiv:2206. 06/29, 2022.
11

---

Under review as a conference paper at ICLR 2023
Hao Wu, Yueyi Zhang, Wenming Weng, Yongting Zhang, Zhiwei Xiong, Zheng-Jun Zha, Xiaoyan Sun, and Feng Wu. Training Spiking Neural Networks with Acumulated spiking flow. In Pro- ceedings of the AAAI conference Artificial Intelligence (AAAI), volume 35, pp. 10320-10328, 2021.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct Training for Spiking Neural Networks: Faster, Larger, Better. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 33, pp. 1311-1318, 2019.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going Deeper with Directly-trainedt Larger Spiking Neural Networks. In Proceedings the AAAI Conference on Artificial Intelligence (AAAI), volume 35, pp. 11062-11070, 2021.
A DETAILED EXPERIMENTAL SETUP
Data Preprocessing To ensure hardware efficient L.T-NNN based training, we convert DVS-events binary frames in the event pre-processing stage. For classification data, we perform sampling different time steps and transform events to binary tensors of shape [Batch. Time, Channels, Height, Width]. Similarly,
follow the same convention to convert Prophesee Genl events binary frames. Then the generated binary frames are synchronized with artificial ground truth from (Perot al., 2020). Finally, the events and annotations are translated the tensors of shape [Batch, Time, Channels, Height, Width] and [Batch, Number of boxes, Bounding box] respectively.
Training Parameters We train our proposed LT-SNN based classification and object detection ar- chitectures using PyTorch. Regarding hyperparameter selection, we use the Adam optimizer where the learning rate set to 0.001 and cross entropy loss plus mean square error is computed between detected and target classes for sock-propagation and output regularization. For output regularization, the moving average weight 3 set to 0.45 for both the full-precision and low-precision training. Fi- nally, each LT-SNN based architecture from Table 3 trained for 200 epochs and observe robust convergence of our proposed LT-SNN based architectures in less than 50 epochs.
12

---

Under review as a conference paper at ICLR 2023
B ALGORITHM
Algorithm LT-SNN Algorithm
Initialize: Input spike S. Membrane potential uf Learnable potential threshold as Vi Output spike as O and regularization ratio as
for t < 1 to I" do
Forward Pass:
Compute ui, O using Eq. 2
for n 1 N do
Forward Pass:
Compute uy, O using Eq. 3
end end
p
Prediction YY OP)
Loss:
Compute 92 using Eq. 13 with detached ViA.
LLT.SNN = LCE(Y, Of) + (1 3)MSE(()
Backward Pass:
Initialize:
LL SNN /00 0
for I 4- T to do
SNLT/SNN/00P for n t- to do
Spatial Gradient(OI.cr/000)) Output (atilaariizaiin@M//////////
Compute gradient using Eq. 6
Compute @Lce/OVi, using Eq. 9 and Eq. 10
Mask ÔLCE OVV with active pixel density using Eq. 11
end
end
13

---

Under review as a conference paper at ICLR 2023
C ADDITIONAL EXPERIMENTAL RESULTS
have trained the VGG-11 architecture reported by DSR and TET using the proposed SGP for the layer-wise learnable threshold to present an apple-to-apple comparison of LT-SNN with DSR and TET work. Table 4 elucidates that LT-SNN shows 3.81% and 2.18% accuracy improvement in comparison to the DSR and TET respectively.
Table 7: Bascline comparison of proposed SGP with current SoTA.
Method
Architecture
Threshold
SG Func.
Top-1 I Accuracy (%)
TET [R3]
VGG-11
Fixed
Triangle
77.33
DSR [R4]
VGG-I1
Fixed
STSG
75.70
This work
VGG-11
Learnable
SGP
79.51
To compare the performance of SGP with lower learning rate based SNN training, we have per- formed extra experiments with VGG-9 using high Ir (0.1) for W and relatively low Ir (0.001) for Vih We observe that such high Ir value for W with "Adam" optimizer didn't help in model con- vergence. We performed another experiment with Ir (0.001) for W and Ir (0.00001) for Vnn The Table summarizes the experimental results, which show that the separate learning rate setting is not sufficient to fully optimize the learnable potential threshold.
Table 8: Performance comparison of SGP and lower learning rate to balance out Vaa and Wiy for LT-SNN training.
Model
W Ir
Vth Ir
SGP
Accuracy (%)
Converged
VGG-9
0.1
0.001
X
20.40
X
VGG-9
0.001
0.00001
X
77.20
VGG-9
0.001
0.001
80.04
We have conducted more experiments by disabling the learnable threshold to compare the perfor- mance of fixed and Iearnable threshold with the identical setting. We have used DVS-CIFARIO data for the classification task and Prophesee Automotive Genl data for the object detection task. Table 9 and Table 10 demonstrate the superiority of our proposed layer-wise learnable potential threshold scheme in comparison to the fixed potential threshold for all the layers. Proposed SGP for learnable potential threshold increases SoTA accuracies by 2.02% and 1% with VGG-9 and MobileNet-vi respectively and mAP by 0.176 with V2-SNN
Table 9: Performance comparison of learnable potential threshold and fixed potential threshold for the classification networks.
Method Architecture
Threshold
SG Func.
Top-1 Accuracy (%)
This work
VGG-9
Fixed
Triangle
78.02
This work
VGG-9
Learnable
SGP
80.04
This work
(LLite
Fixed
Triangle
74.70
This work (LLite
Learnable
SGP
75.70
Table 10: Performance comparison of learnable potential threshold and fixed potential threshold for the object detection networks.
Method
Architecture
Threshold
SG Func.
mAP
This work Custom-YoloV22 VoloV2-SNN
Fixed
Triangle
0.122
This work Custom-Yooke V2-SNN Learnable
SGP 0.298
We choose the sigmoid function for SGP based the experimental results that achieved by setting up an ablation study. From Table 11, we could conclude that the sigmoid function the best choice to ensure stable training performance and high validation accuracy.
14

---

Under review as a conference paper at ICLR 2023
Table 11: Comparison of different surrogate functions performance SGP,
Model
Epochs
SGP
Accuracy (%)
Converged
VGG-9
200
ArcTan
77.81
VGG-9
200
Triangle
70.83
VGG-9
200
Piece-wise
78.81
V
VGG-9
200
Sigmoid
80.04
To ensure the training stability of the proposed SGP, we have trained VGG-9 for 200 epochs. From Figure 6, we can observe that SGP for LT-SNN didn't hurt the training stability for 200 epochs.
DVS.0 CIFCIIAAA AR10 Accuracy (%)
Average Potential Threshold
70
Learnable Threshold
Validation Accuracy
100
120
160
Training Epochs
Figure 6: Stable training process of the proposed LT-SNN algorithm with the extended training effort on WSS-CIFARIO dataset.
15


Reviews:
[{"review_id": "oT6XPwZ1cuU", "reviewer": "Reviewer_fvZZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose LT-SNN, a novel SNN training algorithm with self-adaptive learnable potential threshold to improve SNN performance. To stabilize the SNN training even further, the author propose separate surrogate gradient path (SGP), a simple-yet-effective method that enables the smooth learning process of SNN training. The proposed method is validate on multiple event-based datasets, including both image classification and object detection tasks.", "summary_of_the_review": "Although the paper gets better results with more parameters, the problem importance and the motivation of the proposed method is unclear. Therefore, I give a weak reject for the paper.", "strength_and_weaknesses": "Strength\n\n1. The proposed method is shown to outperform SOTA on multiple event-based datasets.\n2. The paper is well written and easy to follow.\n\n\nWeakness\n\n1. I don't see a clarification for the problem importance. Why learnable threshold is important in SNN? If the authors want to make SNN more biologically plausible, should the threshold lie in the specific range? I don't see any discussion or insight in the paper.\n2. The proposed method is not well motivated. For example, the authors propose a separate gradient path in sec. 5.1, but why we should use a separate path instead of single one? Is there any insight or theoretical guarantee? Also, for Eq.9, the authors select the sigmoid function as the activation. Why would the authors select this particular activation function? Is there any biologically insight?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kt_ftelcWb9", "reviewer": "Reviewer_A7Jv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel training method for Spiking Neural Networks. Typically, the non-differentiability of the spiking neuron was by-passed with a differentiable approximation (e.g. the triangle function). However, each spike depends on two variables: the previous neuron's voltage, and a voltage threshold above which the spike is created. In prior works, the network optimized over the voltage during training, but kept the voltage threshold as a constant, due to training instability issues. This work resolves these instability issues by introducing a separate gradient approximation for the voltage threshold, consisting of the triangle function modulated by a sigmoid function. During backpropagation, the voltage and voltage threshold will receive separate gradients based on this method. Empirical experiments are performed to demonstrate the stability of this method, and the authors demonstrate significant improvements in experiments for image classification and object detection over prior works, with much smaller model sizes.", "summary_of_the_review": "Overall, the proposed method seems like a simple modification to existing SNN networks which provides a significant improvement. However, I may be missing some context as to the novelty of this work.\n\nEDIT: After reviewing the discussions from the other reviewers, it seems like there are legitimate concerns about the novelty and experiments of this work. Taking this in mind, I am modifying my review to a 5 (although more of a 5.5). ", "strength_and_weaknesses": "Overall, the motivation and method is very intuitive, and the proposed solution is simple and makes sense. The authors provide a good amount of background, and justify well the need for the proposed contribution. While the method is quite simple, the improvements in the experiments are quite impressive, especially considering the significant reduction in model size. \n\nOne open question is how much the regularization in Section 5.3 contributes to the work. It seems to only be briefly mentioned, but is not ablated, and it's unclear how important this is to the improvements seen in the experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iosxoUyr5a", "reviewer": "Reviewer_btjV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an effective way to train the spike thresholds of spiking neural networks, while the naive surrogate gradient method fails to do so. The proposed method achieves SOTA performance on several image classification tasks and an object detection task.", "summary_of_the_review": "I give a \"reject\" because:\n\n1. I challenge sec 4 and the motivation of the proposed method totally. You do not need to create a hypothesis and then give some observations. The issue is easy to handle: shrink dl/dVth or shrink the corresponding lr. See weakness 2.\n\n2. The experiments are not carefully designed. I give some examples as following. In fig. 2(a), you do not fine-tune the lr for training vth (see bullet 1). In tables 1 and 2, DSR and TET use vgg-11, but you use vgg-9 (btw, I am confused about the term \"true binary spike\", DSR just uses binary spike). No ablation studies, which are super important in this work, see weakness 3.", "strength_and_weaknesses": "Strength:\n\n1. The proposed method is simple, and can be directly plugged into other existing methods.\n\n2. There are experiments on non-classification tasks. It is encouraged to explore the performance of SNNs on various tasks.\n\n3. It is interesting to see the potential of quantized small-scale SNNs.\n\nWeaknesses:\n\n1. In table 2, does \"GPW for all\" mean that all neurons share the same learnable Vth? And does \"SGP\" mean that each layer has its own Vth? For the experiments shown in Fig. 2, does each layer have its own trainable Vth?\nFor each experiment, the authors need to make clear statements on whether all the neurons share the same Vth, or each layer has its own Vth, or each neuron has its own Vth. I also suggest the authors to combine eqns. 6 and 11 in one place, and clarify the range of the summation in eqn. 11.\n\n\n2. I think the catastrophic performance drop in fig. 2a is not hard to understand. Let me first guess that each layer has its own trainable Vth. For one layer, you need to sum up dL/dVth_i for each neuron i in that layer to get dL/dVth. Since dL/dVth_i, dL/du_i, and dL/dWij have the same order, |dL/dVth| will be hundreds of or thousands of times large than |dL/dWij|, depending on the model width. Therefore, a learning rate suitable for updating Wij may be too large for updating Vth. So we can easily overcome the issue by using a small learning rate for Vth (e.g., set init lr=0.1 and 0.001 for W and Vth, respectively). Your proposed method basically shrinks dL/dVth_i with a sigmoid function, so it has a similar effect as using smaller lr.  Can you compare the two methods experimentally? Can you show the benefit of your proposed voltage-dependent shrinkage scheme?\n\n3. I am concerned about the direct performance gain of the proposed method. The authors do not show the performance when disabling threshold training and keeping other settings the same. So it is not clear whether the sota results are achieved due to the proposed method or, say, network structures. Table 2 seems to show the direct effect of the proposed method, but the baseline uses a different network structure (the title of table 2 is misleading!). I would appreciate it if the authors could conduct ablation studies.\n\nMinor: \n1. Fig.1: DSR-VGG11 adopts learnable thresholds, as repeatedly pointed out in the manuscript.\n2. In eqns. 6 and 9, a minus sign is missed. ds/du and ds/dVth have opposite signs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Z8ma5HlVdn", "reviewer": "Reviewer_kp4g", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the thorny problem of training spiking deep neural networks that are marred with problems of non-differentiable nonlinearity, sparse responses and limited expressive power. The authors propose an adaptive threshold mechanism so that the threshold is trained using a different differentiation path from that used for the weight updates.\nIn extensive experiments the authors demonstrate advantages of their approach.", "summary_of_the_review": "This is an impressive piece of work substantially improving on SOTA SNN training. However, questions remain about replicability of this work. ", "strength_and_weaknesses": "\nStrength \nThe paper offers a new view on learning adaptive thresholds in SNNs. Their SNNs acheve competitive performance on standard tasks even with by an order of magnitude fewer parameters. The paper is well organized and clearly written.\n\nWeaknesses\nWhile the authors bring up the question of biological relevance of SNNs and their learning mechanisms, they don't elaborate on this topic and don't provide evidence that their approach is more biologically realistic as compared to other SNN training schemes.\nThe authors do not comment if their code will be publicly available raising questions about replicability of their research.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BGNIuAvYbm", "reviewer": "Reviewer_Dob4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a layer-wise learnable threshold approach for SNNs. The proposed method design two separate surrogate gradient paths for the gradient of membrane potential and learnable threshold. Different classification and object detection tasks reflect the effectivenesses of the proposed LTSNN training scheme.", "summary_of_the_review": "Too many things are unclear to me at the moment, as discussed in the section on \"strengths and weaknesses.\" In addition, many works have developed different dynamic threshold schemes for SNNs. The authors did not mention them at all. For example, in \"Biologically Inspired Dynamic Thresholds for Spiking Neural Networks,\" the authors define dynamic thresholds without learning. More importantly, I do not think the proposed approach is able to imitate \"the self-adaptiveness of the biological nervous system.\" \n\nIn summary, I do not recommend the submission. ", "strength_and_weaknesses": "\nWeaknesses and questions:\n\n-1. The experiments recorded in Table 1 are not clear to me. \n\n       a. Why do \"such constraints in the training process largely limits the learnability of SNN\"? What's the definition of \"learnability\"?\n       b. What is the definition of \"true binary spikes\"? Biologically, an action potential is triggered when the membrane potential is higher than a threshold. The output potential should NOT be a binary value in any case. The spike (action potential) is binary. \n       c. How hardware deals with layer-wise varied spikes is out of the scope of the paper. I assume the \"hardware\" means neuromorphic hardware. Biologically, each neuron has a different dynamic threshold. The current neuromorphic hardware can barely mimic the simplest biological neuron. Thus, algorithm and theoretical study should not be limited or favored to existing neuromorphic hardware. \n       d. DSR and LTSNN used different base models, VGG-11 for DSR and VGG-9 for LTSNN. Is it fair?\n\n-2. The falsified hypothesis 1 is based only on VGG-9 and DVS-CIFA10 is not convincing. In other words, how general of observation 1? In addition, I am not convinced by the experiments associated with Figure 2. If we stop training STSG before 80 epoch, STSG also has stabilized training, right? In other words, why the authors stopped training LTSNN at epoch 100? What would happen if the authors train LTSNN 200 epoch?\n\n -3. Super unclear of section 5.3! The L in Eq. 12 is the same as the one in Eq. 13? What's L_{CE}, MSE, O(t), and T? More importantly, why the MSE regularizer is necessary? \n\n-4. For the experiments associated with Table 4, why did the authors not use the same architecture of other competing approaches? What's the logic behind the experimental design?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject