A High Performance and Low Latency Deep Spiking Neural Networks Conversion Framework
Authors: Wenjie Song
Yang Li
Yuan Zhang
Di Xie
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Spiking Neural Network
ANN-SNN Conversion
Object Detection
Object Recognition
Event Camera
ImageNet
MS-COCO

Abstract: Spiking Neural Networks (SNN) are promised to be energy-efficient and achieve Artificial Neural Networks (ANN) comparable performance through conversion processes. However, a converted SNN relies on large timesteps to compensate for conversion errors, which as a result compromises its efficiency in practice. In this paper, we propose a novel framework to convert an ANN to its SNN counterpart losslessly with minimal timesteps. By studying the errors introduced by the whole conversion process, an overlooked inference error is reveald besides the coding error occured during converting. Inspired by the quantization aware traning, a QReLU activation is introduced during training to eliminate the coding error theoretically. Furthermore, a buffered non-leaky-integrate-and-fire neuron that utilizes the same basic operations as in conventional neurons is designed to reduce the inference error. Experiments on classification and detection tasks show that our proposed method attains ANNs level performance using only $16$ timesteps. To the best of our knowledge, it is the first time converted SNNs with low latency demonstrate their capability to achieve high performance on nontrivial vision tasks. Source code will be released later.


A High Performance and Low Latency Deep Spiking Neural Networks Conversion Framework
Anonymous Author(s)
Affiliation
Address
ema i
Abstract
Spiking Neural Networks (SNN) are promised to be energy-efficient and achieve Artificial Neural Networks (ANN) comparable performance through conversion processes. However, a converted SNN relies on large timesteps to compensate for conversion errors, which result compromises its efficiency in practice. In this paper, we propose a novel framework to convert an ANN to its SNN counterpart losslessly with minimal timesteps. By studying the errors introduced by the whole conversion process, an overlooked inference error is reveald besides the coding error occured during converting. Inspired by the quantization aware traning, QReLU activation introduced during training to eliminate the coding error theoretically. Furthermore, buffered pon-leaky-ineegaate-ndd that utilizes the same basic operations as in conventional neurons is designed to reduce the inference error. Experiments on classification and detection tasks show that our proposed method attains ANNs level performance using only 16 timesteps. To the best of our knowledge, it the first time converted SNNs with low latency demonstrate their capability to achieve high performance on nontrivial vision tasks. Source code will be released later.
3
5
10
neuron
11
12
13
14
15
18
Introduction
17
Recently, Spiking Neural Networks (SNN) have attracted great deal of researchers' attention due to their essential advantages, such as spatio-tempore information processing capability and energy-efficient with low power consumption. Unlike conventional Artificial Neural Networks (ANN) communicating with every neuron between layers during every inference step, SNN only passes sparse events when particular neurons are fired. This compute-on-demand property of SNN is very friendly to low-power hardware, and suitable for neuromorphic platforms, such as event-based camera [ and brain-inspired chips (32). Such dedicated bionic systems can significantly reduce memory usage and energy consumption, which is very appealing to a wide range of applications like autonomous mobile robots.
19
21
29
24
25
26
27 Despite the promising characteristics, it still a challenging problem to train sigh-pefforance deep SNNs for practical tasks. Directly applying the backpropagation approach is infeasible because of the non -indiffferentable nature of the spike activity. Taking advantage of the ANN's success, conversion methods have been proven to be very promising direction in achieving high performance SNN. Cao et al. 4 first convert ANNs to SNNs by demonstrating the relationship between spiking neuron and ReLU function for rate- based methods. Diehl al. [9] propose weight normalization approach 33 to mapping weights from ANN to SNN. Later, Rueckauer et al. [39 give theoretical analysis on ANN-SNN conversion. A bysutry-subtraction IF neuron proposed better handle the accuracy degradation during the conversion. They also propose the weight normalization algorithm that uses percentile function instead of max value achieve better performance. Kim al. 23 propose channel-win normalization to further reduce the performance gap between ANNs and SNNs.
29
31
34
35
36
37
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

38
Although great progress has been made, those converted SNNs suffer from accuracy-latenc trade- off 9 BT. As result, the converted SNNs need longer time to achieve comparable precision with their counterpart ANN during inference and compromises the efficiency in practice. To deal with this problem, hybrid methods [37 36 are proposed to further finetune a converted SNN with approximated gradients to decrease the inference time. On the other hand, for the theoretical training gaps between ANNs and SNNs, Ding et al. [10] propose Rate Norm to better choose the scale factor and introduce extra loss to decrease the latency. Deng and Gu g utilize training samples to transfers the weights to the target SNN by combining threshold balance and soft-reset mechanisms. However, these works only focus reducing the errors introduced by mapping feature values spike trains.
39
40
41
42
43
44
45
47
With an in-depth analysis of the entire coversion process, we recognize two types of conversion errors, namely coding error and inference error, that degrade the performance of SNN. As other conversion approaches mainly focus on the coding errors, our motivation to reduce both the coding and inference errors. Inspired by the analysis in Rueckauer et al. [39], we notice that the SNN with step function resembles quantized ANÑs. By establishing a striet equivalency between spiking neuron and quantized activation function, the coding error can be eliminated completely under limited timesteps. Further, we observe that the assumption in rate-based conversioin is inconsistent with the actual neuron behavior and the SNN's s dynamics during the inference. Most of the previous conversion
48
49
50
51
53
54
55
literature overlook this problem, which limits the performance of the algorithms and leads to longer simulation time to achieve comparable results of the original ANN.
57
In this paper, we propose novel High Performance Conversion (HPC) method to drastically close the gap between ANNs and SNNs. Specifically, a QReI.U activation is proposed to eliminate the coding error. Since QReIUU is applied to the AÑN training phase, no extra effort required during the conversion. Besides, noval buffered non-leaky IF neuron is designed to compensate for the inference error and retain SNN's efficiency during inference. Utilizing a new form of activation function and novel efficient spiking neuron, SNN obtained by our proposed approach achieves high performance and low latency simultaneously. With only 16 timesteps. HPC achieves state-of-the-art performance across classification and detection tasks. Our major contributions are summarized as:
58
59
60
61
63
64
65
An overlooked inference error is discovered. The conversion errors including both coding errors and inference errors are theorically analized to clearify the reason of performance degration during conversion.
66
67
68
A novel high performance conversion method is proposed to deal with the two types of conversion errors. The coding errors are eliminated completely in theory and the inference error problem is relieved to great extent.
69
70
71
Comprehensive experiments on different tasks demonstrate the efficacy of the proposed method. To the best of our knowledge, it is the first time converted SNN achieves ANN level performance while maintaining high efficiency and low latency across various tasks.
73
74
2
Related Work
75
Up to now, there exist two main categories of supervised training strategies for SNN: approximated error backpropagation methods, and conversion methods.
76
77
Our proposed work is mainly related to ANN-SNN conversion methods with rate-based coding 9
78
35
The goal of conversion approaches is to remain the high performance of ANN in SNN framework. Recently, Xing et al. 44 extend the territory of conversions by adding supports to
4
39
79
80
more structures such as softmax activation and residual block. Instead of using longer timesteps to aproximate ANN accurately, conversion errors are analized and compensated by adjusting the parameters of converted SNN (27] Rate Norm is proposed [10] to better choose the scale factor.
81
83
Deng and Gu [8] utilize training samples to transfers the weights to the target SNN by combining threshold balance and soft-reset mechanisms. As general approach, Kim et al. [23] show that conversion can be applied to detection tasks. However, only coding error are considerd in those work. We notice that the inference error "unevenness error" is described in the parallel work (.). However, the "unevenness error" is ignored during the analysis of conversion error in their work.
84
85
87
88 Other researchers 2 29 21 try to approximate the gradient to train the SNN by replacing the thresh- 89 old function with other functions. A comprehensive summary of surrogated gradient backpropagation methods can be found in (30]. Huh [20] introduce differentiable synapse and neuron models to

---

91
SNNs. Since spike trains are normaly sparse, gradients in SNNs are naturelly sparse compared to traditional ANNs, An sparse spiking gradient descent approach is introduced [4] speed up the backpropgation. To enable very deep spiking neural network, modified residual module with element-wise functions [4 is proposed to mimic directly gradient path of ResNet [17]. Rather than compute the approximated gradient, a neuron with extra inner state is introduced in [42 to estimate the gradient. Different from [42], the buffered neruon proposed in this literature utlizes extra inner state to compensate for inference error. Backpropagation trained SNNs show the potential of achieving ANN level performance. However, extra efforts are required to train the SNN properly, making it hard to train and limit its application on real world tasks.
92
93
94
95
96
96
99
100
Recently, combined methods 37 36] emerges to alleviate the drawbacks that conversion methods require larger timesteps to achieve competative performance. These methods often contain two steps: conversion and fine-tuning. The SNN converted in the first step is served as a weight initialization for further fine-tuning procedure using backpropagation. With the help of roughly mapped weights, several epochs of fine-tuning yield nearly loss-less performance with fewer timesteps [37.. How- ever, compared to conversion methods, extra efforts are required to fine-tune the SNN. Also, the backpropagation imposes additional limitations to the hybrid approach, which further restrains its application.
101
102
103
104
106
3 Methods
108
109
In this section, we first analyse the theoretical equivalency and gaps between a target SNN and an original ANN. Then, a quantized activation is proposed to minimize the coding error during the conversion. A buffered non-leaky IF neuron is presented reduce the inference error. Finally, we integrate the proposed conversion framework to further decrease the difference between SNN and ANN.
110
111
112
113
3.1
Theoretical Errors and Analysis
115
Following Rueckauer et al. 39], we here illustrate the equivalency between SNN and ANN and introduce two types of errors when using the conversion methods with rate-based coding. The target SNN shares a similar overall architecture with the original ANN except two significant discrepancies. First, neurons in the SNN communicate with each other by spike trains S(t) where S(t) E {0, 1} for each timestep E {1....,T) T), while the activation functions in ANN utilize real-value. T is the total timesteps. Second, a non-leaky integrate -nnraate-ndd-aae and-fire (IF) neuron is used instead of the ReLU activation in conventional ANN. The membrane function of the non-leaky IF neuron j can be described as:
117
119
120
121
(t(() = (t(t 1) Veh Σ m S(() St).
VEN,
122
where (t) is the membrane potential of the jth neuron and t is the timestep. N2 is the set of all input neurons in the last layer connecting to jth neuron. wfy is the weight and connection strength between the input ith neuron and current jth neuron. Ven is the threshold for the neuron to fire a signal. (t) and (t) are input and the resulting output spike trains of current neuron, respectively.
123
124
125
128
The firing signals are generated as follow:
1, if Vz((1) 2 Ven
S,(t)
= 0. otherwise.
(2)
Ns where (t(() (t(t- - +)+ + Vil No Sit) denotes the membrane potential before signal firing and potential reset operation.
128
129
Considering ANN-SNN conversion and rate-based coding, we can compute spike rate as Di= 8(1/TT for total timestep T. The goal of conversion process is to derive a function transfers the parameters of an ANN to the SNN as w = f(w") ) so that each neuron in the SNN produces a
T
131
132
spike rate r x a after T timesteps, where a is the corresponding activation value of original networks.
133
To this end, we can derive the spike count of the output spike train by summing Eqn. [ over total timesteps T:
134
T
T
VS(T) - V1(1)
Σ s,(t)
Σ E,N
St(t)
(3)
Vih
1=1
3

---

Potentall
08
Under-Are
R
Underfre spiketrain
Ovarfir membrane potential
02
RaLU
Over-ire spkketrai
02
0.4
0.6
0.8
12
Timestaps
(b) ReLU and neuron.
(a) Over-fire and under-fire phenomenons
Figure I: For each step of the membrane potential line the figure, the left horizontal line indicates V(t 1). The peak point indicates (*(1,, and the right horizontal line indicates (I)) which is transferred to the next step. under-fire spike train(orange).
over-fire spike train(purple). Σi= Sy(1) while Σ (1)
Σ S,(1) 11 while Σ (t)) 15.
135
where (1) is the initial membrane potential. For convenience, V₃(1) in this work without loss of generality. Eqn. gives the relation between the total spike count of current neuron and its input neurons' spike count besides the internal membrane potential.
138
137
138
For an ideal conversion, the spiking neuron intergrates membrane potential uniformly accross time and introduces error during inference. Under this ideal condition, the output spike count should fullfill the following equation,
140
T Σ ()) = max
Va ViA .
(4)
where Va = Veh EieN, " Σ (t) denotes the accumulated membrane potential. Please refer to the appedix for detail proof. Assuming Eqn. hoold we can derive the ideal output spike rate by 143 averaging Eqn. 4 oveer timesteps T:
r = max
Σ INN
wig mggri:0 - R2.
(5)
144
where Rt is defined as follow.
if otherwise.
(6)
0,
145 For the ideal ANN-SNN conversion situation, when R" is approaching zero, the formal expression of simple IF neuron becomes similar to the ReLU function processing information. Thus, the weight of a SNN can be transformed from its counterpart ANN directly. Since spike rate r* is bounded, A. Wa, in which A; is X
147
148
linear mapping function is utilized to convert ANN weights W = f(W") =
149
the scale factor for layer
150
3.1.1
Coding Error
151
As shown in Eqn. [5, R' can be viewed as a coding error term for the ideal conversion process during the ANN-SNN SNN weight mapping. As Va_
V. for Va > 0, the upper bound of the conversion error R < 1/T continuously decreases when timesteps set increasingly. Due to emors accumulated between layers, the converted SNNs with deep layers require large timesteps 7 to maintain high fidelity and performance (39 23 the other hand, the large timesteps impair the efficiency of the SNN. This phenomenon is also known accuracy-latency trade-off [9 31).
152
Ven Vis
153
155

---

157
3.1.2
Inference Error
158
Besides the conversion error, the assumption in Eqn. 4 be contravened easily due to essential nature of SNN neurons during the inference. Intuitively, the inference process of SNN quite dynamic and spiking neurons are unable to accumulate membraine potential uniformly accross timesteps as Eqn. 4 which leads to the divergence of overall output spike counts. We refer this type of error as inference error.
159
161
163
We conclude two situations where the Eqn. 4 is violated during inference: over-fire and under-fire phenomenons, Over-fire indicates the situation where number of total generated spikes is greater than number of expected spikes
184
E (t). Volatile membrane potential may cross the threshold (t) 2 ViA and cause spike S,(t) 1 that makes the number of generating spikes exceeding the number expected spikes. Conversely, under-fire refers to the situation where insufficient spikes have been generated Σ (t) < >i Sj (1). With inadequate spikes, under- fire neurons will have membrane potential large than Vsh the end of inference. Fig. shows an example of these two situations.
167
168
169
170
171
3.2
Quantized ReLU Activation
172
To deal with the coding error, we propose Quantized ReLU activation function in ANN training stage to accurately transform the weights to SNN with limited timesteps. According to Eqn 4 we can
173
174
reformulate Eqn. in different way:
Tmax
( EN
wefrri 0)
rf
175 Let v, = EN, rr, be the weighted sum of input spike rates in SNNs. Taking maximal spike rate 178 into consideration, IF neuron(Fig. lbb can be described with step function follow,
O, man TT
if S
IF(x.)
if 2 mar
(8)
otherwise.
The mar is the maximum spike rate of spike train, which is for conventional spike trains. The 178 proposed function is strictly equivalent to the IF neuron instead of increasing total timesteps T to
177
179 approximate the ReLU function with fine-grain step function. By mapping spike rate and activation
180 value a = Ar, the proposed activation function is derived through Eqn. 8:
Fa S if & 2 X
QReLU(xa A) = AIF(
La
A
(9)
otherwise.
We name the proposed activation function QReLU, since it can be seen quantized version of 182 the ReLU function. The approximate gradients are given by the PACT algorithm 6 originated from
STE 1:
(QReLU(Ea.
[0 ar <
(10)
0A
11 otherwise.
aQ Re (Fa, 2)
1,
0 < x. <A
(11)
Oza
0, otherwise.
The scale factor À here is also referred to as quantization boundary and is learned during the ANN training process. Guaranteed by the success of quantization techniques [ [), neural network
185
consisting of QReLU is capable of achieving similar performance compared to the original ANN. By
5

---

187
mapping weights from an ANN equiped with QReLU activation, the coding error R is eliminated in theory.
188
189
3.3
Buffered Non-leaky IF Neuron
190
For the inference error, observe that reduing over-fire problem alone can boost performance siginificantly based on our experiments. To further improve our conversion framework, we propose novel buffered non-leaky IF neuron to relieve the over-fire situation to compensate for the inference emror.
192
193
The proposed buffered neuron utilizes recurrent mechanism to regulate the spikes. In the proposed neuron, an additional buffered potential V is introduced as follows:
195
(t(() v -1 (t)
(12)
Similar to the membrane potential, the initial buffer potential (1) is set to zero for convenience. The spike generating process is described as
196
if (()) Vih 1,
5,(1
V(()) and (t 1) 2 Vaa
(13)
0, otherwise.
198
Intuitively, the proposed Buffered Non-leaky IF Neuron generates negative spikes to regulate the total spikes count for the inference errors. Thus, the over-fire problem can be alleviated. Please refer to appendix for detail proof. As same as ordinary IF neurons, only addition operation and threshold functions are involved in the proposed neuron. The inference efficiency preserved since the extra
199
200
201
202
power required by transmitting sign bit between neurons is negligible 45
203
3.4
Integration with Multiple Bits Spike Train
204
To further improve the performance the proposed conversion framework, we integrate shift- based multi-bits spike train into the proposed framework. Unlike the previous work [45) which compressed an existing SNN to speed up the inference, we extend the definition of the spike as
206
S(I) E (2 E N, 2n < Smar) facilitate the shift operation directly during the conversion. The
207
208
multi-bits spike train version of Equ 13 is as follow:
17() A(
)
if > Vn if V < 0 and vy( 2 Ve otherwise,
Ven
S,(t)
A(min
(r Ven.
(e )):
(14)
Ven
0,
209
where the adjust function A(-) is defined as
A(x) 2 log, (miin(,,Smer))
(15)
Multiple bits spike train carries more information than ubiquitous on-off spikes 5
22
33
47
211
However, expensive multiplications are required during the integration with non-uniform spikes. To avoid multiplication, we employ the log2 adjust function and shift operation (45) as alternative and achieve high energy efficiency. The adjust function can implemented with a conditional function.
213
214 For QReLU, as the maximal ratio coding is equal maximal spike counts, maz I Smaz the multi-bits version of Eqn. 9 can be described as follow:
Smaz /T

---

Table Classification performance compared other methods in CIFARIO and ImageNet-1k
SNN
SNN Top-1
Model
Arch
Top-
T
Model
Arch
A
y
ImageNet
CIFARIO
30
Rueckaser et
Cony, Lineur
90.85%
-1.06%
400
Sengupta
40.56%
Sengupta et
4D
ResNet20
87.46%
-1.64%
2500
Han st al.
16
VGGI6
73.09%
-0.40%
1096
et al. 26
VGG9
90.45%
100
Rathi ct 57
VGG16
65.19%
-4.16%
250
Wi et 43
Conv, 2 Linear
90.50%
90.53%
25
12
Deng and Gu
Bu et al.
VGG16
YGG16
50.97%
55,8078
23.32%
166.6%
16
Han
ResNet20
2048
Bu ct
VOG16
68.17%
-5.82%
32
Rathi et [37
ResNer20
92.22%
-0.93%
250
VGG16
Rathi and Roy
ResNei20
92.14%
-0.55%
25
Ours
VGG16
75.96%
+0.07%
16
Zheng al. 18
ResNet19
93.16%
et. (5x]
at.l.
VGG19
Cony, 2 Lincar
93 33.90%
92.48%
-0.23%
-0.08%
300
Sengupta
Han
40
ResNet34
ResNet34
65.47%
65.47%
-5. %
4196
2000
Deng and Gu
X
ResNei20
92.41%
0.09%
16
Zhong et
et
48
ResNet34
63.725%
Ding
PreActResNec18
Li 23
ResNer34
64.11%
-11.12%
32.
Bu al.
8
ResNet18 94.82%
-1.22%
Li
ResNet34
Bu
ResNet18
95.92%
-0.12%
al
ResNet34
Ours
ResNet18
95.13%
40.17%
Ru et al.
ResNet34
69.175
-4,98
32
Ours
ResNet18
95.14%
-0.04%
16
Ours
ResNet34
73.20%
-0.52%
ResNet34
0.08%
Table 2: Detection performance
Table SNN computation efficiency.
SNN AP50
Atchitecture (timesteps)
Operation ratin
SNNANN
Model
Arch
T
Methods
energy ratio
PASCAL VOC
CIFARIO
Kim al.
tiny-yoln
51.83%
-1.18%
5000
Rathi and Roy 36
ResNer20(25)
2.55
0.51
Ours
tiny-yolo
65.20%
-0.13%
16
Ours Ours
ResNet18(8)
0.92
0.19
Ours
tony-yolo
65.73%
+0.06%
32
8e5Net18(16)
COCO
ImageNet
Kim
tiny-yolo
25.66%
00.88%
5000
Rathi and Roy 36
VGG16(25)
1.62
0.32
Ours
tiny-yolo
38.6%
-1.1%
Ours 16
VGG16(8)
1.92
0.39
Ours
thny-yolo
39.2%
-0.6%
32
Ours
VGG16(16)
3.03
0.62
0,
if v S if F. 2 A otherwise.
QRa:U(r..:')) X
(16)
216
Here the X Xmmaz the quantization boundary and P TSman is the representation power. As implied by the definition of the representation power, even the minimal shift extension with Smaz can maintain the same representation power with half timesteps. This phenomenon named larencyh trade-off.
217
218
4 Experiments
220
In this section, conduct experiments both classification and detection tasks to demonstrate the efficiency and effectiveness of the proposed method. Since the equivalency of features and spike rates universal throughout the entire network. real-valued input is directly utilized in the converted SNN. We select a unity threshold Vn, = 1 for all our experiments. Max spike value Smar is used for all experiments if not stated otherwise. Other implementation details can be found in the appendix.
222
224
226
227
4.1
Visual Object Recognition and Detection Performance
228
test our proposed High Performance Conversion framework (HPC) on visual object recognition 17 like architecture is utilized as backbone networks to con-
and detection tasks. A ResNet18
230
duct experiments on the CIFAR10 25 dataset and a VGG16 41 like network is trained on the
ImageNet [7] dataset. Table I reports the classification accuracy of the proposed HPC compared

---

VieG
0.74
0.72
1o ANN socuracy immestes ++ -NN accu ocy limestops
0.2
0.08
ANN accu ocy simesleps
Ln SNN sccursccy timeslapa
0.8
-2
Distribution of spike values.
Operation ratio
Figure 2: Left: Operation ratio and performance with ImageNet dataset under 16 timesteps. Data points with operation ratio converted SNN and its corresbonding ANN. Right: Distribution of spike values.
with other methods. Top-1 accuracy is used to evaluate the performance. Since the performance of converted SNN closely related to the ANN baseline, performance drop is also reported. As
233
234
illustrated Table [ the proposed HPC can perform nearly lossless conversion while maintaining
235
low inference latency.
As general method, we also test our HPC with tiny-yolo 38 structure on both PASCAL VOC 13 and MS COCO [28] datasets. Table summarizes the AP50 performance the proposed method. We see that our proposed HPC can achieve state-of of-the-art conversion performance within 32 timesteps, which is over 100 times faster than the previous SNN detection work (23]
237
239
4.2
Energy Efficiency
241
To evaluate the energy efficiency of the proposed method, operation ratio is adopted to measure the overall efficiency described in Equ. [7 In contrast actual energy consumption which may vary
242
243
between hardware devices (32), counting-bres based metric [36] is straightforward and representative for different methods. For fair comparison, integration operations triggered by spikes for both membrane potential and buffer potential are counted in our experiments.
245
operation ratio (addition ops in SNN)
(17)
#(flops in ANN)
246
Operation ratios, as well as estimated energy consumption ratio, are summarized in Table 3 Here we roughly estimate that multiplication consumes 4 times more energy than addition according to Horowitz [18. The results on the CIFARIO dataset show a strong correlation between timesteps and energy efficiency. To further study energy efficiency, we test the relationship between operation ratio and networks performance. As depicted in Fig, 2a under the same timesteps setting, the spike ratio continuously decreases with the overall performance. Our experiments confirm general relationship between inference energy and performance, which holds even under constant latency.
247
248
249
250
252
Fig. 2b depicts the distributions of spike values in classification experiments. Without loss of gener- ality, we conclude that only 20% of the spikes require shift operation. As measured by Gudovskiy and Rigazio [15], the shift operation only cost 10% of energy consumed by addition operation. Thus, bits spike only use 2% extra energy, which is negligible.
254
255
256
4.3
Ablation Study
258
To reveal the effectiveness of the proposed framework, we conduct ablation studies on both clas- sification and detection tasks. Specifically, we remove the quantized training technique and the buffered IF neuron sequentially and compare the results. The baseline ANN is trained and fixed for
259
260
261
experiments without QReLU component, meanwhile quantized ANN networks are trained separately for different timestep configurations. Without the QReLU, the scale factor is calculated using the percentile function with the parameter empirically set to 99.0. Channel-wise normalization is utilized to ensure competitive performances. Since conventional converted SNN requires large timesteps to
263

---

66.5
76
Accuracy
Top-1 Acsuacy
04.5
72
60.5
SNN wi buufered
ANN
SNNN buffered
ANN
5. SNN wia buffored
SNN butfered IF
C-ANN
X C- A-NNN
02.5
butfered
6. 0--0-500 SNN buffered IF
16
125
256
32
64
128
256
Timestops
Timesteps
(a) CIFARIO
(b) ImageNet
84
AP50
AP50
82
60
+SNN
+-SNN
SNN
SNN
O.
SNN buffered
buffered
Timesteps
Timesteps
(c) VOC
(d) COCO
Figure 3: Ablation study of proposed techniques. Q-ANN denotes ANNs trained with QReLU, Q-SNN denotes SNNs converted from ANN trained with QReLU. SNN w/ buffered means that buffered non-lcaky integrated and fire neuron are used. SNN w/o buffered IF are conventional SNNs with out the proposed neuron.
285
achieve plausible performance, we only report those performances with 32 or more timesteps for the CIFARIO dataset and for others.
286
267
Fig. validates the effectiveness of the proposed techniques. Compared to conventional SNNs,
the SNN integrates with buffered neurons only requires a quarter of timesteps reach the same performance. As the buffered neuron mitigates only the over-fire problem, we conclude that the
269
270
over-fire situation the primary cause of performance degradation during inference. The introduction of quantized training further enhances the performance of converted SNN under limited timesteps.
271
272
Similar trends can be observed across experiments.
5
Conclusion and Limitation
274
In this paper, we proposed a novel High Performance Conversion (HPC) method to simultaneously achieve high-performance SNN inference with low latency. Efforts had been made to reduce both coding errors and often overlooked inference errors. A novel QReLU activation was proposed to eliminates the conversion error. Meanwhile, buffered non-leaky neuron was designed to mitigate the over-fire problem and boost the inference performance while maintaining its simplicity and efficiency. For the first time, efficient SNN converted without extra fine-tuning revealed its capability to achieve state-of-the-a performances in nontrivial vision tasks.
275
278
277
279
However, there are limitations. Since actual hardware is involved, the efficiency is estimated using energy consumed by simple operations. As computing platform evolves rapidly, it is beyond our reach accomplish real neuromorphic hardware which can be researched in further work.
282
283
284
For furture work, we believe further study on novel structures such as attention mechanisms SE 19 and transformer [12] extend the current framework and may lead better understanding of the underlying perception mechanism. Moreover, applications SNN on video-like input that achieves ANN level performance with great efficiency another promising direction.
286
References
288
289
[1] Y. Bengio, N. Lonard, and A. Courville, Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, 2013.

---

291
and [2] S. M. Bohte, J. N. Kok, and H. La Poutré. Emorrbacckroouugaton in temporally encoded networks of spiking neurons. Neurocomputiing 48(1):17 37, 2002. ISSN 0925-2312. doi:
292 10.1016/80925- 231 J00658-0.
293
294
[3] T. Bu, W. Fang, J. Ding, P. DAI, Z. Yu, and T. Huang. Optimal ANN-SNN conversion for high-accuracy and allra-lowllaten: spiking neural networks. In International Conference on Learning Representations, 2022.
295
296
297
[4] Y. Cao, Y. Chen, and D. Khosla. Spiking deep convolutional neural networks for energy- mnergy-efficient object recognition. International Journal of Computer Vision, 113(1):54-66, May 2015. ISSN 1573-1405. doi: 10.1007/s1 (11007/s1163--00
298
299
300
[5] R. Chen, H. Ma, S. Xie, P. Guo, P. Li, and D. Wang. Fast and efficient deep sparse multi-strength spiking neural networks with dynamic pruning. In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2018, doi: 10. 10.1109/ICCNN200 201 8, 8.889339.
301
302
303
[6] J. Choi, Z. Wang. S. Venkataramani, P. I.-J Chuang, V. Srinivasan, and K. Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. CoRR, 2018.
304
[7] J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009, doi: 10.1 10.1109/CVPRR... 2009555684209
306
307
[8] S. Deng and S. Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Learning Representations, 2021.
309
310
[9] P. U. Diehl, D. Neil, J. Binas, M. Cook, S. Liu, and M. Pfeiffer. Fass-clas,,,ying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2015. doi: 10.1 (009///NNN.01...280686..
312
313
[10] Ding, Z. Yu, Y. Tian, and T. Huang. Optimal ann-snn conversion for fast and accurate inference in deep spiking neural networks. In IJCAI, 2021. doi: 10.24963/ijci..0222 2021/321.
[1I] S. Dong, T. Huang, and Y. Tian. Spike camera and its coding methods. In Data Compression Conference (DCC), 2017.
316
317
[12] A. Dosovitskiy and et al. An image worth 16x16 words: Transformers for image recognition at scale. In ICIR, 2021.
319
[13] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 2010. doi: 10. 10.1007/s1 1233-09--0277
320
322
[14] W. Fang, Z. Yu, Y. Chen, T. Huang, T. Masquelier, and Y. Tian. Deep residual learning in
323
spiking neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
324
325
[15] D. Gudovskiy and L. Rigazio. Shiftenn: Generalized low-precision architecture for inference of convolutional neural networks. CoRR, 2017.
328
[16] B. Han, G. Srinivasan, and K. Roy. Rmp-snn: Residual membrane potential neuron for enabling
328
deeper high-accuracy and low-latency spiking neural network. In CVPR, 2020.
329
17 K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
330
331
332
[18] M. Horowitz. 1.1 computing's s energy problem (and what we can do about it). In 2014 IEEE International Solid- Circuits Conference Digest of Technical Papers (ISSCC), pages 10-14, 2014. doi: 10. 109//SSCC..0144.777777
334
335
[19] Hu, L. Shen, and G. Sun. Squeezz-and-eexiiattin networks. In CVPR, pages 7132-7141, 2018. doi: 10. 109///VP.2018..007444
[20] D. Huh and T. J. Sejnowski. Gradient descent for spiking neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, pages 1433-1443, Curran Associates, Inc., 2018.
338
340
341
[21] Y. Jin, W. Zhang, and P. Li. Hybrid macro/miero level backpropagation for training deep spiking neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N.
10

---

343
R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, pages 7005-7015. Curran Associates, Inc., 2018.
344
345
[22] J. Kim, H. Kim, S. Huh, J. Lee, and K. Choi. Deep neural networks with weighted spikes. Review of Economic Dynamics, 311:373-380 Oct. 2018. ISSN 1094-2025. doi: 10.1016/j. neucom.2018.05.
347
Funding Information: This work was supported by the KIST Institutional Program (Project No. 2E27330-17-PO26)
348
349
[23] S. Kim, S. Park, B. Na, and S. Yoon. Spiking-yolo- Spiking neural network for energy-efficient object detection. AAAI, 2020.
350
[24] Y. Kim and P. Panda. Revisiting batch normalization for training low-latency deep spiking
352
neural networks from scratch, 2020.
353
[25] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Tech Report, 2009.
354
355
[26]
C. Lee, S. S. Sarwar, P. Panda, G. Srinivasan, and K. Roy. Enabling spike-based backpropagation for training deep neural network architectures. Frontiers in Neuroscience, 14, Feb 2020. ISSN 1662-453X. doi: 10.3389/fiins.202000000
357
358
[27] Y. Li, S. Deng, X. Dong, R. Gong, and S. Gu. A free lunch from ann: Towards efficient, accurate
359
spiking neural networks calibration. In ICML, pages 63 6616-63, 2021.
360
[28] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. ISBN 978-3-319- 1-6602-...
[29] H. Mostafa. Supervised learning based temporal coding in spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 29(7)33227-3235, 2018. doi: 10. 109/
363
364
TNNIS,
385
[30] E. O. Neftei, H. Mostafa, and F. Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-6) 2019. doi: 10.1 10.1109//..2019.99911
366
367
368
[3]] D. Neil, M. Pfeiffer, and S.-C. Liu. Learning to be efficient: Algorithms for training low-latens low-compute deep spiking neural networks. In Proceedings of ACM Symposium on Applied Computing, 2016.
370
371
[32] D. E. Nikonov and I. A. Young. Benchmarking physical performance of neural inference circuits. CoRR, 2019.
372
373 [33] S. Park, S. Kim, H. Choe, and S. Yoon. Fast and efficient information transmission with burst spikes in deep spiking neural networks. In 2019 56th ACM/IEEE Design Automation Conference (DAC), pages 1-6, 2019.
375
378
[34] N. Perez-Nieves and D. Goodman. Sparse spiking gradient descent. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 11795- 11995-11000 11808. Curran Associates, Inc., 2021.
377
378
[35] J. A. Pérez-Carrasco, B. Zhao, C. Serrano, B. Acha, T.
S. Chen, and B. Bararas-B Mapping from frame-driven to frame-free event-driven vision systems by low-rate rate coding and coincidence propessingaapllicttion to feedforward convnets. IEEE Transactions on Pattern Analysis and Machine Intelligence,
380
382
2013. doi:
10.1 10.1109/PAM...013..7.
384
[36] N. Rathi and K. Roy. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. CoRR, 2020.
385
[37] N. Rathi, G. Srinivasan, P. Panda, and K. Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations, 2020.
387
389
[38] Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In CVPR, June 2016.
390
[39] B. Rueckauer, I. Lungu, Y. Hu, M. Pfeiffer, and S. Liu. Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in Neuroscience, 2017. ISSN 1662-453X. doi: 1t.3389/mins..2117.006
382

---

394
[40] A. Sengupta, Y. Ye. R. Wang, C. Liu, and K. Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in Neuroscience, 13:95, 2019. ISSN 1662-453X. doi:
395
10.
387
[41] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, 2015.
398
389
400
[42] J. C. Thiele, O. Bichler, and A. Dupret. Spikegrad: An ann-equivalent computation model for implementing backpropagation with spikes. In International Conference Learning Representations, 2020.
402
403
[43] Y. Wu, L. Deng, G. Li, Zhu, Y. Xic, and L. Shi. Direct training for spiking neural networks: Faster, larger, better. Proceedings of the AAAI Conference Artificial Intelligence, 33(01):
405
1311-1318, Jul. 2019. doi: 10. 10.1609/anai.v33303 v33001.33011111111111
406
[44] Xing. Y. Yuan, H. Huo, and T. Fang. Homeostasis- based enn-to-snn conversion of inception and residual architectures. In T. Gedeon, K. W. Wong, and M. Lee, editors, Neural Information Processing, pages 173-184, Cham, 2019. Springer International Publishing. ISBN 978-3-030- 36718-3.
407
408
410
[45] C. Xu, W. Zhang, Y. Liu, and P. Li. Boosting throughput and efficiency of hardware spiking neural accelerators using time compression supporting multiple spike codes. Frontiers in Neuroscience, 14:104, 2020. ISSN 1662-453X. doi: 2..3389fiins.202200000
411
412
413
[46] Z. Yan, J. Zhou, and W.FF Wong. Near lossless transfer learning for spiking neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):10577-1058 May 2021.
414
[47] Q Yu, C. Ma, S. Song, G. Zhang. J. Dang, and K. C. Tan. Constructing accurate and efficient deep spiking neural networks with
416
and augmented schemes, 2020.
417
[48] H. Zheng. Y. Wu, L. Deng. Y. Hu, and G. Li. Going deeper with directly-trained larger spiking neural networks. Proceedings of the AAAI Conference Artificial Intelligence, 35(12): 11062-11070, May 2021.
419
Checklist
420
421
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes]. [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
422
423
424
425
Did you include the license to the code and datasets? [Yes] See Section ??.
428
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
427
Did you include the license the code and datasets? [N/A]
429
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
430
432
1. For all authors...
433
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
435
(b) Did you describe the limitations of your work? [Yes]
436
(c) Did you discuss any potential negative societal impacts of your work? [No]
437
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
440
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
12

---

442
3. If you ran experiments...
443
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] Source code will be released later.
445
446
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Training details can be found in the appendix.
447
448
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Performance stability is reported in the appendix.
450
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Training details can be found in the appendix.
451
452
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] Popular datasets used.
456
(c) Did you include any new assets either in the supplemental material or as URL? [No]
457
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
460
461
5. If you used crowdsourcing or conducted research with human subjects...
462
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
463
464
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
465
466
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
467
13


Reviews:
[{"review_id": "i0qs6h-jrpy", "reviewer": "Reviewer_HvVX", "summary": "The paper describes a method to convert ANN with quantized ReLu activation functions into a spiking network emitting non-binary spike trains. The theory extends previous work on rate-based conversion from relu ANN to spiking networks: the authors characterize conversion error into sub-types. Beside the coding error (quantization error) they categorize the conversion error as under- and over-spike error and suggest methods to reduce them. One idea is to send signed spikes to signal when the accumulated inputs cross the threshold downward (effectively cancelling the previously transmitted spike and solving the over-spike error). The second is to transmit non-binary spikes with a spike amplitude quantized on a scale of powers of 2. It yields non-binary spiking networks with high-performance (for example 75% top1 accuracy on ImageNet  which is on par with current popular VGG or ResNet baselines). ", "questions": "I wrote suggestions and questions in the Strengths and weaknesses section.\n\n## Ethical concern:\nThe implementation of Yolo with DVS cameras is likely to converge towards low-consumption surveillance applications which raises ethical concerns. Although there are many applications of Yolo with DVS cameras which would not necessary lead there, the current Figure 4 demonstrates how to separate people from objects so it depicts quite explicitly how this technology can be applied for surveillance without mentioning the ethical concerns. There are many ways to reduce this negative impact, here are some concrete options: choose a different example in Figure 4 (and Figures from Section B in appendix) and publish the code so that independent researchers can understand better how to avoid misuses of the technology.", "limitations": "I described limitations in the weaknesses section.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n(1) The characterization of over- and under-fire conversion error is relevant and useful to design better conversion schemes of spiking networks to non-spiking networks.\n\n(2) The performance is very high and it bridges the gap between spiking networks and regular quantized neural networks.\n\n(3) The reduction of  over-fire error with negative spikes seems to have a large positive-effect on the accuracy and its description is understandable.\n\n## Weaknesses:\n(1) The paper is poorly written: most importantly the main text lacks of mathematical rigor and uses confusing notations. For instance in section 3.1, it is not clear what is a definition of a notation for $S^*(t)$. The closest to a definition would be equation (4) but this defines the sum over $t$ and not $S^*(t)$ per-se which is confusing. So the concepts of \"ideal conversion\" is defined in a cumbersome and complicated way although it seems quite important. Also the definitions of $r_i$,  $S^*(t)$ and $r^*_i$ are unclear and maybe redundant. Because of this I did not understand why a proof mentioned at line 135 is necessary, I read through the proof in appendix anyway and it seems to rely on some wrong arguments: for instance it is written in the appendix \"For $V_a$ > 0 we infer $C$ > 0\" but C depends on $t$ although $V_a$ is a quantity integrated overtime so $V_a > 0$ cannot imply $C > 0$. This statement is either wrong or unclear, although it is probably a minor argument for the proof, it shows the general lack of rigor in the theoretical section. I think this paper will be much better if the theory section is re-written, mostly sections 3.1, 3.4 and this proof which are hard to read and unclear at the moment. \n\n(2) The implementation is not clear and the code is not provided, this is a real problem because details are missing to fully understand the algorithm. Concretely I see two possible ways of implementing this model but I do not know which one is implemented. It is not the same if the layer 2 fires spikes S(t) right after layer 1 S(t) or whether layer 2 waits the end of the computation S(T) of layer 1 before computing S(1). \n- First possibility: the code can be something rather simple like replacing the ReLU or QReLU with a spiking version of it which receives the integrated sums of spikes and outputs the integrated sum of spikes, if this is what is implemented it should be acknowledged clearly and the the theoretical section should clearly acknowledge that the latency accumulates at each layer. Also, if this is the implementation the activity is buffered at each layer so the necessity of the additional spike buffer is probably unnecessary.\n\n- Second possibility: the model is implemented as a sort of recurrent neural network where the spikes S(t) at all layers are computed before sending them to S(t+1). However, training a recurrent network in this way would require to use back propagation through time which typically multiply the memory consumption of the algorithm by a factor T and raises a clear drawback to train large ImageNet or VOC models on GPUs. Considering the trade off that is therefore necessary between T and batch size, it should at least be debated somewhere.\n\nIn any case, even if the implementation is done in another way which I did not describe here. This should be acknowledged and described clearly. Ideally the code should be shared so that any ambiguity can be lifted by reading the code directly.\n\n(3) I think that the theoretical statement that the negative spike can correct the over-fire problem is only true with a single hidden layer. A clear counter example with two layers can be designed where a neuron in hidden layer 1 spikes positively at time step 1 (S(1)=1) which engages non linear computations in the hidden layer 2 at the same time step, and then spikes negatively at time step 2 (S(2)=-1) leading to a QReLU output of 0. Assuming the second possible implementation that I described in the paragraph above it is not guaranteed that the accumulated effects of the negative and positive spikes cancel out because of the non-linearity in layer 2. Unfortunately I cannot be sure whether this can occur in the simulation because of the missing information about the implementation. If it does occur, the imperfections of the negative spike solution of over-fire errors should be acknowledged in the main text.\n\n(4) By introducing the concept of non-binary spike trains the conceptual boundary between spiking neurons and quantized ANN is blurred. This should be acknowledged clearly and the remaining difference and benefits of non-binary SNN and Q-ANN should be discussed. About their similarities, one could acknowledge:\n- This conversion method requires a pre-training with a well chosen QReLU activation function so it requires the similar training algorithm as a quantized ANN (say a binary feed-forward neural network trained with straight through estimators). \n- Certainly most current hardware capable of simulating a non-binary SNN could also implement a Q-ANN. Which raises questions about the benefits of the spiking solution.\n\nAbout there differences, I would love to see some quantitative comparisons, for instance:\n- Can one add the Q-ANN into the energy ratio plot ? (not sure how to count integer multiplications into the plot relying on the number of additions, but this was done somehow to integrate spikes with amplitude larger than one in this plot already so why not?). If not, finding common metric to estimate to energy consumption of Q-ANN and non-spiking network would be great in this context but it might be left for future work.\n\n(5) Details of implementation about the Yolo architecture are not provided. It probably requires to implement non-binary communications on some unusual blocks like residual blocks. Although a paper is cited for this purpose it is necessary to explain whether something similar or different is used. I also think that this can be problematic in regards to the theory section depending on the implementation, so a clear description of the implementation of residual block using the notations and concepts from the theory section would be strongly welcome. Again, one option is to publish the code.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"]}, "extra_text": {}}, {"review_id": "EY5eroqewE", "reviewer": "Reviewer_UtSS", "summary": "This paper proposes to improve the ANN-to-SNN conversion method by several techniques including QReLU activation, buffered non-leaky-integrate-and-fire neuron, and multiple bits spike train. Experiments on both classification and detection tasks demonstrate strong performance.", "questions": "See Weakness above.", "limitations": "There is no potential negative societal impact.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper not only considers the common classification task but also demonstrates strong results on the detection task.\n\nWeakness:\n1. The novelty of this paper is limited. The conversion error has already been theoretically analyzed in the literature [1,2,3,4]. Particularly, the inference error in this paper is studied as “unevenness error” in [3] and “deviation error” in [4] (while [4] is very recent and available after the submission deadline, [3] is earlier and should be noted). The quantized ReLU activation in this paper has been adopted in [3] as “quantization clip-floor activation”. The proposed buffered non-leaky IF neuron has been similarly adopted in the literature [5]. And the multiple bits spike train is inspired by existing literature except that this paper realizes it in a different way. So it looks like this paper simply combines multiple existing techniques and the novelty is very limited.\n\n2. It is unclear how the proposed buffered neuron model and multiple bits spike train can be implemented on neuromorphic hardware. The compatibility with hardware should be discussed.\n\n3. This paper does not use common neuron models and tries to enforce the inference of SNNs to be the same as quantized ANNs by proposing new neuron models. Then what is the advantage compared with directly applying well-established quantized ANNs that can also have hardware-efficient implementation? There should be some discussion and comparison. \n\n[1] Deng, S., & Gu, S. (2020, September). Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks. In International Conference on Learning Representations.\n\n[2] Li, Y., Deng, S., Dong, X., Gong, R., & Gu, S. (2021, July). A free lunch from ANN: Towards efficient, accurate spiking neural networks calibration. In International Conference on Machine Learning (pp. 6316-6325). PMLR.\n\n[3] Bu, T., Fang, W., Ding, J., Dai, P., Yu, Z., & Huang, T. (2021, September). Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks. In International Conference on Learning Representations.\n\n[4] Meng, Q., Yan, S., Xiao, M., Wang, Y., Lin, Z., & Luo, Z. Q. (2022). Training much deeper spiking neural networks with a small number of time-steps. Neural Networks.\n\n[5] Thiele, J. C., Bichler, O., & Dupret, A. (2019, September). SpikeGrad: An ANN-equivalent Computation Model for Implementing Backpropagation with Spikes. In International Conference on Learning Representations.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4GwtVo5QYb", "reviewer": "Reviewer_PgR1", "summary": "The authors present a well-designed ANN-to-SNN conversion method. They claimed to discover and discuss the overlooked conversion errors: coding error and inference error. A QReLU activation for the training phase and a buffered none-leaky neuron for the inference phase were proposed. The authors gave good results on classification and detection tasks. To some extent, the authors did not conduct detailed literature research and over-claimed their contributions.", "questions": "1.The idea of QReLU is similar to paper [1] & [2]. Please demonstrate the differences between them. Comparisons of the performance should also be included. \n2.V_(T_o) < 0 is equivalent to the over-fired case. My question is: Does this condition restrict that there is only one negative pulse before T_o? And, given that the final membrane potential may not exceed Vth, how can the effects of the over-fired situation be mitigated?\n3.I don't understand how the experiment of spike camera detection is implemented. Please specify more details, including how long the sequence you input to the SNN and how the data used was obtained.\n4. What does the time step of ANN in Fig.2 mean?\n5. Line 205 says that the computational-expensive multiplication operation needs to be avoided. Please explain how log2 operation in the newly designed neuron will be implemented in hardware?\n6. Line 197 says \"we integrate shift-based multi-bits spike\", I didn't find a statement about \"shift\" in the paper. What is \"shift\"?\n7. There is no ablation study on “QReLU” and “Multiple Bits Spike Train” found in the main paper.\n8. If λ is learnable, does T need to be pre-set in QReLU? Can the performance of different time steps be provided under the same model?\n9. As the aim of this work is to get loss-free conversion, if the buffered potentials are the integration of all inputs across time, why not just output the buffered potential to get loss-free conversion.", "limitations": "This paper does not do enough quantitative analysis and explanation of the proposed negative spike method. Neurons that generate negative spikes belong to the special category of spiking neurons.\nSince the authors use a new type of neuron, and the new neuron is obviously more complex than the traditional neuron, the authors should include more analysis about the power consumption and hardware implementation of this neuron.", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. This paper is easy to follow. The authors introduce the QReLU function, buffered none-leaky neurons to promote accuracy and integrate the shift-based multi-bits spike train to increase representation efficiency.\nWeakness:\n1. The major weakness is the novelty. The main contributions are the error identification and a quantization ReLU activation, both of which have been proposed in some previous work [1], [2]. Also, the negative spike has been proposed by spiking-yolo. Please clarify the difference, or is this work a combination of these techniques?\n2. Experiments are not enough to demonstrate the effectiveness of the proposed method. For example, it is mentioned in line 177 that λ is learnable, according to Eqn. 9. Please clarify whether T needs to be set. If T needs to be set, what is it equal to?\n3. The authors use a newly proposed neuron model which may not be supported by the current hardware, which will diminish the significance of this paper.\n4. There are some ambiguous expressions in the paper that are hard to understand. \n\n[1] Yan, Z., Zhou, J., & Wong, W. F. (2021). Near lossless transfer learning for spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence.\n[2] Bu, T., Fang, W., Ding, J., Dai, P., Yu, Z., & Huang, T. (2022). Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks. In International Conference on Learning Representations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject