Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks
Authors: Shikuang Deng
Shi Gu
Venue: ICLR 2021 Poster (2021)
Keywords: spiking neural network
weight balance
second-order approximation

Abstract: Spiking neural networks (SNNs) are biology-inspired artificial neural networks (ANNs) that comprise of spiking neurons to process asynchronous discrete signals. While more efficient in power consumption and inference speed on the neuromorphic hardware, SNNs are usually difficult to train directly from scratch with spikes due to the discreteness. As an alternative, many efforts have been devoted to converting conventional ANNs into SNNs by copying the weights from ANNs and adjusting the spiking threshold potential of neurons in SNNs. Researchers have designed new SNN architectures and conversion algorithms to diminish the conversion error. However, an effective conversion should address the difference between the SNN and ANN architectures with an efficient approximation of the loss function, which is missing in the field. In this work, we analyze the conversion error by recursive reduction to layer-wise summation and propose a novel strategic pipeline that transfers the weights to the target SNN by combining threshold balance and soft-reset mechanisms. This pipeline enables almost no accuracy loss between the converted SNNs and conventional ANNs with only $\sim1/10$ of the typical SNN simulation time. Our method is promising to get implanted onto embedded platforms with better support of SNNs with limited energy and memory. Codes are available at https://github.com/Jackn0/snn_optimal_conversion_pipeline.

Published as a conference paper at ICLR 2021
OPTIMAL CONVERSION OF CONVENTIONAL ARTIFI- CIAL NEURAL NETWORKS TO SPIKING NEURAL NET- WORKS
Shikuang Deng' & Shi Gu School of Computer Science and Engineering University of Electronic Science and Technology of China dengsk1 1 90s st.. uesta. edu. cn, gus@ueste edu. cn
ABSTRACT
Spiking neural networks (SNNs) are biology-inspired artificial neural networks (ANNs) that comprise of spiking neurons to process asynchronous discrete sig- nals. While more efficient in power consumption and inference speed on the neuromorphic hardware, SNNs are usually difficult to train directly from scratch with spikes due to the discreteness. As alternative, many efforts have been devoted to converting conventional ANNs into SNNs by copying the weights from ANNs and adjusting the spiking threshold potential of neurons in SNNs. Researchers have designed new SNN architectures and conversion algorithm- to diminish the conversion error. However, effective conversion should address the difference between the SNN and ANN architectures with an effi- cient approximation of the loss function, which missing in the field. In this work, we analyze the conversion error by recursive reduction to layer-wise sum- mation and propose a novel strategic pipeline that transfers the weights the target SNN by combining threshold balance and soft-reset mechanisms. This pipeline enables almost no accuracy loss between the converted SNNs and con- ventional ANNs with only ~ 1/10 of the typical SNN simulation time. Our method is promising to get implanted onto embedded platforms with better sup- port of SNNs with limited energy and memory. Codes are available at http- ss/ggituub.com//ccd//ssm optimal conversion pipeline
INTRODUCTION
Spiking neural networks (SNNs) are proposed imitate the biological neural networks (Hodgkin & Huxley 1952a McCulloch & Pitts 1943) with artificial neural models that simulate biological neu- (Hodgkin & Huxley 1952b). Izhikevich (Izhikevich] 2003), and sasonnte-and-Fing (Izhikevich] 2001) models. The most widely used neuron model for SNN the Integrate-and-Fire (IF) model (Barbi et al. 2003] Liu Wang 200T), where neuron in the network emits spike only when the accumulated input exceeds the threshold voltage. This setting makes SNNs more similar to biological neural networks.
ron activity, such as Goodgkin-Huxbee
The past two decades have witnessed the success of conventional artificial neural networks (named as ANNs for the ease of comparison with SNNs), especially with the development of convolutional neural networks including AlexNet (Krizhevsky et al. 2012). VGG (Simonyan & Zisserman 2014) and ResNet (He et al. (2116). However, this success highly depends on the digital transmission of information in high precision and requires large amount of energy and memory. So the traditional ANNs are infeasible to deploy onto embedded platforms with limited energy and memory. Distinct from conventional ANNs, SNNs event-driven with spiking signals, thus more efficient in the energy and memory consumption on embedded platforms (Roy et al. 2019). By far, SNNs have been implemented for image (Acciarito et al. 2017 Diehl & Cook 2014 Youselzadeh al. 2017)
and voice (Pei et al. 2019 recognition.
X Corresponding author

---

Published as a conference paper at ICLR 2021
Although potentially more efficient, current SNNs have their own intrinsic disadvantages in training due to the discontinuity of spikes. Two promising methods of supervised learning are backpropa- gation with surrogate gradient and weight conversion from ANNs. The first routine implants ANNs onto SNN platforms by realizing the surrogate gradient with the customized activation function (Wu eet al. 2018). This method can train SNNs with close or even better performance than the conven- tional ANNs on some small and moderate datasets (Shrestha & Orchard 2018 Wu et al., 2019 Zhang& & Li 2020; Thiele et al. 2 However, the training procedure requires lot of time and memory and suffers from the difficulty in convergence for large networks such as VGG and ResNet.
The second routine to convert ANÑS to SNNs by co-training a source ANN for the target SNN that adopts the IF model and soft-reset mechanism (Rueckauer et al. 2016 Han et al. (020). When a neuron spikes, its membrane potential will decrease by the amount of threshold voltage instead of turning into the resting potential of a fixed value. A limitation of this mechanism is that it discretizes the numerical input information equally for the neurons on the same layer ignorant of the variation in activation frequencies for different neurons. As consequence, some neurons transmit information with short simulation sequences. Thus the converted SNNs usually require huge simulation length to achieve high accuracy (Deng al.] 2020) due to the trade-off between simulation length and accuracy (Rueckauer et al.) (017). This dilemma can be partially relieved by applying the threshold balance on the channel level Kim al. (2019) and adjusting threshold values according to the input and output frequencies (Han et al.) 20200] Han & Roy 2020). However, as far as we know, it remains unelear how the gap between ANN and SNN formulates and how the simulation length and voltage threshold affect the conversion loss from layers to the whole network.
difficult to
In addition, the accuracy of converted SNN is not satisfactory when the simulation length as short as tens.
Compared to the previous methods that focus on either optimizing the conversion process or modi- fying the SNN structures, we theoretically analyze the conversion error from the perspective of acti- vation values and propose a conversion strategy that directly modifies the ReLU activation function in the source ANN to approximate the spiking frequency in the target SNN based the constructed error form. Our main contributions are summarized as follows:
We theoretically analyze the conversion procedure and derive the conversion loss that can be optimized laser-
We propose a conversion algorithm that effectively controls the difference of activation values between the source ANN and the target SNN with a much shorter simulation length than existing works.
We both theoretically and experimentally demonstrate the effectiveness of the proposed algorithm and discuss its potential extension to other problems.
2 PRELIMINARIES
Our conversion pipeline exploits the threshold balancing mechanism (Dichl et al. 2015 Sengupta al. 2018) between ANN and SNN with modified ReLU function on the source ANN reduce the consequential conversion error [Fig[A). A). The modification on regular ReLU function consists of thresholding the maximum activation and shifting the turning point. The thresholding operation suppresses the excessive activation values so that all neurons could be activated within shorter simulation time. The shift operation compensates the deficit of the output frequency caused by the floor rounding when converting activation values to output frequencies.
We here introduce common notations used in the current paper. Since the infrastructures of the source ANN and target SNN are the same, use the same notation when it is unambiguous. For the (-th layer, we denote Wi as the weight matrix. The threshold on activation value added to the ReLU function in the source ANN is Yth and the threshold voltage of the spiking function in the target SNN is Vin- The SNN is simulated in T time points, where (t()) = {e((t)) } is the vector collecting membrane potentials of neurons at time t, and 6' = (t)} records the output to the next layer, i.e the post synaptic potential (PSP) released by 1-th layer to the (1 + 1)-th layer. Suppose that within the whole simulation time T, the 1-th layer receives an average input a and an average PSP af from the 1)-th layer corresponding the source ANN and target SNN, the forward process
2

---

Published as a conference paper at ICLR 2021
can be described by
01+1 h(W, -ai)
(I)
df+1 ((Wi.a)
where h-(-) and h-()) denote the activation function for the source ANN and target SNN on the average sense respectively. For the source ANN, the activation function fi-)) is identical to the activation function for the single input, which is the threshold ReLU function in this work, i.e.
0
œ S 0;
h(x)
x, if < < Vohi
(2)
Yth if 2 Uth
where You is the threshold value added to the regular ReLU function. For the target SNN, the activation function (-) will be derived in the following section.
B
ANN
SNN
Belli ShiN
Sox www.khheeiscllll...... shall
Ereer
RetU Rodiitimmhutti
X
K.
ReLU neuron
Theeshold talancing
Sp nouron
Activation
V K2/27
Va Pro activation
Soft reset
E. 27
Figure 1: Schematics on the conversion pipeline, (A) The SNN propagates spiking frequencies with the activation sequence x = {af(1), ({)) and averaged output af for / = I..., through L layers. (B) Activation functions of regular and threshold ReLUs for ANNs, and the step function for SNNs. (C) The error between ReLU and step function with /2 shift. See Section 5 for the detailed discussion.
In the following sections, first derive the equations for weight transform from the source ANN to target SNN. Next, we prove that the overall conversion eтror can be decomposed to the summation of error between source activation value and target output frequency on each layer. Finally, we estimate an optimal shift value for each layer based on the threshold balancing mechanism (Diehl jet al. 2015; [Sengupta et al.) (0188) and formulate the overall conversion error. Putting these three parts together, we demonstrate both that our approach almost achieves the optimal solution when converting ReLU-based ANNs to spike-based SNNs.
3 CONVERSION EQUATION FROM ANN TO SNN
Following the threshold balancing mechanism (Diehl et al. 2015 Sengupta al. 2018 that copies the weight from the source ANN to the target SNN, we here derive the forward propagation of PSP through layers in the target SNN. For the I-th layer the SNN, suppose it receives its (t+ 1-+11)x 1) input a((++] 1) at time point /+1 1. Then there will be an additional membrane potential W1 o/ b added to its membrane potential n' (1)) at time point f, resulting in temporal potential
d'eemp (t + 1) o'))) + W. af(1 + 1).
(3)
any element in v (t + 1) exceeds Vzn. it would release a spike with potential Vo, decrease its
membrane potential by Vna and update the membrane potential to of (L+1) time point (++ 1. Thus we can write the membrane potential update rule as:
( n + Wi af + 1) 0¹(1 + 1),
(4)

---

Published as a conference paper at ICLR 2021
where ('(( + 1) equals Ves to release PSP to the next layer if Blemp (t 1) 2 Vis and remain as 0 if Vremp(e (t 1) < Vea. We accumulate the Eqn 4 from time 1 to T, divide both sides of the equation by T, and have
(T)
(1 t=1 T
T
T Σ t=1
(1) T
W
(5)
T
We use aj = Ei= (t)/TTuu to denote the averaged input to the 1-th layer. Then Equ gives that the input to the (1 + 1)-th layer equals the expected PSP of the (-th layer, i.e. a'++1 O' (t)/TT. If we set the initial membrane potential (0) as zero, we can reformulate the Eqn as:
d/41 Wi d (T) T
When the threshold potential VAa is set greater than the maximum of the source ANN's activation values, the remained potential v' (T) would be less than Vi thus would not be output finally. With the clip-operation, the output then can be expressed as
d/41 = N'W af) = Ven T
(1 (WWi.
Wi aj Vin/T .O.T).
clip
when 0 < x < T:; and ca, O) = T when x T. So the output function of SNN is actually a step function (see the blue curve Fig B). The clipping by Eq is accurate in most cases but can be slightly different from 6 the summation of membrane potential cancel between the positive and negative parts while the positive parts get spiked along the sequence. The forward equation for ANN is shown as the green line in B. Since the SNN output discrete and in the form of floor rounding while the ANN output continuous, there actually would be an intrinsic difference in t'+1 and a1+1 as shown in Fig B.C even if we equalize af and ate We will analyze how to minimize this difference later.
where clip(x, 0, = 0 when S 0; (r, 0,
4 DECOMPOSITION OF CONVERSION ERROR
The performance of the converted SNN is determined by the source ANN performance and the conversion error. While the former one isolated from the conversion, we discuss how to optimize the latter one here. The loss function L can also be viewed as function of the last layer output thus the conversion error can be formulated as
AL E[L(a',)] - B[[(az)),
(8)
where the expectation is taken over the sample space. For the 1-th layer in the SNN, we can reversely approximate its output with the souree ANN's activation function hi(-) and get
a N/( a!_1) h.(W a/_1) + Aar',
where Aaj' is the error caused by the difference between the activation functions of the ANN and SNN. The total output error Day between the souree ANN and target SNN on the 1-th layer, ie. the difference between the activation a and ai can be approximated as
Da: a.' a Aai' + [W; a{-1) (W; a=11) Sai B W/ Dai-1 (10)
where the last approximation is given by the first order Taylor's expansion, and Bi is the matrix with the first derivatives of h on the diagonal. Expand the loss function Eqn around az, we get
AL R E E C. dau) + 1 [Da Ha,
TH,T Aaz).
(11)
where the last approximation is given by the 2-order Taylor's expansion, and Ha, is the Hessian of C w.r.t az, As L is optimized on the source ANN, can ignore the first term here. Thus it suffices to find Dar, to minimize the second term. By substituting Eqn. in EqulI] we further have
E Aaa "Ha, Daz] AElAat" Ha, Aa',) + E[AaL-1" TB BB Ha, Wr + 2EE T W,, Ha, Sa't,
(12)
4

---

Published as a conference paper at ICLR 2021
where the interaction term can either be ignored by decoupling assumption in (Nagel al. 2020) or dominated by the sum of the other two terms by Cauchy's s inequality. Applying similar derivation as in (Botev et al. 2017), have Hap-1 B,W," Ha, W,B,. Then Eqn 12 can reduce
ElA T Ha, Aar. 2 ElAar." 'T Ha, so'2 + E|Aar.- Haz Aar-11
= Σ B|Sa," Ha, Aai].
(13)
Here the term Ha; can either be approximated with Fisher Information Matrix (Liang et al, 2019) similarly assumed constant as in (Nagel et al.) (020). For simplicity, take constant and problem of minimizing the conversion error then reduce to the problem minimizing the difference of activation values for each layer.
LAYER- WISE AND TOTAL CONVERSION ERROR
In the previous section, we analyze that minimizing the conversion error equivalent minimizing output errors caused by different activation functions of the ANN and SNN on each layer. Here we further analyze how to modify the activation function so that the layer-wis, wise error can be minimized.
First, we consider two extreme cases where (I) the threshold VeA is so large that the simulation time T' not long enough for the neurons fire spike (2) the threshold Voh so small that the neuron spikes every time and accumulates very large membrane potential after simulation (upper bound of Eqn 7).. For these two cases, the remaining potential contains most of the information from ANN and it is almost impossible to convert from the source ANN to the target SNN. To eliminate these two cases, we apply the threshold ReLU instead of the regular ReLU and set the threshold voltage Vaa in the SNN as the threshold yea for ReLU in the ANN.
Next, we further consider how to minimize the layer-wise squared difference B| BlIk((Wa( h a For the case of using threshold ReLU for h4, the curve of hu and hij are shown in Fig with their difference in Fig C. We can see that when the input is equal, h; and h actually have systematic bias that can be further optimized by either shifting h or h4- Suppose that we fix h; and shift h by 8, the expected squared difference would be
Eg (z 5) h2(2))
(14)
If assume that = Wa is uniformly distributed within intervals [(t 1)Va. VT//TT for t = I,...,T T, optimization of the loss in Eqn 14 can then approximately be reduced to
Vin T
Vin 27
arg min
8
+
+
(15)
2
Thus the total conversion error can be approximately estimated as
LV 4T
ACmin
(16)
which explicitly indicates how the low threshold value and long simulation time decrease the con- version error. In practice, the optimal shift may be different from Vt./2T. As we illustrate above, the effect of shift is to minimize the difference between the output of the source ANN and the target SNN rather than optimizing the accuracy of SNN directly. Thus the optimal shift is affected by both the distribution of activation values and the level of overfitting in the source ANN and target SNN.
The conversion pipeline is summarized in Algorithm | where the source ANN pre-trained with threshold ReLU. The threshold voltage Vn can also be calculated along the training procedure.
RELATED WORK
Cao et al. 2015) first propose to convert ANNs with the ReLU activation function SNNs. This work achieves good results on simple datasets but can not scale to large networks on complex da- sets. Following Cao et al. (205)), the weight-normalizatiio method is proposed to convert
5

---

Published as a conference paper at ICLR 2021
Algorithm 1 Conversion from the Source ANN to the Target SNN with Shared Weights
Require: Pre-trained source ANN, training set, target SNN's simulation length T.
Ensure: The converted SNN approximates the performance of source ANN with an ignorable error.
Initial V.h
for =1 L to save the threshold value for each SNN layer.
2: for
to # of samples do
3: ar <- layer-wiss activation value
4:
for 1 - to L do
5:
ViA. = max (V max(ai)) [a]]
6: end for
7: end for
8: for I= 1 to L do
9: SNN.layer V < Vh
SNN.layer| (1 weight la
11: SNN.layer bias + ANN.layer(] bias + /
end for
2016 The spike subtraction mechanism (Rueckauer et al., 2016), also called soft-reset (Han et al. (020). is proposed to elim- inate the information loss caused by potential resetting. More recently, /Sengupta al. 2018) use SPiKE-NORM to obtain deep SNNs like VGG-16 and ResNet-20, and Kim et al. 2019 utilize SNN for object recognition.
three-layer CNN structure without bias to SNN (Dichl al. 2015
The most commonly used conversion method is threshold balancing (Snguutaa et al. 2018), which equivalent to weight normalization (Dichl al. 2015 Rueckauer al.) (016). The neurons in the source ANN are directly converted into those in the target SNN without changing their weights and biases. The threshold voltage of each layer in the target SNN is the maximum output of the corresponding layers in the source ANN. The soft reset mechanism (Rueckauer et al. 2017; Han al. (020) effective in reducing the information loss by replacing the reset potential Vrest the hard reset mechanism with the difference between membrane potential V and threshold voltage Vva so that the remaining potential still informative of the activation values. Another big issue for direct conversion is the varied range of activation for different neurons, which results in very long simulation time to activate those neurons with high activation threshold values. Rueckauer et al. (017) suggests using the p-th largest outputs instead of the largest as the weights normalization scale to shrink the ranges. Kim et al.] (2019) applies threshold-balame than the layer level for better adaption. Rathi al. (2019) initializes the converted SNNs with roughly trained ANNs to shorten simulation length by further fine-training the SNN with STDP and backpropagation. |Han al. (2020). Han & Roy (2020) adjust the threshold and weight scaling factor adapted to the input and output spike frequencies.
on the channel level rather
7 EXPERIMENTS
In this section, we validate our derivation above and compare our methods with existing approaches via converting CIFAR-Net, VGG-16, and ResNet-20 on CIFAR-10, CIFAR-100, and ImageNet. See Appendix A. for the details of the network infrastructure and training parameters.
7.1 ANN PERFORMANCE WITH THRESHOLD RELU
We first evaluate the impact of modifying ReLU on the source ANN from the perspective of maxi- mum activation value distribution and classification accuracy. A smaller range of activation values will benefit the conversion and but may potentially result in lower representativity for the ANN.
We set the threshold value Yos. = 1 on CIFAR-10, Vth 2 on CIFAR-100 and examine the impact of threshold on the distribution of activation values. From Fig we can see that the threshold operation significantly reduces the variation of maximum activation values across layers and the shift-operation won't cause any additional impact significantly. This suggests that with the modi- fied ReLU, the threshold voltage Vnn of spiking could be more effective activate different layers. We next look into the classification accuracy when modifying the ReLU with threshold and shift. Comparisons CIFAR-10 10 and CIFAR-100 shown in Fig 33 On the CIFAR-10. 10, the perfor-

---

Published as a conference paper at ICLR 2021
A
Maximum activation VGG-16 16 layers
B
CIFAR-10
CIFAR-100
30
ReLU thresholld
ReLU
94
ReLU+threshald
92.95
71
20
ethreshold
70.43
93
69.82
70
10
91.62
91,67
69
0
92
68
0.5
90.37
67.04
90.191
67
99
66
Layer
CIFARNet VGG- ReaNet-20
VGG-16 ResNet-20
Figure 2: Impact of threshold on ANN. Maximum activations of VGG-16's layers on CIFAR-10. (B) The accuracy of ANN with regular threshold ReLUU different networks. Average over repeats on CIFAR-10 and 3 repeats on CIFAR-100.
mance of networks with modified ReLU actually get enhanced for all three infrastructures (+0.18% on CIFAR-Net, +0.26% on VGG-16 and +1.28% on ResNet-20). On the CIFAR- dataset, the method of adding threshold results in slight decrease in accuracy on VGG-16 (-0.16%), but huge increase on ResNet-20 (+2.78%). These results support that the threshold ReLU can serve as a reasonable source for the ANN to SNN conversion.
7.2 IMPACT OF SIMULATION LENGTH AND SHIFT ON CLASSIFICATION ACCURACY
In this part, we further explore whether adding threshold and shift to the activation function will affect the simulation length (T) required for the converted SNN match the source ANN in classi- fication accuracy and how shift operation affects differently in different periods of simulation.
B
RoLU threshold
69,9
ReLl
T=200
T-16
T-200
Baccuncy
Accuracy
g
40-
69.5
scale 1.1
Scale 1.0
Scale U.1
69.3
00
69.1
0
2
3 scale
Simulation length
Shilt scale
ShiTi scale
Shift
Figure 3: Impact of threshold and shift on convergence for converting ResNet-20 on CIFAR- 100. (A) SNN's accuracy losses w.r.t different simulation lengths. (B-D) Optimal shifts w.r. different activation functions and simulation lengths.
that matter applied separately or together, the threshold and shift oper- ations always significantly shorten the simulation length to less than 100 when the accuracy loss is much smaller than using regular ReLU with T 200. When T < 50, the combination of threshold and shift achieves the fastest convergence and the adoption of threshold is more efficient than shift. However, the difference is minor when simulating with T 100. This inspires us to further explore the impact of shift at different simulation length. From the green and blue lines in Fig 2A and accu- racy changes w.r.t the variation of shift scales in Fig we can conclude that the shift by 22T is almost always optimal even when T = 200 for the source ANN with regular ReLU. However, the shift works differently when the ReLU with threshold. When the simulation length very short like T = 16, the conversion can benefit bugely (> 0.30 accuracy improvement) from the de- rived optimal shift (see FCg When the simulation length is long enough, the shift mechanism no longer works and may sligtlly damage the accuracy (Fig 2D) while adding threshold alone will almost eliminate the conversion error that consistently remains above zero in the original threshold
From F
balancing approach (Diehl al. 2015 Sengupta al. 2018 shown in Appendix Fig 4
7

---

Published as a conference paper at ICLR 2021
7.3 COMPARISON WITH RELATED WORK
In order to validate the effecctiveness of the whole proposed pipeline, here we compare our method with NORM (Sengupta et al. 2018), Hybrid Training (Rathi et al. 2019), RMP (Han et al. 2020), TSC(Han & Roy, 2020) and direct training method including STBP (Wu et al.
2019) and RSSL-BPPA Zhang & Li 2020) through classification tasks on CIFAR-10, CIFAR- 100 and ImageNet [ For the network with relatively light infrastructure like CIFAR-Net, our model
con-
verge to > 90% accuracy within 16 simulation steps. Direct training methods like STBP and TSSL- BP are feasible to achieve a high accuracy with short simulation length. However, their training cost is high due to RNN-like manner in the training procedure and cannot extend to complex network infrastructures like VGG-16 and ResNet-20. RMP and TSC achieve the highest accuracy for SNN with VGG-16, probably a result from more accurate pre-trained source ANNs. For ResNet-20, our model outperforms the compared models not only in accuracy but also in the simulation length.
Table 1: Comparison between work and other conversion methods. The conversion loss is reported as the accuracy difference (aCCANN oCCsNN) between the source ANN with regular ReLU and threshold ReLU (in brackets) and the converted SNN. Ourwork-X-TS denotes the SNN converted from the source ANN with both threshold (T) ReLU and shift (S).
CIFAR-10 + CIFAR-Net
CIFAR-10 VGG-16
CIFAR-10 10 RezNet-20 Conversion Less
Method
Comersiin Loss
Leagth
Accuracy
Conversion Loss
Length
Accuracy
Length
STBP Twic NeuNarm)
59.83%
0.68%
NA
NA
STBP (w/ NeuNama)
90.53%
-0.04%
8
NA
NA
TSSL-BP
91.11%
-0: 5..2%%
5
NA
NA
SPIKE-NORM
NA
91.55%
0.15%
2500
87.46%
166%
2500
KMP
NA
93.6256
0.01%
1536
91.367
0.11%
2048
Hybrid Training
NA
92.
250
TSC
93.63%
91.42%
0.05%
1536
Our Wock-1-TS
90.22%
(115%
16
92.20%
-O 0.200))000)
16
92.41%
-0.00% (1 102005)
16
Our Work-2-TS
90.46%
-0.18% 17%)
32
02.280%
-O (5(00.06%)
32
93. 13.
-O 989 (0.3155)
32
Our Work-3-TS
90.58%
0.20%
64
92.22%
0.133% 12%)
64
93.517
23% (0.D656)
64
Work-4-18
90.58%
(0.22%)
128
92.24%
-.155%00 1015
93.565 5677
2575 (0.0056)
128
Our Work-S-T
0.23%%
400-600
02.20%
-0.173600.068
2655 (0.055
400-600
Our Work-66S
90.13%
D.0G5S
512
92.03%
0.06%
92.14%
0.18%
512
CIFAR-TO0 a VCG-16
CIFAR-IO0 ResNee20
ImageNet= a VCG-16
Method
Accuracy
Conversion Loss
Length
Accuracy
Conversion Loss
Length
Accuracy
Conversion Loss
Length
SPIKE-ORM
70.77%
0.45%
2500
64.09%
4.63%
2500
69 96531
0136%
2500
RMP
70.90%
0.29%
2048
67.82%
0.9%
73.09%
0.4%
4096
Hybrid Triining
NA
4.16%
250
TSC
70.97%
0.25%
1024
68.18%
0.54%
2018
73.46%
0.03%
2560
Our Work-1-TS
65.94%
4.8956 53%)
16
63.75%
3. (6.0%))
16
55.80%
16.69% (16.38%)
16
Our Work-2-TS
69.80%
0.8256 0.0%%)
32
888.00%
-1 1.82%)(1-4/55
32
67. 75%
4 L.77%i..4553
32
Our Work-3-TS
70.35%
0.2756 14%)
64
60.27%
-2.193/00 5395)
64
70.077
1. 1.43% (111%
64
Work-4-TS
70.47%
0.1656:0 (222)
128
60.40%
2.41%/033115
71. 80%
0.61%/0..29%)
128
Our Wadk-S-T
70.66%
0.07%6 0.00%)
400-600
800822
2.74%EC 0.02%)
400-600
72.17%
0.23%/0..155)
480-600
Our Work-6-S
70.28%
0.31%
512
66.63%
0.45%
512
72.34%
0.06%
512
We next pay attention to the accuracy loss during the conversion. For VGG-16, our proposed model averaged on length 400-600 achieves a similar loss with RMP on CIFAR- (es 0.01%). On CIFAR- 100 and ImageNet, our model maintains the lowest loss compared to both the ANN trained with regular ReLU and threshold ReLU. For 20sNet-20. there appears an interesting phenomenon that the conversion to SNN actually improves rather than damage the accuracy. Compared with the source ANN trained with regular ReL.U, the accuracy of our converted SNN has a dramatic increase with 1.26% on CIFAR-10 and 2.74% on C10.R-100. This may be result from the fact that the source ANN for conversion is not compatible with the batch normalization and max pooling thus bears a deficit by overfitting that can be reduced by the threshold and discretization of converting to SNN.
In addition to the excellence in accuracy, our model is even more advantageous in the simulation length. When we combine the threshold ReLU with shift operation through the layer-wise approxi- mation, our model can achieve a comparable performance with RMP for VGG- 16 with 16 simulation steps on CIFAR-10 and 32 simulation steps on CIFAR- much shorter than that of RMP (1536 on CIFAR-IO, 10, 2048 on CIFAR-100) Similar comparisons are observed on ResNet-20 with better performance than RMP, TSC, SPIKE-NORM and Hybrid Training, On ImageNet, although we do not achieve the highest accuracy due to the constraint of accurate source ANN and short simulation length, we can still observe that the 400-600 simulation steps can return conversion error smaller than that of RMP, TSC and SPIKE-NORM with 4096 and 2500 steps correspondingly. When the simulation length is 512, the conversion with only shift operation works accurately enough as well.
8

---

Published as a conference paper at ICLR 2021
See Appendix A.2 for the repeated tests averaged with simulation length 400 600 and Appendix A.3 for the comparison with RMP and TSC when the simulation length is aligned on a short level.
The proposed algorithm can potentially work for the conversion of recurrent neural network (RNN) as well. See Appendix[A.4ffor an illustrative example that demonstrates the superiority of the current work over directly- -tainnnng approach when the source architecture is an RNN.
8
DISCUSSION
Since Cao et al. 2015 first proposed conversion from ANN with ReLU SNN, there have been many follow-up works (Diehl et al., 2016; Rueckauer et al.) 2016 Sengupta et al. 2018 Han et al.] 2020) optimize the conversion procedure. But they fail to theoretically analyze the conversion error for the whole network and neglect to ask what types of ANNS are easier to transform. We fill this gap and performed a series of experiments to prove their effectiveness.
Our theoretical decomposition of conversion error actually allows a new perspective of viewing the conversion problem from ANNs to SNNs. Indeed, the way we decompose the error on activation values works for the problem of network quantization (Nagel et al. 2020) as well. This is because the perturbation in edge weight, activation function and values would all affect the following layers through the output values of the current layer. However, one limitation to point out here is that the interaction term Eqn 12 may not be small enough to ignore when different layers are strongly coupled in the infrastructure. In this case, the layer-wise recursion is suboptimal and blockwise approaches may help for further optimization. The trade-off between representativity and generaliz- ability applies for the adoption of threshold ReLU in the training of ANN and provides intuitive interpretation on why our converted SNN outperforms the source ResNet-20 in the conversion.
We also show that shift operation is necessary for the conversion considering the difference between ReLU and step function. This difference has also been noticed in direct training approach as well. Wu et al. 2018 2019) use modified sign function to model the surrogate gradient. This choice makes the surrogate curve coincide with the red line in Fig 2B. Yet, we need to point out that the shift operation cannot ensure the improvement when the simulation length is long enough. We can understand the shift operation as an approach to pull the target SNN to the source ANN. At the same time, pulling the SNN to ANN also causes it to reduce the generalizability out of discretization. As we describe above, the target SNN is possible to outperform the source ANN by benefiting more generalizability than losing the representativity. In this case, it makes no sense to pull the target SNN model back to the source ANN anymore. This is potentially the reason why the optimal shift no longer /2T when the simulation length is long enough to make the target SNN hardly gain more representativity than lose generalizability by getting closer to the source ANN.
Compared to the direct training approach of SNNs, one big limitation of converting SNNs from ANNs is the simulation length. Although SNNs can reduce the energy cost in the forward process (Deng et al.] 2020), the huge simulation length will limit this advantage. Our method greatly reduces the simulation length required by the converted SNNs, especially for large networks and can be combined with other optimization methods in Spiking- YOLO, SPIKE-NORM and RMP. Further, as we point above about the duality between input x and edge weight w. future optimization could potentially reduce the simulation length to about 2 = 16 that is equivalent to 4 bits in network quantization with very tiny accuracy loss.
In the future, it makes the framework more practical if we can extend it to converting source ANN trained with batch normalization and activation functions taking both positive and negative values.
ACKNOWLEDGMENT
This project is primarily supported by NSFC 61876032.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Simone Acciarito, Gian Carlo Cardarilli, Alessandro Cristini, Luca Di Nunzio, Rocco Fazzolari, Gaurav Mani Khanal, Marco Re, and Gianluca Susi. Hardware design of lif with latency neuron model with memristive stdp synapses. Integration, 59:81 - 89, 2017.
Michele Barbi, Santi Chillemi, Angelo Di Garbo, and Lara Reale. Stochastic resonance in a sinu- soidally forced lif model with noisy threshold. Biosystems, 71(1-2):23-28. 2003.
Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical gauss-newton optimisation for deep learning. arXiv preprint
2017.
Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1):54--655 66, 2015.
Lei Deng, Yujie Wu. Xing Hu, Ling Liang, Yufei Ding, Guoqi Li, Guangshe Zhao, Peng Li, and Yuan Xie. Rethinking the performance comparison between snns and anns. Neural Networks, 121:294 - 307, 2020.
Peter U Diehl and Matthew Cook. Efficient implementation of stdp rules on spinnaker neuromorphic hardware. In 2014 International Joint Conference on Neural Networks (LJCNV), pp. 4288-4292 4295. IEEE, 2014.
Peter U. Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. 2015 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2015.
Peter U Diehl, Guido Zarrella, Andrew Cassidy, Bruno U Pedroni, and Emre Neftci. Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware. In 2016 IEEE International Conference Rebooting Computing (ICRC), pp. 1-8.
IEEE, 2016.
Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In European Conference Computer Vision, 2020.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13558-13567, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Sun Jian. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Alan L Hodgkin and Andrew F Huxley. A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of physiology, 117(4):500-544 1952a.
Allan L Hodgkin and Andrew F Huxley. Currents carried by sodium and potassium ions through the membrane of the giant axon of loligo. The Journal of physiology, 116(4):449 472, 1952b.
Eugene M Izhikevich. Resonate-and-fim neurons. Neural networks, 14(6-7):883-885 2001.
Eugene M Izhikevich. Simple model spiking neurons. IEEE Transactions on neural networks, 14(6):1569-115) 1572, 2003. 2003.
Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for energy-efficient object detection. arXiv preprint arXiv:/903.06530, 2019.
Alex Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25(2), 2012.
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metrie, ge- ometry, and complexity of neural networks. The 22nd International Conference on Artificial Intelligence and Statistics, pp. 888-896, 2019.
10

---

Published as a conference paper at ICLR 2021
Ying- Hui Liu and Xiao-Jing Wang. Spike-frequeeee adaptation of generalized leaky integrate- and-fire model neuron. Journal of computational neuroscience, 10(1):25 45)):25-5,,00 2001.
Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5(4):115-133, 1943.
Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort, Up or down? adaptive rounding for post-training quantization. arXiv preprint arXiv:2004. 10568, 2020.
Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):006-111 2019.
Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations, 2019.
Deboleena Roy, Indranil Chakraborty, and Kaushik Roy. Scaling deep spiking neural networks with binary stochastic activations. In 2019 IEEE International Conference on Cognilive Computing (ICCC), pp. 50-58. IEFE, 2019.
Bodo Rueckauer, Aalia-Ale Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion of analog to spiking convolutional neural networks. arXiv: Statistics/Machin Learning, (1612.04052):0-0, 2016.
Bodo Rueckauer, Iulia- Alexa-Aae Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Con- version of continuous valued deep networks efficient event-driven networks for image classifi- cation. Frontiers in neuroscience, 1 11:682, 2017.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in Neuroence, 13, 2018.
Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems, pp. 1421, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.15555 2014.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Ports. Recursive deep models for semantic compositionality over a sentimen- t treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Johannes C Thiele, Olivier Bichler, and Antoine Dupret. Spikegrad: An ann-equivalent computation model for implementing backpropagation with spikes. In International Conference on Learning Representations, 2019.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi.
backpropagation for
training nigh-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger. better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311-1318, 2019.
Amirreza Yousefzadeh, Timothée Masquelier, Teresa Gerrano-Gotarredona, and Bernabé Linares- Barranco. Hardware implementation of convolutional stdp for on-line visual feature learning. In 2017 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1-4. IEEE, 2017.
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. arXiv preprint axXiv:2002.10088 2020.
11

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 NETWORK STRUCTURE AND OPTIMIZATION SETUP
We adopt three network structures, CIFARNet (Wu et al. 2019). VGG-16 and ResNet-20 (Sengupta et al. 2018). All the pooling layers use average pooling to adapt to SNN. CIFARNet is 8 layer structure like: 128C3
-2oiing where C means convolutional layer, AP means average pooling and FC means fully connected layer. We use the VGG-16 and ResNet-20 model provided by Rathi et al. (0119) on the CIFAR dataset and standard VGG-16 on ImtgeNet. ResNet-20 is ResNet-18 adding two convolutional layers for pre- processing in front, and an activation function is added after the basic blocks (Sengupta et al. 2018) Following (Sengupta et al. 2018), before training, initialize convolution layers (kernel size k, and n output channels) weight normal distribution and standard deviation v 22(kk2) for non-residual convolutional layer, v2/(k2)) for residual convolutional layer. And then we add dropout layer that p 0.2 between convolutional layers and p 0.5 between fully connected layers.
Our work is based on Pytorch platform. For CIFAR-IO and CIFAR-100, the initial learning rate of each network is 0.01, batch size is 128, total epochs is 300. We use cross entropy loss function and SGD optimizer with momentum 0.9 and weight decay 5e - 4. And learning rate decay by a factor of 0.1 at epochs of 180, 240 and 270. For the sake of better conversion, in the first 20 epochs, use regular ReI.U for training, and then add the threshold to ReI.U. On CIFAR- 100, our network is initialized with the network parameters The training on CIFAR-100 adopts the pre-trained weight without threshold for initialization in the first few epochs as a compensation of the lack of nalthhoormalization (Han et al., 2020 Rathi al. 2019) to enable the convergence in early steps.
CIFAR-I0 except the output layer (Pei et al.) 2019).
A.2 SNN PERFORMANCE WITH SIMULATION LENGTH 400-600
We provide the average accuracy and variance of the converted SNN over the simulation length of 400-600 on CIFAR-IO (Table2) and CIFAR-100 datasets (Table3), the shift operation shifting constant V /1600, except for the output layer. The results include (1) regular ReLU without our method, (2) threshold ReLU and (3) both threshold and shift operation. Since simulation length of 400-600 is long enough for conversion from the source ANN with threshold ReLU to SNN, shift operation may reduce the converted SNN accuracy.
Table Performance of the converted SNN on CIFAR-10 (averaged on simulation length 400-600). ReLU-threshold:shiit
Structure
ReLU
ReLUsthreshold
CIFAR-Net
90.16 + 0.03
90.61 + 0.01
90.37 + 0.02
VGG-16
91.82 + 0.09
92.26 0.01
92.26 0.01
ResNet-20
92.14 = 0.03
93.58 + 0.01
93.59 + 0.01
Table 3: Performance of the converted SNN on CIFAR-100 (averaged on simulation length 400- 600).
Structure
ReLU
ReI.ushheelhlld
Rel.U-thrresood--shift
VGG-16
70.10 + 0.05
70.55 + 0.03
70.43 + 0.03
ResNet-20
66.74 + 0.04
69.82 + 0.03
69.81 + 0.02
A.3 SNN PERFORMANCE WITH SHORT SIMULATION LENGTH
Here compare the performance of our method versus RMP and TSC when the simulation length short. The conversion loss is reported as the accuracy difference (acCANN -cccsNN) between the converted SNN and the source ANN with regular ReL.U and threshold ReLU (in brackets). Our
12

---

Published as a conference paper at ICLR 2021
work-T/S denotes the SNN converted from the source ANN with both threshold (T) ReLU and shift (S). The performance of using shift operation and regular ReLU is similar to TSC on CIFAR- 10 (Tabld)). but is better than TSC and RMP on CIFAR-100 dataset (Tabld5). For VGG-16 on ImageNet, our model can achieve nearly zero conversion loss with simulation length 256 (Table 6.. Using both threshold and shift the same time reduces the conversion enror the fastest when simulation length is short. And only using threshold achieves the minimum conversion eπrоr on relatively long simulation time.
Table 4: Compare the conversion loss with short simulation length for VGG- and ResNet-20 on CIFAR-10.
Method
32
128
256
512
VGG-16 16 on CIFAR-IO
RMP TSC
33.33%
3.28%
1.22%
0.59%
0.24%
NA
0.84%
0.30%
0.18%
0.06%
Our Work-S
29.71%
3.10%
0.90%
0.60%
0.06%
Our Work-T
(0.31%)))
-0.21% (.0%)
0.18%
0.18%(0.07%%)
-0.16% (0.09%)
Our Work-TS
0.2%%(0.055()
0..3%
(0.
0. (0.1%))
0. 0.19%%0
0. 0.16%((0.09%)
Res-ett-20
CIFAR-10
RMP
16.78%
NA
3.87%
2.1%
0.84%
TSC Our Work-S
NA
22.09%
2.9%
1.37%
0.38%
4.24%
1.13%
0.39%
0.21%
0.18%
Our Work-T
19.72%(22..01%
-0.98% (0.31%)
-0. 15%)
-1.27%
-0.00%
Our Work-TS (0.39)))331%)
-1.23% (0.06%)
(0.25%)
-1.25%((0.05%)
-1.28%(0.011%
Table 5: Compare the conversion loss with short simulation length for VGG-16 and ResNet-20 on CIFAR-100.
Method
128
256
512
VGG-16
CIFAR-I00
RMP
NA
NA
7.46%
2.88%
1.82%
TSC
NA
NA 1.36%
0.57%
0.35%
Our Work-S
NA
NA
1.13%
0.55%
0.31 %
Our Work-T
NA
NA
0.21%
0.12%(-0.00)
0.07% (0.06%)
Our Work- TS
NA
NA
0. 0.15% 02%)
0.08% 0.05%)
0. (0.4%)
ResNet- 20 on CIFAR-100
RMP TSC Our Work-S
41.08%
2181% NA
11.03%
4.66%
2.56%
NA
10.03%
3.45%
1.55%
6.15%
1.61%
0.70%
0.52%
0.43%
Our Work-T
0.14%((..38%)
(0.5%
2.62%
2.78%) 0.06%)
2.76% 0.04%)
Our Work-TS
-1.3%%((.44)
(0.533%))
-2.41%(0.319))
-.41%%(0.31%))
-2.54%((..18%))
Table 6: Compare the conversion loss with short simulation length on VGG-16 and ImageNet.
Method
256
512
RMP TSC
24.56%
3.95%
3.75%
0.87%
Our Work-S 0.41%
0.06%
Our Work-T
0.22%(..02%) (0.5%))
Our Work-TS (026%))
0.25%(0.05%)
A.4 PERFORMANCE SNN CONVERTED FROM RNN
Here provide illustrative example of how the proposed conversion pipeline be extended to the case of converting RNN on the dataset for Sentiment Analysis on Movie Reviews (Socher al. 2013). The rules slightly different from conversion in the main text considering the
13

---

Published as a conference paper at ICLR 2021
implicit definition of RNN'' hidden states. For the fairness of comparison, and simulation length for the RNN and SNN and adjust the structure as follows. (1) The source RNN adopts the threshold-ReLD as its activation function with the remaining value the hidden state value. The hidden value will be added to the pre-activation value at the next time point. (2) We add non-negative attenuation
the same input
the hidden state values in order to enhance the nonlinearity. (3) The converted SNN keeps the same infrastructure, weight parameters, attenuation value as the source RNN. (4) On the oulput layer, we use the threshold balancing method and loop its input multiple times to fire enough spikes to obtain a good approximation to the fully- cully-connected layer for the final prediction.
We compare the performance the source RNN, converted SNN and directly-trained SNN. On the validation set, the converted SNN achieves an accuracy of 0.5430 that is close to the source RNN (acc 0.5428), while the directly- SNN with surrogate gradient only gets an 0.5 106 aceuracy. We also find that using the regular ReLU instead of threshold-ReLU a big accuracy drop for the converted SNN (ace = 0.5100). Since the surrogate gradient error of the directly-trainet frastructures and tasks usually require a long simulation to achieve an effective spiking frequency distribution, the typical directly-training approaches for SNNs are often not optimal in complex net- work structures and tasks. Our results illustrate the potential efficiency of converting SNN from RNN, In future works, would be promising to investigate how design better conversion strat- egy that can simultancously combine the strength of RNN and SNN.
the source RNN produces
method will accumulate
the whole simulation process while complex in-
A.5 COMPARE THE PERFORMANCE ON LONG SIMULATION
As supplement to Fig 3 A, here we compare the conversion accuracy loss of the SNNs converted from the source ANNs with regular and threshold ReLU functions along extended simulation length. Although the conversion loss in both cases decays fairly fast, the SNN converted from the ANN with regular ReLU function converges loss while the SNN converted from the ANN with threshold ReLU is almost loss-free. This result indicates that the improvement by adding threshold is not only the converging efficiency but also on the final performance, which cannot be easily compensated through extending the simulation time in the original threshold balancing approach.
plateau that suffers about 0.75% accuracy
ResNet20 on CIFAR-100
3
2.5
ReLUi threshold
2
AAccuaacy
1.5
0.5
0
0
500
1000
1500
2000
Simulation length
Figure 4: The accuracy gap between the source ANN and target SNN along an extended simulation length.
14


Reviews:
[{"review_id": "oKSNjjleynB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors seek a mechanism to train a spiking neural net to duplicate the function of a non-spiking one. This is desirable for energy-efficient inference, although the training process becomes challenging due to the discrete nature of the spiking process.\n\nTo achieve their goal, they described the spiking neuron non-linearity by a \"staircase\" function of the input (spiking output increases by 1 each time the input gets big enough to reach the next stair), and related that to the ReLu function used in the non-spiking neural net. They then determined parameters for the modified ReLU that would minimize the deviation between these activation functions, and computed the minimum conversion error (for converting ANN -> SNN). This scales with the square of the threshold voltage for spiking, divided by the simulation time. As one might expect, lower thresholds, and longer simulation times, both of which lead to potentially higher spike counts and thus lower discretization errors, lead to smaller conversion errors.\n\nUsing this, they defined their procedure for training SNN to mimic ANN as follows: they trained the ANN with their modified ReLU (which is closer to the SNN activation function but more readily differentiable), and then used the weights from that ANN in their SNN.\n\nNext, the authors evaluated their procedure on several different image categorization networks. Nice performance was obtained in all cases: better than using a normal ReLU, or other comparison activation functions, in the \"target\" ANN.\n\nOverall, this is a reasonably nice piece of work. I'd like to see this applied to recurrent neural nets: there, the dynamics of the SNN could be used more naturally, and the results might be more meaningful.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q-qB6_aVp75", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength: \n(1) This paper proposes a layer-wise optimization method for ANN-SNN conversion. I appreciate the theoretical analysis of how to minimize the conversion error.\n\n(2) This work significantly reduces the simulation time since long simulation time is usually required for converted SNN to reduce error. \n\n(3) It's interesting that conversion to SNN actually improves rather than damage the accuracy on ResNet.\n\nWeakness:\n(1) Although the proposed method is much more efficient, it does not show obvious performance improvement compared to existing methods.\n\n(2) In the experiment, the ResNet consistently show better performance. I hope the authors can provide more comments on this.\n\n(3) I'm not sure if I missed anything. The threshold RELU is not defined in the paper which may cause confusion.\n\n(4) I hope the authors can summarize the whole steps by formula or algorithm to help readers understand the entire process. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nzg6I9vupip", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Edit on second review\n\nI apologize again for the tone of my first review, I sincerely tried to understand the paper but I could not when I first read it. A re-read the paper and finally understood it during the review. I left a comment to the authors in the discussion below and they appropriately addressed my new recommendations. With the new equation (1) the paper is hopefully more understandable now. \n\nI increase my grade from 3 to 5. The findings are quite interesting but I still believe that the paper is not well written: the equations are interesting but the explanations between the equations are often unclear. One has to understand each equation and be quite imaginative to finally identify the contributions of the paper (even for somebody only \"very slightly\" off from the research topic).\n\n## Summary\n\nThe authors suggest a relationship between a leaky relu and a spiking integrate and firing neuron model.\nThis relationship suggests a mapping between the two models which is imperfect, a loss seems to be derived to reduce this mismatch along the network training. The method is tested on CIFAR-10 and CIFAR-100, and compared with some other methods for converting ANNs to SNNs.\n\n## Critical review\n\nThis topic is potentially important since spiking neural network are gaining popularity. But this paper is clearly badly written and it is extremely hard to understand, both in the math and in the text. I don't think it would help the progress of the field to publish the article in the current form.\n\nI tried to read that carefully and got lost after equation (4), the transition to equation (5) and (6) are not clear at all. I do not understand what is an approximation, what is a definition and what is a derivation.\n\nAlso (5) seems wrong in itself, the authors are trying to approximate a rectified linear network but it suggest that the activity will be equivalent to a linear network (at least when v(T) is small) ? And magically this changes in (6), and a clip non-linearity is introduced ?\n\nThe Figure 1 seems very encouraging at first, because it suggests that there is a clear and easy mapping between accumulating the spikes and computing a relu. I did not understand where this is appearing in the math and I cannot check whether the intuition conveyed by the figure is correct or not.\n\nI was therefore hoping to see an empirical study of the difference between the SNN and the ANN: do the activity of the spiking neural network match the activity of artificial network? This is not shown.  I do not even understand if it is necessary to re-train the network to go back from the SNN to ANN or vice versa.\n\nSince I had not understood the basics of the paper, it was impossible for me to understand the later section about the conversion error. My only take is that it seems wrong at first sight: how minimizing the error in the loss would minimize the mismatch between the network activity?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work tackles the task to convert an artificial neural networks (ANN) to a spiking neural network (SNN). The topic is potentially important for energy-efficient hardware implementations of neural networks. There is already quite some literature available on this topic. 
Compared to these, the manuscript exhibits a number of strong contributions: It presents a theoretical analysis of the conversion error and consequently arrives at a principled way to reduce the conversion error. The authors test the performance of the conversion on a number of challenging data sets. Their method achieves excellent performances with reduced simulation time / latency (usually, in order to achieve comparable performance to ANNs, one needs to run the SNN for many simulated time steps- this simulation time is reduced by their model). 
One reviewer criticized that the article was hard to read, but this opinion was not shared by other reviewers and the authors have improved the readability in a revision.

In summary, I believe that this manuscript presents a very good contribution to the field.
Decision: Accept (Poster)