A unified optimization framework of ANN-SNN Conversion: towards optimal mapping from activation values to firing rates
Authors: Haiyan Jiang
Srinivas Anumasa
Giulia De Masi
Huan Xiong
Bin Gu
Venue: Submitted to ICLR 2023 (2023)
Keywords: ANN-SNN conversion

Abstract: Spiking Neural Networks (SNNs) have attracted great attention as a primary candidate for running large-scale deep artificial neural networks (ANNs) in real-time due to their distinctive properties of energy-efficient and event-driven fast computation. Training an SNN directly from scratch is usually difficult because of the discreteness of spikes. Converting an ANN to an SNN, i.e., ANN-SNN conversion, is an alternative method to obtain deep SNNs.
The performance of the converted SNN is determined by both the ANN performance and the conversion error. The existing ANN-SNN conversion methods usually redesign the ANN with a new activation function instead of the regular ReLU, train the tailored ANN and convert it to an SNN. The performance loss between the regular ANN with ReLU and the tailored ANN has never been considered, which will be inherited to the converted SNN.  
In this work, we formulate the ANN-SNN conversion as a unified optimization problem which considers the performance loss between the regular ANN and the tailored ANN, as well as the conversion error simultaneously. Following the unified optimization framework, we propose the SlipReLU activation function to replace the regular ReLU activation function in the tailored ANN. The SlipReLU is a weighted sum of the threhold-ReLU and the step function, which improves the performance of either as an activation function alone.
The SlipReLU method covers a family of activation functions mapping from activation values in source ANNs to firing rates in target SNNs; most of the state-of-the-art optimal ANN-SNN conversion methods are special cases of our proposed SlipReLU method. We demonstrate through two theorems that the expected conversion error between SNNs and ANNs can theoretically be zero on a range of shift values $\delta \in [-\frac{1}{2},\frac{1}{2}]$ rather than a fixed shift term $\frac{1}{2}$, enabling us to achieve converted SNNs with high accuracy and ultra-low latency. We evaluate our proposed SlipReLU method on CIFAR-10 dataset, and the results show that the SlipReLU outperforms the state-of-the-art ANN-SNN conversion in both accuracy and latency. To our knowledge, this is the first work to explore high-performance ANN-SNN conversion method considering the ANN performance and the conversion error simultaneously.

Under review as a conference paper at ICLR 2023
A UNIFIED OPTIMIZATION FRAMEWORK OF ANN- SNN CONVERSION: TOWARDS OPTIMAL MAPPING FROM ACTIVATION VALUES TO FIRING RATES
Anonymous authors Paper under double-bling review
ABSTRACT
Spiking Neural Networks (SNNs) have attracted great attention as a primary can- didate for running large-scale deep artificial neural networks (ANNs) in real-time due to their distinctive properties of energy-efficient and event-driven fast com- putation. Training an SNN directly from scratch is usually difficult because of the discreteness of spikes. Converting an ANN to an SNN, i.c., ANN-SNN con- version, is an alternative method to obtain deep SNNs. The performance of the converted SNN is determined by both the ANN performance and the conversion error. The existing ANN-SNN conversion methods usually redesign the ANN with a new activation function instead of the regular ReLU, train the tailored ANN and convert it to an SNN. The performance loss between the regular ANN with ReLU and the tailored ANN has never been considered, which will be inherited to the converted SNN. this work, we formulate the ANN-SNN conversion optimization problem which considers the performance loss between the regu- lar ANN and the tailored ANN, as well as the conversion error simultaneously. Following the unified optimization framework, tion function to replace the regular ReLU activation function in the tailored ANN. The SlipReLU is weighted sum of the threhold-ReLU and the step function, which improves the performance of either as an activation function alone. The SlipReLU method covers a family of activation functions mapping from activa- tion values in source ANNs to firing rates in target SNNs; most of the state-of- the-art optimal ANN-SNN conversion methods are special cases of our proposed SlipReLU method. We demonstrate through two theorems that the expected con- version error between SNNs and ANNs can theoretically be zero on a range of shift values 8 E [- 2: 2 rather than a fixed shift term 1. enabling us to achieve converted SNNs with high accuracy and ultra-low latency. We evaluate our pro- posed SlipReLU method on CIFAR- 10/10- 00 and Tiny-ImageNer datasets, and the results show that the SlipReLU outperforms the state-of-the-art ANN-SNN con- version methods and directly trained SNNs in both accuracy and latency. To our knowledge, this is the first work to explore high-performance ANN-SM -NN conver- sion method considering the ANN performance and the conversion error simulta- neously, with ultra-low latency, especially for 1 time-step (T = 1).
unified
propose the SlipReLU activa-
2.
INTRODUCTION
Spiking neural networks (SNNs) are biologically-inspired neural networks based on biological plau- sible spiking neuron models process real-time signals (Hodgkin Huxley. 1952: Izhikevich, 2003). With the significant advantages of low power consumption and fast inference on neuro- morphic hardware (Roy al., 2019), SNNs are therefore becoming primary candidate to run large-scale deep artificial neural networks (ANNs) in real-time. The most commonly used neuron model SNNs is the Integrate-and-
(IF) neuron model (Liu & Wang. 2001). Each neuron the SNNs emits spike only when its accumulated membrane potential exceeds the threshold voltage, otherwise, it stays inactive in the current time-step. This setting makes SNNs more similar to bio- logical neural networks. Compared to ANNs, event-driven SNNs have binarized/spiking activation values, resulting in low energy consumption when implemented on specialized neuromorphic hard- ware. Another significant property of SNNs is the pseudo-simultanein of their inputs and outputs

---

Under review as a conference paper at ICLR 2023
for making inferences in a spatial- -eeppoal paradigm. Compared conventional ANNs that present a whole input vector at once, and process pass in SNN can efficiently process streaming varying inputs.
to produce one output value, the forwarding
Generally, there are two distinct routes to obtain an SNN: (1) training an SNN from scratch (Wu et al., 2018; Neftci et al., 2019; Zenke& & Vogels, 2021), and (2) ANN-SNN conversion (Cao et al., 2015; Diehl et al., 2015; Deng & Gu, 2021), i.e. converting ANNS to SNNs. Training from scratch uses a gradient-based supervised optimization method in back-propagation, pretending that SNNs are specialized ANNs. Due to the non-differentiabiiity of the binary activation function in SNNs, surrogate gradients are usually used (Neftci al., 2019), but it essentially optimizes different net- works in forward and backward passes. This method can only train SNNs on small- and moderate- size datasets (Li et al., 2021). ANN-SNN conversion is an effective method to obtain deep SNNs, with comparable performance as ANNs on large-scale datasets. There are two main types of ANN- SNN conversion mechanism: (1) one-step conversion, which converts the pre-trained ANN to SNN without changing the architecture of the pre-trained ANN, for example Diehl et al. (2015); Li et al. (2021), and (2) two-step conversion, which involves redesigning the ANN, training it and converting it to SNN, for example Cao et al. (2015); Deng & Gu (2021); Bu et al. (2021).
In this work, we investigate the two-step ANN-SNN conversion methods, where we usually redesign the ANN by replacing the regular ReLU activation function to new activation function, train the tailored ANN and convert to an SNN. A tailored ANN that deviates too much from the regular ANN will degrade its performance, resulting in performance loss which will be inherited to the converted SNN. However, the performance degradation between the regular ANN and the tailored ANN has never been considered in the existing ANN-SNN conversion studies. To achieve high- accuracy and low-latency SNNs (e.g., I or 2 time-steps), we are the first to consider the performance loss between the regular ANN with ReLU and the tailored ANN, as well as the conversion error simultaneously. Our main contributions are summarized as follows:
(1) We formulate the ANN-SNN conversion as a unified optimization problem which considers the ANN performance as well as the conversion error simultaneously.
(2) We propose to use the SlipReLU activation function in the tailored ANN, in order to minimize the layer-wise conversion error and keep tailored ANN performance good as the regular ANN.
(3) The SlipReL.U method covers a family of activation functions mapping from activation values in source ANNs to firing rates in target SNNs; most of the offte-of-the-ar optimal ANN-SNN conver- sion methods are special cases of our proposed SlipReLU method.
(4) We demonstrate through two theorems that the expected conversion error between SNNs and ANNs can theoretically be zero on a range of shift values ō E [- I 2: 3) rather than a fixed shift ! 2-
Experiment results also demonstrate the effectiveness of the proposed SlipReLU method.
2 PRELIMINARIES
Given a classification problem on an image dataset (x, y) E D, where y E {1.... .C} is the true class label for image æ E RTM. we train neural network f : x + f(x) in the form of an ANN/SNN, by optimizing the standard cross-entropy (CE) loss, LCE(U. p) = Σ₁= ye log(p.), where Ye and Pa are the e-th elements of the label and the network prediction p = f(x). Since the infrastructures of the source ANN and target SNN are the same, we use the same f notation when is unambiguous. And fANN or fsNN, otherwise. For the notations, refer to Table $1.
ANN Neuron Model. In conventional ANN, a whole input vector is presented the network at one time, and processed layer-by-laye through continuous activation to produce one output value. In ANNs, the forwarding computation of analog neurons formulated
a (t) (z = FANN (ANN(W(0))) (0()((" ((-1)),
(1)
where z()) and a (8) are the pre-activation and post-activation vectors of the é-th layer considered, ((O) denotes the weight matrix, and F.N is the activation function of the ANN.
SNN Neuron Model. Compared with ANN, SNN employs binary activation (i.e. spikes) in each layer. To compensate the weak representation capacity of the binary activation, the time dimension (or the latency) is introduced to SNN, where the inputs of the forwarding pass in SNN are presented as streams of events, by repeating the forwarding pass T time-steps to get the final result.
2

---

Under review as a conference paper at ICLR 2023
Here consider the Integrate-and-Fii (IF) neuron model (Cao et al., 2015; Bu al., 2021; Deng & Gu, 2021) for SNNs. We derive the forward propagation of PSP through layers the target SNN which is equivalent to the forwarding computation of the analog neurons in the source ANN. Suppose at time-step& the IF neuron in é-th layer receive its binary input x (((=))(t) from the previous layer, the neuron will temporarily update its membrane potential by
(t) = (t ([(t-)= 1) (( (t),
(2)
where (t) denotes the membrane potential at time step t, (t) denotes the temporary inter- mediate variable that would be used to determine the update from v() 1) to v(0 For each IF neuron, if the temporary intermediate potential u, ( exceeds the membrane threshold th would produce a spike output s! (t). Otherwise, it would release no spikes 8/ (t) = 0.
(9) (t) H(u (1)
(
I.
(8) if uf (t) (()
(3)
otherwise.
The vector s (t) - {5} (6) (t)) collects spikes of all neurons of f-th layer at time t. Note that ( can be different each layer. We update the membrane potential by the reset-by-subtraction mech- anism (Rueckauer et al., 2017; Han et al., 2020), which means the temporary membrane potential (E) ui (t) is subtracted by the threshold value v(()) if the neuron fires, 8s (t) 1. That is
th
(1) u (1) (0)) (L)V:))
(4)
Similar to Deng & Gu (2021); Bu et al. (2021), if the neuron in the current layer fires a spike, then will release an unweighted PSP (postsynaptic potential) (1) as input to the next layer,
x (0 = (t
As for the input to the first layer and the output of the last layer of the SNN, we do not employ any spiking mechanism in Li et al. (2021). We directly encode the static image to temporal dynamic spikes as input to the first layer, which can prevent the undesired information loss introduced by the Poisson encoding. For the last layer output, we only integrate the pre-synaptic input and do not fire any spikes.
3 UNIFIED OPTIMIZATION FRAMEWORK OF ANN-SNN CONVERSION
In this section, we formulate the ANN-SNN conversion problem as an optimization problem deter- mined by two terms: one is used to make the tailored ANN not far away from the regular ANN with ReLU, the other one is used to control the ANN-SNN conversion error.
Starting from the first ANN-SND SNN conversion work (Cao al., 2015), all the previous LNN-SNN conversion methods (Cao et al., 2015; Dichl et al., 2015; Deng & Gu, 2021) have focused on opti- mizing the ANN-SNN conversion error. The performance of the converted SNN is determined by both the ANN performance and the conversion error. However, no research work has considered the ANN performance in the two-step ANN-SNN conversions, which considered in our unified optimization framework. In the two-step conversion method, we need to redesign the new activa- tion function of the regular ANN get tailored ANN, train the tailored ANN and convert it to SNN. By considering the performance loss between the tailored ANN and the regular ANN, the new activation function should not deviate too much from the regular ReLU.
3.1 ANN-SNN CONVERSION IN UNIFIED OPTIMIZATION FRAMEWORK
Definition (Unified Optimization Framework of ANN-SNN Conversion). The ANN-SNN conver- sion can be formulated into unified optimizatioon framework with an implicit variable, T.
min EwEE (z: W. ReLU) f(z W,T,F. FANN))
(5)
+11- - Ez (l: W FANA)) - f(z; W, W, (TTTFsNN)))
where [0, 1]. Specially, if FANN is designed by considering the deviation from the regular ReLU, the layer-wise conversion error becomes
Ex (0) = Ez (IFANN(a (e-11, (())) - FSNN (1(11), ((N),
(6)
3

---

(C) Whenever Under review as a conference paper at ICLR 2023
Here f denotes the same neural network infrastructures shared by the source ANN and tar- get SNN (see description in Sect. 2), f(z W. ReLU) denotes the regular ANN with ReLU, f(z: W, T, FANN) is the tailored ANN with activation function FANN. f(z; W, FFNNN) is the converted SNN, z is the input to the neural network, W (W(0) are the weight matrix trained from the tailored ANN and copied to the target SNN, F = FANN U FSNN is the space of activation functions of the tailored ANNS and the target SNNs, and the latency T (or time-steps) is seen as an implicit variable inherently inherited from the target SNNs. Moreover, the latency also allows the flexibility of adjusting T to balance between the latency and the accuracy of the converted SNN for different applications.
Remark 1. (A) An effective activation function FANN of the tailored ANN should address the per- formunce lose caused by the deviation from the regular ReLU. (B) When FANN is designed by considering the deviation from the regular ReLU, the layer-wise error Eq. (6) would come from any mismatch of the three parts: (1) different activation values from source ANNs and target SNNs, i.e. (1) and x0), (2) different activation functions, i.e. FANN( and FsNN( (.), and (3) the latency vari- able T which implicitly affects both the activation values and activation functions.
the conversion error Ez Exx ((Err(())) achieves its minimum, it is called an "optimal" " ANN-SNN corver-
sion. For example, Deng & Gu (2021) ) achieves optimal minimum error of (V (2021) can theoretically achieve optimal minimum error of O.
whereas Bu et al. 4T
3.2 ANN-SNN CONVERSION ERROR ANALYSIS
3.2. FIRING RATES IN SNNS AND ACTIVATION VALUES IN ANNS
To make the layer-wise error as small as possible, ideally, the converted SNN is expected to have approximately the same activation values as the source ANN for each layer, i.e.,
T
1 Σ (t)
x
Σ
S 10 (t)V
v V (0)(
Here (*)) denotes activation value of the ANN, and x is activation value of the SNN which is actually the average postsynaptic potential (i.e. average PSP) released by the f-th layer as input to the next layer. Note g(t) is the firing rate over latency T of E-th layer. Note that the thresholding Vih in SNN can be different from layer to layer, we make it a trainable parameter that can be learned in the source ANN and copied to the target SNN. Any mismatch between the activation values (*)) and (()) can lead conversion error.
3.2.2 ACTIVATION FUNCTION IN SNNS
We use the derivation in Deng & Gu (2021); Li al. (2021) to deduce the SNN activation function FSNN which gives the relationship between activation values ((t-1) and ( of successive layers of SNN. By combining Eq. (2) and Eq. (4) and summing up the time-step from to T, then we get
T
(0) (T) - (0 = w( Σ
(1)
Σ (0)V
The accumulated spikes are m the total number of spikes of neuron i. Further assume (0 [0, v!l)). Therefore, we have
Σt (() {m)) where each m, E 1,, T) denotes
( (t)
(e())((-1)
with shift 5 (()
th + +8<m m
+ 5
((0
v(()
v
th
th
Then, we use the clip and floor functions to determine m.
(1
TW TW
(
m clip
.
+8
8
.0
We provide the detailed analysis of the activation function of SNNs in Appendix A. Here the clip (I, a, function sets the lower bound and upper bound b. Floor function [x] gives the
4

---

Under review as a conference paper at ICLR 2023
greatest integer that is less than equal æ. With x activation function gives the relationship between activation values (t- and x(1) as follows,
th
mV /T, finally, the SNN
x FSNN (w(rx(e-1))
Vill clip
1 T
0.1
(8)
The SNN activation function FSNN (-) is a step function in interval [o, v:(0) with a step size - (see the green curve in Fig. 1). Since the SNN output is discrete while the ANN output is continuous, there actually would be an intrinsic difference between (t)) and x(4) as shown in Fig. 1.
SNN Step Function
SNN Step Function
Shift-threshold ReLU
QCFS Activation
g(z)
g
(/
Activation value
(A)
(B1)
(B2)
2014)
20
2010
g00
g(0)
g/0
7
T
0
ap(()
Vi
1:5
20
1:50
00
Pro-activation,
Pre-activation, No6, Tw3,
Pre-activation, Na3, Tw6,
SNN Step Function
SlipReL.U Activation
g(g)
Activation value
(CI)
(C2)
(C3)
2010
26
e00
ato
660
0
0
V/2
1/0
V/
16
Pre-activation. N=T=3, shift-0
Pre-activation. N=T=3, shift-I
Pro-activation. N=3, T=6, shift-0
Figure 1: Activation functions of source ANNs, i... the Shitt-thresholdReLL. (blue curve) (Deng & Gu, 2021) in (A), quantization clip-floor-shift (QCFS) activation (orange curve) (Bu et al., 2021) in (B1)-(B2), and our proposed SlipReL.U (red curve) in (C1)-(C3). The activation functions of target SNNs is the step function (green curve). The error between the activation function (of ANNs) and the step function (of SNNs) is obtained by summing up of all the shaded area together, which the ANN-SNN conversion error.
4 PROPOSED SLIPRELU: OPTIMAL ANN-SNN CONVERSION
4.1 THE SLIPRELU ACTIVATION FUNCTION
In this section, by following our unified optimization framework, we will exploit the two-step con- version mechanism. We redesign the ANN with to get a tailored ANN, train the tailored ANN and convert it to SNN by copying the weights from the tailored ANN to the target SNN. Perfor- mance loss will occur if the the new activation function of the tailored ANN deviates too much from the regular ReLU activation function, and we need to minimize the conversion error at the same time. By keeping that in mind, an effective activation function FANN of the tailored ANN should not deviate too much from the regular ReLU and be close to the step function. Therefore, we propose the SlipReLU activation function which is a weighted sum of the threshold-ReLU and the step function to balance the trade-off between the regular ReLU and step function. Assume that both ANN and SNN receive the same input from the previous layer, (f-1) = x(f-1) Denote (t) ((() ((-)) (t-1) (f (f-1)
Proposed Reip activation function. Following the unified optimization framework Sect. 3, new activation functions of the tailored ANNs are designed by minimizing the mismatch to the step
5

---

Under review as a conference paper at ICLR 2023
function of the target SNNs, and minimizing the deviation from the regular ReLU. We propose the SlipReLU activation function for the tailored ANN (see the red curves in (CI)-(C3) of Fig. D),
(2
FANN(Z! cf00
0.1 (1-c)
0, (0,1)
(9)
8
(C0
From definition in Eq. (9), the SlipReLu activation function is just a weighted sum of the threshold- ReLU (first part) and the step function (second part), with the slope 0 < & 1 balancing its weight.
With some linear algebra, the SlipReLU can be formulated as piece-wise linear function with a constant slope (see the red curves in (CI)-(C3) of Fig. 1 and detailed derivation in Appendix B), (k-1)ø(0)
SlipReLU(Z(0)) = (z + (1 - c) k ko < & the constant slope of the piece-wise linear function. From the definition and red curves in (CI) of Fig. I, we see that the new proposed function is very similar to slippery step function with a slope, hence the name "SlipReLU".
N: N & 2 <
N A = 0, N - 1. Here
SlipReLU extension
The SlipReLU extension with shift (see Appendix B) can be formulated as ( g(t) + 6,, (1- (1-c) c)
N2 ((0) + + 00.1)
FAN) = cf(0) clip
(10)
An application of the unified optimization framework. Recall the step activation function of target SNNs in Eq. (8), by setting = 0 the SlipReLU becomes the step function,
Nz
( = FANN(Z = clip
O()) + . .0.1)
(1I)
N
Note that Eq. (11) is exactly the step function in Eq. (8), except V2(() 010) N. Because the latency T in Eq. (8) is an inherent property of the target SNNs, so it cannot be used in the source ANN. Therefore, instead of T. we use quasi-time-steps N in SNN. As mentioned in Sect. 3.2.1, the threshold (
th in SNN can be different from layer to layer, and we make it a trainable value o( in the ANN which can be learned and copied to the target SNN. Coincidentally, Bu al. (2021) uses this function defined in Eq. (11) l) as the activation function in the source ANNs, and they name it the quantization clip-floor sloor-shift (QCFS) activation function and call N the quantization steps.
Special cases of SlipReLUU Here we list some related works which fall in to our proposed unifined optimization framework and are special cases of the SlipReL.U. (1) When = 0, 8 [1]. the proposed SlipRLU activation function becomes the quantization clip-floorshhif in Bu et al. (2021). It only considers to be close to the SNN step function, but neglects the deviation from the regular RELÚ, (2) When c = 1, = | N. the proposed SlipRLU activation function becomes shift- threshold ReLU in Deng & Gu (2021). It considers the deviation from the regular ReLU but neglects the closeness to the step function of the target SNN. Our proposed SlipReLU balances the trade-off between the regular ReLU and the step function. Refer to Appendix B for the details.
4.2 THEOREMS ON THE CONVERSION ERROR
The following two theorems gives the conversion error of the proposed unified method.
Theorem 1. An ANN trained with SlipReLU activation function Eq. (10) is converted to an SNN with the same weights. Let v-0 ((e), (())(0) v(l: 5, 0. Then for arbitrary T and N, the expectation of the conversion error of the proposed unified method reaches i.e..
VT.L E, (e)(())
6 =0,
(12)
holds for any the shift term â in the source ANN when 8 E
=1-1 2 7
Theorem indicates that for c = 0 the expected conversion error reaches zero even though N + T provided that the shift term J [- 2: )) The proof is in Appendix C.

---

Under review as a conference paper at ICLR 2023
Theorem 2. An ANN trained with SlipReLU activation function Eq. (10) is converted to SNN
with the same weights. Let Vi = v(0(0) vll 6, 51 T [ Then for arbitrary and N and arbitrary
th [0,1] the expectation of the conversion error of the proposed unified method
reaches the optimal ol ((
i.e.
12
<(Vile)
Ey ((Ere(O)) 6c(-
L
(13)
4T
holds for any the shift term 6 in the source ANN when 8 E
11 [ 2
Theorem indicates that for any V c E [0, 1]. the expectation the conversion error can reach the 47 provided that the shift term ö in the source ANN is in the interval |- 2 }) and
minimum <(V
61 T 1/2 The proof is in Appendix C. These results indicate we can achieve high-performance
converted SNN at ultra-low time-steps.
4.3 ALGORITIIM FOR TRAINING SLIPRELU ACTIVATION FUNCTION IN BACKPROPAGATION
Training an ANN with SlipReLU activation instead of ReLU also a challenging problem. Al- though the SlipReLU has constant slope as its derivative from Eq. (9), however, small slope values
c E [O, 1] can cause the gradient vanishing problem. Therefore, inspired by Bu et al. (2021); Bengio et al. (2013),
the surrogate gradient the derivative of the floor function d overall derivation rule is given as follows,
1. The
dJ dF
if E D2 otherwise
dz (2)
where D1 = [- 8,0). D2 = [-500 50), and 2!º is the i-th element of z ((), Then we can train the ANN with SlipReLU activation using Stochastic Gradient Descent algorithm, and convert it to the SNN. Refer to Appendix D for our proposed ANN-SNN conversion algorithm.
RELATED WORK
The first study of ANN-SNN conversion is proposed by Cao al. (2015), which convert the ANNs with the ReLU activation function to SNNs. Afterwards, Diehl et al. (2015) proposed data-based and model-based ever, it usually requires hundreds of time-steps for the converted SNN to get accurate results due to the error analyzed in Sect. 3. To address the potential information loss, the "reset-by-subtracttinn mechanism (Rueckauer et al., 2017 7), also called "soft-reset" (Han et al., 2020) rather than "reset- to-zero" is proposed. Recently, many methods and algorithms have been proposed eliminate the conversion error. Sengupta et al. (2019) proposed novel weight-normalizzttoo technique which considers the actual SNN operation in the conversion step. For direct conversion from a pre-trained ANN to an SNN, Ding et al. (2021) proposed Rate Norm Layer to replace the ReLU activation function in source ANN training, and Li et al. (2021) proposed calibration for weights and biases using quantized fine-tuning to correct the error layer-by-layer. Our work share similarity with Deng & Gu (2021); Bu et al. (2021) which are also on optimal conversion. Deng & Gu (2021) min- imized the laser-wise error by shift-threshold ReLU which only considers the deviation from the ReLU the unified optimization framework in Sect. 3. Bu et al. (2021) proposed to use the quan- tization clip-floor-shift activation function train ANNs and the clip-floor-shift activation function only minimizes the conversion error neglecting the performance loss of the tailored ANN with new activation function. They all got the theoretical "optimal" results with some fixed shift term. In comparison, our proposed unified framework gives more flexibility for different application scenar- ios to covert ANN into SNN with techniques eliminating the conversion error and keeping the ANN performance with less deviation from the regular ANN with ReLU. Our SlipReLU can balance the trade-off between the ANN performance and the conversion error simultaneously.
eeigl-nnrmmalization method convert three-layer CNN to SNN. How-
6 EXPERIMENTS
In this section, we compare our SlipReLU method with existing state-of-the-an approaches for im- age classification task CIFAR-10 (LeCun et al., 1998) and CIFAR-100 (Krizhevsky & Hinton,

---

Under review as a conference paper at ICLR 2023
Table I: Comparison between the proposed SlipReL.U method and previous works on CIFARIO. 10.
Architecture
Method
ANN
T=1
T=2
T=4
T=8
T=16
T=32
T=64
T=128
RTS
91.46
10.00
10.00
10.00
10.00
10.00
21.10
90.34
90.54
SNNC-AP
97.11
55.02
66.56
82.03
89.38
94.75
96.23
96.83
97.02
QCFS
91.29
56,60
67.53
79.73
88.06
90,99
91.73
91.69
91.56
ResNet-20
ReLU
93.71
11.58
12.54
16.05
36.47
70.84
83.47
85.93
86.46
Ours
93.37
80.30
82.80
84.69
86.12
90.45
92.95
93.49
93.52
RTS
92.09
10.00
10.00
10.00
10.00
10.00
89.48
91.84
91.25
RNL
86.10
10,00
10.00
10.00
10.00
10.10
17.20
38.56
60.20
SNNC-AP
95.93
73,82
75.16
86.58
90.26
92.87
94.53
95.41
95.78
VGG-16
QCFS
92.69
75.51
83.81
88.58
91.47
92.50
92.83
92.83
92.90
ReLU
95.92
10.00
10.00
11.51
70.97
88,39
93.05
94.76
95.19
Ours
95.60
85,40
86.59
88,27
89.67
95.20
95.66
95.65
95.66
RTS
91.94
10.00
10,00
10.00
10.00
10.00
11.00
89.60
90.14
QCFS
95.84
88.30
91.52
93.89
95.02
95.51
95.72
95.70
95.69
ReLU
96.71
11.00
25.07
55.21
73.80
88.44
94.50
96.00
96.50
ResNet-18
Ours
96.67
93.11
93.97
94.59
94.92
95.08
96.31
96.53
96.52
2009). Similar to previous works, we utilize VGG-16, ResNet-18, and ResNet-20 network structures for source ANNs. We compare our method with the state-of-the-and ANN-SNN conversion meth- ods, including RNL from Ding al. (2021), ReLU-Threesoold-hhiff (RTS) from Deng Gu (2021), SNN Conversion with Advanced Pipeline (SNNC-AP) from Li et al. (2021), I), and ANN-SNN con- version with quantization clip-floor-shift activation function (QCFS) from Bu et al. (2021), well as ANN-SNN conversion with regular ReLU activation function (ReLU). Refer to Appendix E for the network structures and training setups. We use SlipReLU with shift setting 81 0, 0.5, refer to Appendix H for ablation studies on SlipReLUU and SlipRel.Usshin activation.
6.1 COMPARISON WITH THE STATE-OF-THE- METHODS
Table shows the performance comparison of the proposed SlipReLU with the state-of-the-ar ANN-SNN conversion methods on CIFAR-10. 10. For ultra-low latency inference (T = or T 2), our proposed SlipReLU has the best performance compared existing of-te-of-the-art art ANN-SNN conversion methods. Specially, when the latency T 1, our SlipReLU method is able to achieve an accuracy of 93.11% for ResNet- 18, with a good margin compared to the next best baseline QCFS (88.3%%); the accuracy for VGG-16 is 85.40% with SlipReLU activation, while the next best curacy is 75.51% with QCFS activation. For ResNet-20,, we achieve an accuracy of 82.8% with 2 time-steps. Our proposed SlipReLU method indeed gives the best SNN accuracy for ultra-low latency inference. When considering low-latency inference (T < 4), our model outperforms almost all the other methods with the same time-step setting. Notably, our ultra-low latency performance is comparable with other mate-of-the-art supervised training methods, which is shown in Table S2.
The most competitive method of our SlipReLUU is the QCFS method, however, it cannot pro- vide as high performance as our SlipReL.U in terms of ANN accuracy, which can be seen from the ANN testing accuracy. Here the ANN accuracy of ReLU activation is the baseline. The results with SlipReLU activation shows that it has higher ANN accuracy than the QCFS (step function). The reason is that QCFS only considers the conversion error but not the ANN performance, while our SlipReLU proposes to consider the conversion eтror as well as the ANN performance. The highest ANN accuracy can sometimes achieved by the SNNC-AP method, which a one-step conver- sion method, however, the SNNC-AP usually fails to give moderate accuracy for low-latency SNN inference. Considering both ANN accuracy and SNN inference accuracy. the SlipReLU performs the best among all the other state-of the-ar art models and it is the closest to the regular ReLU acti- vation function, as our SlipReLU method is designed to consider both the ANN accuracy and the conversion error simultaneously. We further test the performance of our method on the large-scale datasets are reported in Table $3
dataset. Experimental results on CIFAR-100 100 and and Table $4 of Appendix G.
6.2 EFFECT OF THE SLOPE AND EFFECT OF THE QUASI-LATENCY N
In our SlipReLU method, the slope balances the weight of the threshold ReLU and the step func- tion, which affects the accuracy of the converted SNN. To analyze the effect of and better deter-
8

---

Under review as a conference paper at ICLR 2023
vggl6 on cifar10
re cifar10
vgg16 dffr100
re6. cifar100
0.2
0.40
2.6-
5MN 2000000
0.69
(x61
0.5
6-61
0.60
(x82
(40.
0.4
0402
0.75
0.4
0.70
0.5
0.3
(605
660
0405
0686
0.05
010.3
0.4
02-
C10
040.
03
0+09
0H00
6 64 324256
32 61 128256
32 64 326256
32
129256
(a) Influence of different slopes with the quasi-latency N = 2 on CIFAR-10 10 and CIFAR-100
vgg16 dfar10
resnet20 on cfar10
v9g16 cifar100
resnet20 citar200
0.-
0.8
0.6
0.6
5246 6000100)
co21
0.7
30.5
c40
0-61
0-0.2
0.4
140
0402
0.44
c025
016.5
0164
00.
5105
4104
0.65
0164
0.
210
0156
0.2
0.
040.3
0.2
0.2
C10.8
0409
0x00
6
16
1262255
8 16 32
129256
16 32
129256
16 32
129256
Timeddep
(b) Influence of different slopes with the quasi-latency N 32 CIFAR-10 and CIFAR-100 100
Figure 2: Effect of different slopes e with different quasi-latency N on CIFAR-10 and CIFAR-100.
mine the optimal value, we train VGG-16 and ResNet-20 networks with quasi-latency N 2 and N = 32, and then convert the trained ANNs to SNNs. The experimental results on CIFAR-10 10 and CIFAR-100 dataset are shown in Fig. 2, where each of the colored curves shows the effect of the slope on the SNN accuracy over different time-step/latency T, under different quasi-latency N.
Results in Fig. show that for small values of quasi-latency N, the slope has large effect on SNN accuracy for ultra-low and low-latency inference. In particular, for small quasi-latency N, different slope values e can result in different SNN accuracy when the time-step T is small. But for large values of quasi-latency N, the colored curves close to each other, and different values of slope c give similar results no matter whether the time-step T is small or large. This brings the flexibility to apply our SlipReLUU to different scenarios. When we need ultra-low/low- latency inference for the converted SNN, we choose small quasi-latency N, but when we do not care about the inference time (the time-step T can be large), we then choose large quasi-latency N. Refer to Appendix for more detailed results.
7 DISCUSSION AND CONCLUSION
The performance of the converted SNN is determined by both the ANN performance and the con- version error. The performance loss between the regular ANN with regular ReI.U and the tailored ANN has never been considered in the existing ANN-SNN conversion methods, which will be in- herited to the converted SNN. In this work, we formulate the ANN-SNN conversion optimization problem which considers the performance loss between the regular ANN with and the tailored ANN, well as the conversion eтrоr simultaneously. Following the unified optimization framework, we propose the SlipReLU activation function to replace the regular ReLU activation function in the tailored ANN. The SlipReI.U is weighted sum of the shift-threhodd-Rll the step function, which improves the performance of either as an activation function alone. The SlipReLU method covers a family of activation functions mapping from activation values in source ANNs to firing rates in target SNNs; most of the state-of-the-art optimal ANN-SNN conversion methods are special cases of our proposed SlipReLU method. We demonstrate through two theo- rems that the expected conversion eтror between SNNs and ANNs can theoretically be zero on a range of shift values E 1- 11 rather than a fixed shift term 1. enabling us to achieve converted SNNs with high accuracy and ultra-low latency. We evaluate our proposed SlipReLU method on CIFAR-10 10 dataset, and the results show that our proposed SlipReLU outperforms the state-of-the- art ANN- SNN conversion in both accuracy and latency. To our knowledge, this is the first work to explore high-perfoneee ceth-peeffrmmacee ANN-SNN conversion method considering the ANN performance and the conversion error simultaneously, with ultra-low latency, especially for time-step (T 1).
unified
and

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1rxXiv:308.344 3432, 2013.
Léon Bottou, Stochastic gradient descent tricks. In Neural networks: Tricks of the trade, pp. 421- 436. Springer, 2012.
Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ANN-SNN conversion for high-aceuracy and httrallowllatency spiking neural networks. In International Conference on Learning Representutions, 2021.
Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1)544-66, 2015.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 123,, 2019.
Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. International Conference on Learning Representations, 2021.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 1708.04552. 2017.
Peter U Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. ieee, 2015.
Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ann-snn conversion for fast and accurate inference in deep spiking neural networks. In International Joint Conference on Artificial Intelligence, pp. 2328-2333 2021.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. RMP-SNN: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), pp. 13558- 13567, 2020.
Alan I. Hodgkin and Andrew F Huxley. A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of physiology. 117(4))500, 1952.
Eugene M Izhikevich. Simple model of spiking neurons. IEEE Transactions on neural networks, (116:1166--172,,000 2003.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. https://www.cs.trrmtoooek edu/ kriz/cifar:tml, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-233) 1998.
Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ANN: towards efficient, accurate spiking neural networks calibration. In International Conference Machine Learning, pp. 6316-6325. PMLR, 2021.
Ying-Hui Liu and Xiao-Jing Wang. Spike-frequency adaptation of a generalized leaky integrate- 2001. and-fire model neuron. Journal of computational neuroscience, 10(1):25-45, 2001.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. 2017. URL https::t//// http///ppeeeeeeeeeeeeeeee net forum? d
Patrick Mehlitz and Matúš Benko. On implicit variables optimization theory. Journal of Nons- mooth Analysis and Optimization, 2, 2021.
10

---

Under review as a conference paper at ICLR 2023
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 2019.
Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep
spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXhv:2005.0180) 2020.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike- spike-based machine intelligence with neuromorphic computing. Nature, 575(7784)x07-6117, 2019.
Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Con- deep networks to efficient event-driven networks for image classifi-
version of cation. Frontiers neuroscience, 1 11:682, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1153))211-222, 2015.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: VGG and residual architectures. Frontiers in neuroscience, 13:95, 2019.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training gegh-performanc spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function spiking neural networks. Neural computation, 33(4):899-925, 2021.
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagattin for deep spiking neural networks. Advances Neural Information Processing Systems, 33: 18022-123233 2020.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11062-11077 2021.
NOTATIONS IN THE APPENDIX
Throughout the paper and this Appendix, we use the following notations in Table S1. Bold-face
lower-case letters refer vectors, and normal-face letters refer scalars. Note v( tth and ( are vectors whose dimensions match the number of neurons in the layer of interest, and denote vi()) [v:[0] and g( [Ø(*) respectively. Namely, vector v(t) [v-(0] means that each
element is the same (e) Denote 5 = =0].
Table S1: Summary of notations in this paper.
Symbol
Definition
Symbol
Definition
N
Quasi-time-steps of ANNs
FANN (-)
ANN activation function
T
Total time-steps of SNNs Activation values ANNs
FSNN (-)
SNN activation function
(
(())
Spike outputs of SNN
(0)
Average PSP of SNNs
(()
PSP released by 1-th layer Membrane potential after firing
(
Trainable threshold in ANNs
v(0)
(8)
Firing threshold in SNNs
(())
Weight matrix
th
11

---

m mv(0 th = ( Under review as a conference paper at ICLR 2023
A ANALYSIS OF ACTIVATION FUNCTION IN SNNS
We will derive the activation function of SNN, F-NNN() in this section.
The activation function of SNN gives the relationship between activation values x((-1) and x( of successive layers of SNN, which defines input-output function mapping for adjacent layers.
Specifically, we can get the potential update equation by combining Eq. (2) and Eq. (4),
(0 (t) = ( (t - 1) + w(x) - (t)
(A.1)
By summing the time-step from time 1 to T. then we get
T Σ s (DV
(8) (T) (0) = w Σ (e (t)
(A.2)
t=1
Due to the inkke----spike-out property of the IF neurons in SNN, the output at each time-step be ether 1. For each neuron i, let m, = --= 8i (1), and each m, E {0, 2, ,T) T denotes the total number of spikes of each neuron i. Then m = {m:} is the vector collecting all the number of spikes of all neurons in the f-th layer. The accumulated spikes m Ei=1s()) (()) denotes the total number of spikes. According to the above equations, we have
v (T) (00 (0) w ((-11) mv(1)
(A.3)
Then we get
(A.4)
A.1 ELEMENT-WISE VERSION DERIVATION
Denote
( (e-1)
We use Zi vi (T), (0), and m, to denote the i-th element in vector (0), (T) v()()) and
m respectively. That is, (0) fzz ( ((0 [v. (T)), (() = {v. (2) (0)). and m {m,).
(T))))),)))))))))())))
Then we have
mV(() T2(() - (T) (0)
(2) = Tz. (0) - (v!2 v (0)) (For each neuron i with m = {m;}. z = 1
m,
Note that we assume the terminal membrane potential v(() (T) lies within the range [0, Vill),, ) by
further assuming v (0) O, get
(0) (T) < v(0 0 < V
th v(() <v
(00 (T) 0 (adding Tz," (2) to each term)
1/ vl( <T>, - (9)(T) < Tz (m, Tz, (f) - -v, (T)) (T))
T2 (f)
< m() < Tz, (8)
th
T2.7 (1) Vin (1)
Tz (8)
m;
<
(() th
(
th
Then we use floor operation and clip operation to determine the totoal number of spikes, mi.
Tz
(2)
m, clip
T'
and m, -15(0))
(((
(0)
clip
1
.0.1
(and xX9) 520)s00)
T
V((e)
Tz (0)
x
0
(1(0) clip
T
()
,
.0.1
12

---

Under review as a conference paper at ICLR 2023
The assumption v; (0) 0 may be too strong, without it, we will get
T7. (2) V-(1)
th + v < my mmv( & T7, + +v
T2.
(6)
v- + V; (0)
T4 (0) + (0)
m E
V(t)
v
th
th
Tz:
(0)
th
(E)
Tz, (8)
(E)
+ <
m
&
Vi(e)
+ 8
with
8
V (E)
th
th
(2((0) (0) Vrb
Denote
Then we have
Tz.
m, clip
0,, (and m, -
((()
th
Tz (e)
(0 clip
1
( (and x!() Vih v((0s(())
.0,
((()
( vth clip
1
Tz
(2)
+
00,1
T
(t)
The relationship between activation values x((-1) and x(9) of successive layers of SNN can be formulated as
(9
Cliip
Tz
+ 8 0,1
th
T
vih
A.2 VECTOR VERSION DERIVATION
The accumulated spikes E (1) denotes the total number of spikes, and m {m,} is the vector collecting all the number of spikes of all neurons in the E-th layer. Each m, {0, 1, ... T} denotes the total number of spikes of each neuron i. According to the above equations, we have
v(0 (T) (0) = w(O x( mV/e th
(A.5)
Then get
( th = -v
(v(9
(0).
(A.6)
Note that we assume the terminal membrane potential (T) lies within the range [0, vi(l). by further assuming v(0)(0) O, we get
E v < v
+ vi < (v 0
(e-1) (1(()
th < <TW +
& ((())(--
TW( (e-1) ((() mm(0
<TW()
th
TW (E)
((())
(0)
tt
Then we use floor operation and clip operation to determine the totoal number of spikes, m,
(e-))(E-))
clip
0.
(and Ts(
((()
(")
(1) clip
.0,1
( (nnd x(1) = Vih v(0)g())
T
v(()
1
Tw
(e-11
(0) (-00)
.0.
T
(1)
13

---

Under review as a conference paper at ICLR 2023
The assumption v (0) 0 may be too strong, without it, will get
TW(0x v(l v() < my 19)
& (T ())
TW(0g(1-1) V( + v()
7W(0x(1-1) + (0)
V
th
TW(0x(--1 v(()
v()
+8<ms
with 6
V
((
(E)
(0) Vi
Denote 8
Note å is vector whose dimension matches the number of neurons in that
layer. Then we have
(W(0)(e-1)
clip
+ 0
.0,7)
(and m = =75(0)
V-0
th
( = clip
+
.0,
(and x(() = -v100/
T
th
1
x*
X 0 Vilo clip
+ .0.1
T
th
The relationship between activation values x(e-1) and x(() of successive layers of SNN can be formulated as
( v:l0) clip
TW
+ 8 .0.1) 00.1)
T
v(
th
Note (((() is vector whose dimension matches the number of neurons in that layer, and v th means each element is the same (-())
Denote
z (
(e-1)
Then
Tz(1)
x(1
(
clip
+
0,01
T
(-(
th
B DERIVATION OF SLIPRELU ACTIVATION FUNCTION
In this section, we will give detailed derivation of the proposed SlipReLU activation function Eq. (9) and extension in Eq. (10) with different shift modes. In ANNs, denote
(0xx
Then the forward propagation of activation values through layers in the ANN is
( = (2) = FANN (())))
B. DERIVATION OF SLIPRELU ACTIVATION FUNCTION
Derivation of LL activation function Eq. (9). SlipReLU function in Eq. (B.1),
We start with the initial definition of the
if z (4)
< 0
(()
cz(() + (1 c)
kg(e)
if k
(() if (
(k+1)0))
(B.1)
N
N
(
Here 0,1, .N 1. Note (() should a vector whose dimension matches the number of neurons in that layer, (00) [0t0]]
14

---

Under review as a conference paper at ICLR 2023
Then rewrite it to
SlipReLU(Z()) Ytemp + (Zremp (
Ytemp)
CZtemp + (1 c)ytemp
where
Zremp
g(0)
010) 0.1 ) and Stemp
((0)
N Zremp
N
((0)
Here
((2)
N Zremp ((0)
y Yromp
N
((0)
2((0)
Yremp
N clip
0(e) 0,
N
z(()
1
Ytemp
clip
N
N
O(e)
,0,
1
(z
Ytemp clip
.0.1
N
0(0)
Then Eq. (B.1) can be written as follows,
(
SlipReLU(Z!!)
Cheemp + (1-c)ytemp
(1)
Nz
= clip
0,
+ (1 (1-c)0(0cig
.0,1
016)
0(t)
That is the SlipReLU activation function in Eq. (9).
z(()
Nz 00) ,0,1
FANN co(0)lip
Oio 0, (1 (1-c))0
c)
B.2 SLIRRELU ACTIVATION FUNCTION WITH DIFFERENT SHIFT MODES
Derivation of SlipReLUU extension in Eq. (10) with shift As mentioned in Sect. 4, the SlipReL.U activation function in Eq. (9) in a weighted combination of the threshold-Rel.U (first part) and the step function (second part), with the slope & c balancing the weight, then any shift to these two parts will lead to shift in the SlipReLU activation function. The SlipReLU extension with in Eq. (10) can be formulated follows.
a = (Z) clip
(
(()) + 61.00.1)
+ (1-
(1
1 Nz
0(0) + 0,1
The shift term 51 [-N, 0) and E 1-2, 2 for the source ANNs. And ö [81]. 5 [8]. Here we list several examples of the proposed SlipReLU with different shift modes.
Mode We set 5, = = 0, then
(2() 0.1) + (1 (1- -c)o(0clpp
1 (t)
( = (1)) clipp
.0.1
N ((2)
Mode We set 0,, then
(1) = FAN cll
00 0, - +1- (
1 (2
1 0,1
N
Q(t)
B.3 SPECIAL CASES OF THE SLIPRELU ACTIVATION FUNCTION
Here list four different special cases of the proposed SlipReL.U.
15

---

Under review as a conference paper at ICLR 2023
Threshold-ReLU
When = 1 and 61 = 0, the SlipReLU becomes the threshold ReLU activation function which is studied in Deng & Gu (2021).
Shift-threshold-ReL.U When
and 61 -1/(2N), the SlipReLU becomes the shift- threshold ReLU activation function which studied in Deng & Gu (2021).
Quantization clip-floor (QCF) When 0 and 5 0, the SlipReLUU becomes the quantization clip-floor (QCF) activation function which is studied in Bu et al. (2021).
Quantization clip-ftoor-shift (QCFS) When 0 and 1/2, the SlipReLU becomes the quantization clip-floor-shift (QCFS) activation function which is studied in Bu et al. (2021).
C PROOF OF THEOREMS
Before we proof Theorem and Theorem 2, we first introduce an important Lemma.
Lemma 1. If random variable E [0.0) is uniformly distributed in every small interval
(me, m++1) with Pt (t 0, o... ,T') where mo 0, 0.T++1 0,m, (21-1)6 for 1,2, T. Po = pr. For any value [- 21 }1 then we can conclude that 2T
Ex r 0 Tr
+
= 0.
(C.1)
T
Proof. We consider in different small intervals (m,,,++1) m.+1).
(1) For r E (0, 2T),
Tx
Tx
0 x
8<
+ < +
+
00
2T
2
(2) For x E
(2t- 1)0
12++1)0
and 1=
....... 1.,
21
(2t 1)6
(2t + 1)0
Tz
Tx
< I
+
+
<
+
+0
+ 8
= t.
2T
2T
2
0
2
0
(3) For
(2T-1))
0
2T
(2T - 1)0
1
Tx
1
Tx
<
x <
-
+
+0
+
+
+
+
+ = =T.
2T
2
f
2
8
Then we have
0
Tx
1)
E
+
T
0/2T
0
Tr
(2++1)0/2T
0 Tx
Po
T
T
+
da + Σ Pt
121-1)0/2T
(21-
T
+
dr
0
Tr
+
Pr
+ 8
dr
T
(2T-1))0
1)0//T
T-1
(20+1)8/22
Po
|0|| dr + Σ
F T
dr + pr
x 0 dx
(21-1)0/22
(2T-117) 11)0/22
6/2T
T-1
(21+1)0/22
10
Po
vdr.
+
Pt
dx + PT
(x 0) dx (2T-1)0/2T
(21-1)0/22
T
1=1
@2
8
=P0
+ 0
PT
0.
87"2
87
16

---

Under review as a conference paper at ICLR 2023
Lemma 2. Let P be a probability distribution R. If random variable R and z P.
function : z - gz)) E IR" and g(z) 2 almost surely for v z E D, and
Eg g()) = 0
then we have
E, (ggg2)) 0
Proof. By the definition of L2-norm, we have
(z(2)]]2 - V )/(()) + (3(2) +... g2(2) < (2) +g2=2|]+++++++ (9n()))
Then, get
Ex [1(22)l1 < Fz 9n (=)] + E [2(()
Ez (=)]
= Exgi (z) + Ex02(2) ................................ + Ez = 0.
Then
Eg (lg(2))112 =
C.1 PROOF THEOREM
For Theorem I, we need to prove
A TT, L Ex (EEr(()
661--.22
Proof. The activation function of the SNN is
(z + v00
(z) (E( th clip
0011
T
((0)
For 0, the SlipReLU activation function used in the source ANN then becomes
Nz
(2) = (00)
+
...1
N
0(0)
With v:(() = =0((), , then the error becomes
010)
Nz(()
v(O)
Tz + +v(0(0)
(t) (2 F2NN(2(0))
+ +
th
N
((2)
T
(t)
Then
Ex
Frr(())
66 +.
((E)
(e)
Vil)
(t) + (1)
-En
+
N
0C0)
T
V-
th
((E)
(2
(2 v(0
< Ez
N
(())
+ 8
(0)
OEI
2.21
2.
+ Ex
(()
T
th
(
th
Denote (6) (0) and 7; the i-th element of vector v (0) and z. Denote 8 10). consider every element of vector
Then we need to
g(
Nz ( ((0)
(()
Tzj
(8)
v (0)
Ez,
N
(e)
th
Er.
g(
N
NZ
((E)
(8)
)
6ET
2.21
EE
2
(0)
(f)
Tz (8)
V((0)
+ (0)
1)
(C.2)
17

---

Under review as a conference paper at ICLR 2023
Then according to Lemma 1, we have
(
No.
(0)
Ez
N
B(0)
5
(8)
6E1- =0
dEI
Ex
(2)
vi(t)
th
T7,
+ ( (0)
1) 10
10000--005 (0)= = 0.
T
v((1)
This holds for any shift value 8 in the ANNs when the Theorem I.
2/2
$ 8 < 1. which gives the conclusion of
VT,L
Eg
(xr(()) del- 2.21 = -0.
))
C.2 PROOF OF THEOREM
For Theorem 2, we need to prove.
TT,L Eg (EET!O)) [c(
th
(C.3)
4T
Proof. The activation function of the SNN is
Tz + (0)
(0)) =
v+(*) th clip
.0.1
T
For arbitrary E [0, 1]. the SlipReLU activation function used in the source ANN then becomes
FANN(z (0) clip (x() 0.10.1) + (1-
(
N2
+ 5 0.1
.0.1.)
N 0
With ((() = =())) then the error becomes,
Err FAN) (2))
@l0p (2(0)
Tz(t) + v(0(0)
((0) + t51,0,1
(100 clip
.0.1
T
(1())
1
(z)
1
Tz + vv()(0)
+ +11 c)
+
8
.0.1
vib clip
.0.1
O(e)
T
v(0
th
(2)
(0)
10 + 8,100)
th
(with (0)) (e) 8, v(() 0(0)
T
vih
(0()
(2 + (0)(0)
V(() T2(9) + v(()
+ (1 c)
N
O10)
(t)
V(()
(t)
2 (8) + 1.00
61
th
+ 8
Err1
(C.4)
T
(
(00)
(2)
V-(
(2)() + v(0(0)
+ (1 c)
+ 8
th
P (1 c) Err (C.5)
N
((A)
T
(2)
tt
Then
Eur (1)
Err, + (1 c) -Err
Emr( (0)
|c- Err, + (1 c) Erry < c. |Err]] + (1 1--c -
18

---

Under review as a conference paper at ICLR 2023
So minimize the whole error by minimized each of the two terms.
Let ô1
o+6 For Eq. (C.4), we have
T.
V (E)
V-(t)
Tz (
Err1
th
(Φ
+ 6)
+
T
V (2)
V(()
(2(
2 (1)
th
th
(C.6)
T
T
Vth
Here
((())
th
Φ
is the activation function of ANN
T
((2)
(2
is the step activation function of SNN.
T
V(t)
This Eq. (C.6) recovers the loss of the shift-threshold ReLU (with a shift value () and the step function, which is the same as Deng & Gu (2021). And as shown (A) of Fig. 1, the conversion error is the shaded area. The error between the activation function (of ANNs) and the step function (of SNNs) obtained by summing up of all the shaded area together, which the ANN-SNI conversion error.
Then the objective becomes minimize
T 2
V (E)
(1)
2
(e)
27
(V(0)
)
1
min (E
Erri]]
min
+
Φ
T
4T
Then
1/2 6
81
(C.7)
T
And the minimum L2-norm of the first error becomes
(2)
Ez (Err;
4T
For Eq. (C.5), with (0)
th 8, (() O(0)
(1)
we have
Erra A (())
(2)
((()
Tz + (1)
th
N
O(c)
((
From Lemma 1 and Theorem 1, we have
(())
(2)
v-(0) T
(z(
+ (0)
Ez ((Erzz)) = Ez
dEl- 6el- 2.
N
((2)
(())
th
=0.
Then
Eg (Er(")) 1) = Ex (FANN (Z)N(((()) Ez (lc ErTi + - c) Erral)
< c Ez (Erril) 1) + cc) Ez ((Err2))
(v))))
+ -)
4T
(()
47
This concludes the Theorem 2.
c(V/10))
A TT,L L E, (ETT(0)
4T
19

---

Under review as a conference paper at ICLR 2023
D PSEUDO-CODE FOR THE UNIFIED ANN-SNN CONVERSION ALGORITHM
Here the pseudo-code for our proposed unified ANN-SNN conversion algorithm.
Algorithm 1: Algorithm for ANN-SNN conversion.
Input: ANN model structure fANN (x; W) with initial weights W (W Quasi-latency
N; Shift value from the interval å E [-2, ž]: Initial dynamic threshold {0())
Learning rate
Output: SNN model (x; W)
Data: Dataset D
for 8 = 1 to TANN-laeer do
if is ReL.U activation then
Replace ReLU(x) SlipReLU(x; N, (()))
if is MaxPooling layer then Replace MaxPooling layer by AvgPooling layer
for e epochs do for length of Dataset D do Sample minibach (xx)))) from D
for é 1 to JANN- do
L x S N,000)
Loss = CrossFntropyj(x00..y
11
12
for l fANN Jayers do
( - W c aLoss (2)
0(0) t g(t) BLoss 2013)
14
15
for l = 1 to fANN layers do
SSNN. W + JANN- W(
16
JSNN.V ( JANN .(E)
17
JSNN.V (0))) JSNN. v
x 6
Return JSNN
20

---

Under review as a conference paper at ICLR 2023
E EXPERIMENTS DETAILS
E NETWORK STRUCTURE AND TRAINING SETUPS
There are three steps in proposed ANN-SNN conversion,
Step 1:
Tailor the ANN;
Step 2: Train the tailored ANN;
Step 3: Convert the trained ANN to an SNN.
In the first step, we first replace max-pooling with vveaae-ppolling and then replace the ReLU activation with the proposed SlipReLU activation function. The tailored ANN is also called the source ANN. In the second step, train the tailored ANN. After training the tailored ANN, we copy all weights from the trained-tailored source ANN to the converted SNN, and set the threshold v- in each layer of the converted SNN equal to the threshold value g(() of the source ANN in the vL same layer. Besides, we set the initial membrane potential (0) in converted SNN as vth ā to match the optimal shift 5 of the SlipReLU activation in the tailored source ANN, where the optimal shift 5 can be any value in the interval 8 E | 2 21.
Common data normalization and some data pre-processing techniques are used in the experiments.
For example, we resize the images in the CIFAR- --00 datasets into 32 x 32. Besides, random cropping images, Cutout VDevs Taylor, 2017) and AutoAugment (Cubuk et al., 2019) used for all datasets. The Stochastic Gradient Descent (SGD) optimizer (Bottou, 2012) is used in the experiments with a momentum parameter of 0.9. We set the initial learning rate to 6 == 0.1 for CIFAR-10 10 and CIFAR-100. We use cosine decay scheduler (Loshchilov & Hutter, 2017) to adjust the learning rate with a weight decay 5 x 10 10-4 for CIFAR-1O/CIFAR-100 100 datasets. All models trained for 300 epochs. When considering small quasi-latency N 1 and N 2, for models that can not be trained properly with learning rate = 0.1, we set the initial learning rate to 0.05 We train all the networks different settings; we set the quasi-latency N = 1 with the slope с = 0.3, 0.4, 0.5 for low-latency inference (I' & 8), and we set the quasi-latency N 4 with the slope We set 81 = 0, 5 2 for the SlipReLU activation for all the models and all the datasets.
for CIFAR- 1R-110/FFFR-O0M
CIFAR-IO/CIFAR R-00 dataset with two
0.9 for latency T 8.
As for the input to the first layer and the output of the last layer of the SNN, we do not employ any spiking mechanism as in Li et al. (2021). We directly encode the static image to temporal dynamic spikes input to the first layer, which can prevent the undesired information loss introduced by the Poisson encoding. For the last layer output, we only integrate the pre-synaptic input and do not fire any spikes. We use constant input when evaluating the converted SNNs.
E.2 INTRODUCTION OF DATASETS
CIFAR-10: The CIFAR- 10 dataset (Krizhevsky & Hinton, 2009) consists of 60, 000 32 x 32 color images 10 classes of objects such as airplanes, cars, and birds, with 6, 000 images per class. There are 50, 000 samples in the training set and 10, 000 samples in the test set.
CIFAR-100: The CIFAR-100 dataset (Krizhevsky & Hinton, 2009) consists of 60, 000 32 color images in 100 classes with 6, 000 images per class. There are 50, 000 samples in the training set and 10, 000 samples in the test set.
Imay-ImageNet:
Imageem (Le & Yang. 2015) is subset of ImageNet- 1k (Russakovsky et al., 2015) with 200 classes. Training data contains a total of 100, 000 images, 500 images from each class. The test data contains total of 10, 000 images, 50 images per class. The dimension of the images is 64 x 64.
F COMPARISON WITH THE STATE-OF-THE-ART SUPERVISED TRAINING METHODS ON CIFAR-10 DATASET
Our proposed SlipReLU method is comparable with other ofate-of-the-art supervised training meth- ods in terms of the ultra-low latency performance. Table S2 reports the results of the proposed models against the state-of-the-an supervised training methods on CIFARIO dataset. These state- of-the-art supervised training methods include Hybrid-Conversion (HC) from Rathi al. (2020),
21

---

Under review as a conference paper at ICLR 2023
STBP from Wu et al. (2018), TSSL from Zhang & Li (2020) and GDDP from Zheng et al. (2021), which are back- propaggatontnonn or hybrid training methods.
Our approach for CIFARNet achieves accuracy of 95.31% with time-step T' = 4, and the achieved accuracy is higher than any other supervised trained models. Sufficient time-step required for back propaaappagiinn to train the SNN directly. The hybrid training method involves training the verted SNN model using back- proopprrpagation as a second step, still it requires 200 time-steps to achieve good accuracy, which is high compared to the time-step required for our method. For VGG-16, the hybrid training method requires 200 time-steps to obtain 92.03% accuracy, whereas our method achieves 91. 08% accuracy with time-steps.
Table S2: Comparison with state-of- oatte-of-thhaart supervised training methods on CIFAR-10 dataset.
Model
Method
Architecture
SNN Accuracy
Time-step
HC
Hybrid
VGG-16
92.03
200
STBP
Backprop
CIFARNet
85.82
12
GDDT
Backprop
CIFARNet
87.35
TSSL
Backprop
CIFARNet
88.23
5
Ours
ANN-SNN
VGG-16
91.08
Ours
ANN-SNN
ResNet-18
92.86
2
Ours
ANN-SNN
CIFARNet
95.31
G RESULTS ON CIFAR-100 DATASET AND TINY-IMAGENET DATASET
We report the results on CIFAR-100 in Table S3, and the results on Tiny-ImageNer in Table S4. From Table S3, we that our SlipReLUU method also outperforms the others both in terms of high accuracy and ultra-low latency. For VGG16, the accuracy of the proposed method can achieve an accuracy of 64.2 64.21% which is 29.1% higher than QCFS and 39. 99.98% higher than SNNC-AP when the time-steps is only 1. For ResNet-18, when T = 1, we can still achieve an accuracy of 71.51%. These results demonstrate that our method outperforms the previous conversion methods.
Training an SNN on large-scale dataset such as Iny-emmeeNeet is considered to be one of the chal- lenges the SNN literature. From Table S4, we can infer our SlipReLU method outperforms other baselines in terms of the SNN accuracy when the time-step T is & 4. When the time-step is 1, for ResNet-34 our SlipReI.U method achieves an accuracy of 40.55% which 6.94% higher than the baseline QCFS (33.61% 1)). For VGGI6, our SlipReLU method outperforms other baseline methods with the accuracy of 43. 73% when time-step is 1, which is 12.14% better than the baseline QCFS (31.59%) ) and 32.93% better than the baseline SNNC-AP (10.80%).
H COMPARISON OF SLIPRELU AND SLIPRELU-SHIC SHIFT ACTIVATION
Here further conduct ablation studies on SlipReLU and SlipReLU-shift, by comparing the per- formance of SNNs converted from ANNs with Slip activation and ANN with SlipReLU-shift activation. Sect. 4. we prove that for arbitrary T and N, the expectation of the conversion error reaches 0 with SUspRt activation function when T and N and arbitrary e E [o, 1), the expectation of the conversion error of the proposed unified method reaches the optimal e(V.,)))) (44T). To verify these, we set N = 1, 8, 16, 32 and train ANNs with SlipReL.U activation and SlipRel.U-shin activation, respectively.
O. We also prove that for arbitrary
Fig. S1 shows how the accuracy of converted SNNs changes with respect to the time-step T under different quasi-latency N settings. The accuracy of the converted SNN from ANN with SlipReLU activation (in the first and third columns) first increases stays flat for time-step T < 4. and then decreases rapidly with the increase of time-steps, because we cannot guarantee that the conversion error is zero when e O. The best performance still lower than the SlipReLU-shif activation. The non-shifted SUipReLU activation shows no advantage for ultra-low latency inference when T & 4. In contrast, the accuracy of the converted SNN from ANN with SlipRel.U-shin activation (in the
22

---

Under review as a conference paper at ICLR 2023
Table S3: Comparison between the proposed SlipReI.U method and previous works on CIFAR-100.
Architecture
Method
ANN
T=1
T=2
T=4
T=8
T=16
T=32
T=64
T=128
RTS
66.05
1.00
1.00
1.00
1.00
1.00
9.54
63.22
62.49
SNNC-AP
81.08
16.16
25.71
49.60
65.97
75.80
76.64
80.63
80.90
QCFS
65.08
15.91
23.93
40.57
57.59
64,90
66.65
66.74
66.67
ResNet-20
ReLU
70.18
1,28
1.16
1.76
2.91
4.03
6.17
8.95
11.66
Ours
69.45
47.08
51.34
54.51
56.00
62.26
68.71
70.06
70.29
RTS
70.59
1.00
1.00
1.00
1.00
64.13
69.88
68.74
SNNC-AP
76.17
24.23
40.16
49.86
62.97
69.89
74.36
76.51
77.63
QCFS
71.50
35.10
43.85
53.66
62.72
68.47
70.99
72.01
72.23
VGG-16
ReLU
73.39
1.00
1.53
15,55
28.56
46.03
62.42
70.05
72.16
Ours
75.25
64.21
66.30
67.97
69.31
70.09
71.84
74.61
75.13
RTS
50.01
1.00
1.00
1.00
1.00
1,00
15.70
44.64
41.17
RNL
56.98
1.00
1.00
1.00
3.40
17.30
33.50
40.90
43.38
QCES
76.68
49.19
58.65
68.31
74.46
76.70
77.24
77.37
77.43
ResNet-18
ReLU
77.16
1.00
1.64
4.99
11.40
34.08
60.44
71.90
75.63
Ours
78.56
71.51
73.91
74.89
75.40
75.46
77.79
78.24
78.55
Table S4: Comparison between the proposed SlipReLU and other methods on Tiny-ImageNet.
Architecture
Method
ANN
T=1
T=2
T=4
T=8
T=16
T=32
T=64
T=128
QCFS
56.87
33.61
41.11
47.95
53.98
56.76
$7.65
57.53
56.94
ResNet-34
Ours
53.32
40.55
45.95
50.64
53.28
54.38
54.61
54.08
54.54
SNNC-AP
57.15
10.80
18.51
28.88
41.37
49.29
53.55
55.80
56.73
VGG-16
QCFS
55.59
31.59
43.43
49.72
54.08
55.48
56.02
55.97
55.93
Ours
52.75
43.73
47.95
51.23
52.89
54.01
53.84
53.74
$3.68
second and fourth columns) increases with the increase of time-step T. It converges to the same accuracy when the time-step is larger than 16. The SlipRel. activation shows advantages for ultra-low latency inference when 7 & 4.
I EFFECT OF THE SLOPE C AND THE QUASI-LATENCY N
In our SlipReLU method, the slope balances the weight of the threshold ReLU and the step func- tion, which affects the accuracy the converted SNN. To analyze the effect of e and better determine the optimal value, we train VGG-16/ResNets and then converted the trained networks to SNNs. The experimental results on CIFAR-10/100 dataset are shown in Fig. S2, where each of the colored curves shows the effect of the slope on the SNN accuracy over different time-step/latency T, under different quasi-latency settings. Table S5, Table S6 and Table S7 are the detailed data used to plot the curves.
networks with quasi-latency N 1, 4, 8, 16, 32,
J FUTURE STUDY
Remark 2. Our unified conversion framework exploits both the one-step conversion mechanism
and the two-step conversion mechanism. The one-step conversion method uses a pre-trained source ANN, such Li et al. (2021), however, the two-step conversion method needs to redesign the acti- vation function of the ANN to get a tailored source ANN, train it and convert il to SNN, such as Deng & Gu (2021); Bu al. (2021).
Remark 3. Usually, implicit variables of optimization problem are variables which do need to be optimized but are used to model feasibility conditions (Mehlitz & Benko, 2021), and they are often interpreted as explicit ones (Mehlitz & Benko, 2021), I), by using union image associated
23

---

Under review as a conference paper at ICLR 2023
with given set-valued mappings to make the implicit variables explicit variables, which can be interesting future work but not what we are interested in this paper.
As mentioned in Sect. 3.1, the multi step output feature of SNN implies that higher-lateney output depend on the outputs of all previous time-steps, which can be explored through multi-task learning. Therefore, reasonable use multi-task learning for ANN-SNI conversion where the different time-steps can be seen different but related tasks.
24

---

Under review as a conference paper at ICLR 2023
vgg16
vgg26
5WA 90000000777
0.6
0.6
0M62
0.3
04
SWN
0.4
0604
0.
6 15 22 $4 129295
32 Time-tep
26
128226
25 22
128256
15 31 Time-cepp
64 120259
Time-tap1
Tim-rrpp
Compare SlipReLU activation with/without shift when the quasi-latency N
with shift
0.95
0.93
0.8
SWN accurtty
0.85
1.6
0.80
06
8 32 04 226236
32
120256
18 32 126255
28 Time-tep
32
120216
Timestep
Timestes
Time-steg stop
(b) Compare SlipReLU activation with/without shift when the quasi-latency N
vgg16 without shift
v9o16 with shift
resnet20 without shift
resnet20 with shift
SP94 000001C
2.5-
0.6
0.11
0.4
CH05
0606
0.2
0H08
0-00
128255
16
(c) Compare SlipReLU activation with/without shift when the quasi-latency N
vogl6 shift
vag16
resnet20 without
reshet20
0.p
0.8
0.8
D.
SWN scourcey
16
0.6
046.3
0484
0.44
C40
05
0484
016.5
C10
04
016.
0-6-3
(10
C10
0.3
C-67
126256
129259
129236
TIme-dap1
Time-tap 7
Tme-dapp
Time-siap
(d) Compare SlipReLU activation with/without shift when the quasi-latency
shift
resnet20 without shift
resnet20 with shift
c.0.3
CH0.2
CHDA
SW u0c-r000
CHAS
6,6
0.6
016.2
0-6-3
CHOLE
0.4
0.4
016.5
C40.5
CH
046.8
CH06
046-2
0407
0.2
2483
02
18
326255
16 32
158220
38 32 125158
16 32 1202ie
(e) Compare SlipReLU activation with/without shift when the quasi-latency N 16
vgs16 without shift
vpg16 with shift
resnet20 without shift
resnet20 with shift
0.8
0.8
CHE2
CH0.1
0.G
0402
2403
C404
0.4
0.1
02-
0.2
2-28
0.2
0.2
CL04
6-38
225255
119235
Time-tea
(f) Compare SlipReLU activation with/without shift when the quasi-latency N 32
Figure S1: Ablation studies on SlipReLU activation and SlipReLU-shift activation on CIFAR-10 dataset under different slopes with different lusec-latency N.

---

Under review as a conference paper at ICLR 2023
vgg16 offr10
resnet20 cfar30
on
resnet20 cifar200
0.8
0.6.
5M0 300u000
0.7
29.5
0.6
040.2
CHO
0H62
0.4
0.5
040.3
2464
0.3
240
190
033
0+63
0604
02
CH0.8
0.2
046.7
0.2
140
6 15 32 64 129255
8 26 32 54 128256
2
25 52 128255
15 31
120259
Time-tap
Time-tep 7
Tim-dopp
Time-cepp
(a) Influence of different slopes with the quasi-latency N on CIFAR- 10 and CIFAR-100
v cifar10
cifar10
vgg16 dffr100
resne120 cifer100
0.55
0.3
0.90-
0.5-
SNN a0
0.65
0.8-
0.5
0.00
0.3
C10
(40
0.75
0.3
0.3
6404
640
0.4
090
03
36 32 64 125256
32 04 126256
18 32
126255
28
32
138226
Time STRp
Time-tep
1tp-sttp 7
Sine-stpp
(b) Influence of different slopes with the
2 on CIFAR- 10 and CIFAR-100
vgg16 difr10
resnet20 on cfar10
vgg16 cifar100
resnet20 cifar100
0.7
Sk0t acarro)
0.0.
(.0.1
86
G:22
0.0
0-8-7
0.
0.4
046.
0.6-1
4.04
1405
0.
CH05
140
0606
C40
CH08
1409
0-00
178255
0 16
120256
129256
12
128256
(c) Influence of different slopes with the quasi-latency N = 4 on CIFAR-I0 and CIFAR-100
vag16 efar10
resnet20 cPar10
vggl6 onf6r100
resnet20 cifar100
3.0
0.9
0.0-
0.8
0.5
5WN 2couroe
R.c
2.0.
0.6
0-0.3
c.03
C403
05
0403
21
0484
4404
0484
016.
0-633
0.3
010.
210
02
C-67
0605
0-6.
210
0.
0.3
128259
22
129236
Time Tmme-dpp1
Time-tap
Tme-dap1
Time-sapp
(d) Influence of different slopes with the quasi-latency N = 8 on CIFAR- 10 and CIFAR-100
dfar10
resnet20 cfar10
vgg16 cffar100
resnet20 cifar200
SW 000-0000
0461
c401
8.6
016.2
0-6-3
110
2103
0.6
0662
0.
8.4
0-26
c025
0.4
016.8
0.3
C40
CH66
CH
0.2
02
0.3
0483
0-622
C403
61
0686
0407
1
18
54 326255
16 32
158222
2
18 32 129258
16
32
1202le
Step
TimestepT
(e) Influence of different slopes with the quasi-latency N 16 CIFAR- 10 and CIFAR- 100
v9916 on cifar10
resnat20 en dfar10
vog16 on cifar100
mesne120 on cifar100
0.9
0.7
0.8
0.8
0.6
0.5
Ce0.1
205
50.5
2403
0.6
0402
6-11
C404
0.4
6-3
CJD5
0.3
02
02
62-
02
6-24
6-38
0.0
0.1
0:64
C.04
C104
110236
225255
11923
128255
Time-tea7
Time-tepp
(f) Influence of different slopes with the quasi-latency N = 32 on CIFAR-10 10 and CIFAR-100 100
Figure S2: Effect of different slopes e with different quasi-latency N on CIFAR-10 and CIFAR-100.

---

Under review as a conference paper at ICLR 2023
Table S5: Influence of different slope with the quasi-latency N 1.
Slope c
ANN Ace.
T=1
T=2
T=4
T=8
T=16
T=32
T=64
T=128
T=256
VGG-16 on CIFAR-10
c=0.1
90.93
10.00
10.00
10.00
10.00
10.05
22.78
69.91
87.55
87.72
c=0.2
88.68
10.00
10.00
10.00
10.00
10.00
16.25
34.22
48.16
68.69
c=0.3
86.73
10.00
9.74
13.37
22.29
43.48
68.81
81.81
85.71
86.35
c=0.4
91.90
10.00
10.00
10.00
10.77
32.66
68.87
88.74
90.77
90.55
c=0.5
90.86
85,40
86.59
88.27
89.67
90.67
90.93
90.91
90.81
90.59
c=0.6
90.97
81.35
85.18
87.21
89.32
90.32
90.79
90.70
90.52
90.42
c=0.7
92.15
75.68
82.96
86.52
89.64
91.68
92.05
92.10
91.94
91.90
c=0.8
93.51
10.00
10.19
15.76
59.78
91.17
93.11
93.40
93.45
93.35
c=0.9
94.93
10.00
10,00
8.95
20.52
80.80
92.47
94.24
94.76
94.90
ResNet- 18
CIFAR-
c=0.1
89.27
10.00
10.00
10.66
26.07
75.84
89.25
89.92
89.75
89.60
c=0.2
93.36
92.47
92.68
93.17
93.74
93.86
93.82
93.82
93.81
93.73
c=0.3
94.09
92.86
93.35
94.06
94.37
94.47
94.48
94.42
94.29
94.27
c=0.4
94.61
93.11
93.97
94.59
94.92
95.18
95.07
94.81
94.71
94.67
c=0.5
94.79
68.75
10.48
10.14
47.34
89.64
93.96
94.67
94.55
94.49
c=0.6
94.99
87.49
88.80
89.29
90.87
94.37
94.83
94.73
94.71
c=0.7
95.39
45.02
50.13
60.06
80.77
91.06
94.72
95.24
95.27
95.17
==8.8
95.92
10.00
10.00
10.00
41.71
92.91
94.93
95.54
95.70
95.71
c=0.9
96.28
9.99
10.02
19.72
59.28
78.32
90.47
94.63
95.80
95.98
ResNet-
CIFAR-10
c=0.1
81.53
80.65
81.87
83.06
83.68
84.11
84. 14
83.88
83.78
83.75
c=0.2
82.07
80.99
82.25
83.52
84.46
84.70
84.85
84.89
84.80
84.69
=03.
83.46
80.03
82.17
83.81
84.84
85.32
85.26
85.22
85.01
84.95
c00.4
84.97
80.30
82.80
84.69
86.12
86.79
86.79
86.75
86.71
c=0.5
86.49
79.06
82.53
85.36
87.06
87.93
88.13
88.02
87.87
87.77
c=0.6
88.48
76.21
81.74
85.86
88.30
89.19
89.11
88,99
88.93
88.85
c=0.7
89.70
16.04
13.97
24.14
64.91
84.75
87.43
87.86
88.06
88.08
c=0.8
91.07
40.97
44.14
43.90
55.70
73.42
84.29
88.08
89.52
89.93
c=0.9
92.98
33.81
43.71
59.40
78.30
91.09
91.66
91.78
91.83
VGG-16 on CIFAR-100
c=0.2
65.04
1.00
1.00
1.01
1.53
4.57
30.82
58.80
65.33
65.13
c=0.3
66.05
1.00
1.00
2.15
18.58
54.26
64.34
65.96
c=0.4
68.46
64.21
66.30
67.97
69.31
70.09
70.19
70.05
69.79
69.62
c=0.5
69.30
61.99
64.31
66.71
68.91
70.42
70.50
70.18
70.03
69.85
c=0.6
69.49
49.34
53.22
57.83
62.58
66.67
69.11
70.07
69.63
68.96
c=0.7
70.97
30.19
34.77
41.37
50.01
59.17
66.61
70.07
70.85
70.45
c=0.8
72.13
12.81
22.37
32.70
47.78
62.35
69.54
71.31
71.11
c=0.9
74.76
1.00
1.00
1.03
2.02
9.97
32.00
$5.97
67.82
71.85
ResNet- 18
CIFAR-100 1-100
c=0.1
71.84
71.11
72.51
73.32
73.41
73.38
72.63
72.19
72.06
71.88
c=0.2
72,32
34.00
39.42
48.16
59.34
67.41
70.63
70.14
67.63
64.74
c=0.3
74.01
71.51
73.91
74.89
75.40
75.41
75.30
74.98
74.90
74.71
c=0.4
73.90
51.56
55.56
60.20
64.74
69.16
71.99
72.89
72.76
71.94
c=0.5
74.88
53.01
55.92
57.37
60.59
67.62
73.15
74.53
73.70
72.81
c=0.6
75.93
4.45
1.01
2.05
4.33
44.28
69.24
72.85
71.71
c=0.7
76.44
1.13
1.00
1.00
1.00
1.66
34.47
66.96
72.57
73.35
c=0.8
78.41
1.00
1.00
1.00
1.00
2.31
37.50
62.62
71.34
73.83
c=0.9
78.18
1.00
1.00
1.04
30.44
66.73
73.81
77.04
77.52
77.66
ResNet-20
CIFAR-100
c=0.1
48.62
46.80
49.85
51.61
52.19
51.95
51.23
50.31
49.56
49.12
c=0.2
50.79
48.12
51.35
53.27
54.17
53.91
53.11
51.75
50.89
50.35
c00.3
52.84
47.08
51.34
54.51
56.00
56.31
55.46
54.46
53.82
53.42
c=0.4
55.18
45.58
50.63
54.72
57.44
57.67
56.69
55.38
54.54
53.97
c=0.5
57.51
40.65
47.14
54.15
58.37
59.59
58.47
57.33
56.41
55.88
c=0.6
59.98
25.56
34.28
47.01
56.64
59.60
59.16
57.74
56.73
56.35
007.
64.71
18.87
25.93
37.26
46.92
51.28
$1.68
51.52
51.12
50.84
=0.88
66.96
9.73
12.76
21.48
39.48
46.84
48.91
49.90
50.01
50.18
c=0.9
69.36
5.82
11.01
22.58
47.32
61.57
65.26
65.96
66.32
27

---

Under review as a conference paper at ICLR 2023
Table S6: Influence of different slope e with the quasi-latency N
Slope
ANN Acc.
T=1
T=2
T=4
T=8
T=16
T=32
T=64
T=128
T=256
VGG-16 CIFAR-10
c=0.1
92.73
87.94
89.45
90.91
92.06
92.72
93.06
93.00
93.01
93.02
c=0.2
93.02
88.17
89.57
91.08
92.26
92.96
93.19
93.25
93.24
93.25
c=0.3
93.11
86.81
88.60
90.19
91.83
92.96
93.25
93.41
93.40
93.40
c=0.4
93.10
83.57
86.65
89.47
91.48
92.81
93.08
93.18
93.18
93.12
c=0.5
92.84
75.46
81.96
86.56
90.51
92.19
92.53
92.62
92.56
92.48
c=0.6
93.44
73.83
83.96
88.69
91.73
92.79
93.39
93.36
93.42
93.41
c=0.7
94.31
72.17
80.16
86.63
91.38
93.52
94.11
94.41
94.47
94.54
c=0.8
94.93
61.41
81.37
89.14
92.70
94.32
94.80
94.95
94.91
94.93
c=0.9
95.47
59.55
80.19
89.92
93.77
95.13
95.42
95.39
95.44
95.42
ResNet-18
CIFAR-I
c=0.1
94.20
89.30
91.26
92.61
93.76
94.19
94.42
94.42
94.43
94.50
c=0.2
95.16
90.79
92.68
94.11
95.08
95.28
95.41
95.37
95.39
95.35
c=0.3
95.42
89.97
92.13
93.90
95.14
95.68
95.82
95.70
95.74
95.69
c=0.4
95.56
90.37
92.32
93.85
94.96
95.62
95.68
95.71
95.75
95.78
c=0.5
95.97
90.63
92.77
94.36
95.44
96.07
96.14
96.15
96.11
96.10
c=0.6
95.98
86.23
90.08
93.02
94.96
95.81
96.10
96.14
96.12
96.15
c=0.7
96.06
85.96
89.72
92.81
94.81
95.60
95.93
95.95
96.10
96.14
c-8.8
96.46
82.69
88.09
92.04
94.81
95.99
96.29
96.39
96.31
96.29
c=0.9
96.48
68.90
77.39
86.43
92.76
95.38
96.15
96.36
96.45
96.48
ResNet-20 20
CIFAR-10
c=0.1
87.91
75.92
80.90
85.29
88.14
89.10
89.35
89.19
89.01
88.95
0-2.2
88.66
75.06
80.65
86.17
88.65
89.51
89.90
89.83
89.69
89.61
0=3.3
89.53
71.77
78.42
84.76
88.82
90.24
90.45
90.37
90.20
90.15
c=0.4
89.73
68.57
76.85
84.33
88.71
90.05
90.14
90.20
90.25
90.20
c=0.5
90.72
66.82
76.31
84.58
89.57
91.13
91.46
91.45
91.35
91.32
c=0.6
91.48
56.46
67.99
80.94
88.52
90.99
91.77
91.89
91.84
91.88
c=0.7
92.17
60.32
70.93
81.52
88.88
91.72
92.26
92.26
92.25
92.27
=08
92,91
50.95
60.84
74.04
86.55
91.83
93.14
93.40
93.35
93.26
c=9.9
93.11
35.47
46.10
62.81
82.57
90.93
92.71
93.14
93.21
93.18
VGG-16
CIFAR-100
c=0.1
70.03
54.68
58.66
62,56
66.31
69.35
70.65
71.23
71.52
71.47
c=0.2
70.73
48.02
52.87
58.53
64.34
68.35
70.66
71.52
71.79
71.76
c=0.3
71.16
48.14
53.15
58.71
64.57
68.66
70.93
71.83
72.00
71.98
c=0.4
71.43
42.45
48.41
55.32
62.68
68.34
70.84
71.88
72.17
72.07
c=0.5
72.66
36.01
43.11
51.25
59.92
67.10
70.95
72.48
72.91
73.15
c=0.6
72.73
27.72
33.97
42.64
53.60
63.91
70.07
72.61
73.26
73.35
c=0.7
73.47
19.30
25.04
33.69
46.26
60.29
69.07
72.51
73.46
73.46
=08.8
74.12
13.40
17.44
25.41
39.23
56.54
68.80
73.11
74.18
74.41
-=9.9
75.18
22.41
28.52
38.27
51.58
64.70
71.73
74.32
75.14
75.25
ResNet- 18 on CIFAR-100
c=0.1
75.38
61.23
67.17
71.52
74.64
76.20
76.50
76.46
76.29
76.30
c=0.2
76.15
61.14
67.49
72.24
75.16
76.66
77.04
76.96
76.95
76.94
c=0.3
76.60
58.29
65.68
71.51
75.28
77.07
76.99
77.00
76.99
c=0.4
77.32
55.72
62.98
70.09
74.72
77.02
77.99
77.98
77.82
77.79
c=0.5
77.08
51.01
60.03
68.72
74.59
77.29
78.04
77.97
77.99
77.91
c=0.6
77.42
41.69
53.51
64.96
73.17
76.90
77.57
77.68
77.85
77.80
c=0.7
77.93
33.40
45.77
58.96
70.54
76.15
77.53
78.02
78.02
78.08
c00.8
78.22
18.51
2.38
43.49
74.16
78.05
78.59
78.64
c=0.9
78.22
1.00
19.97
1.71
17.25
65.52
76.47
78.10
78.26
78.23
20
CIFAR-100
c=0.1
59.83
32.76
42.03
52.20
59,45
62.15
62.47
62.81
62.65
62.40
c=0.2
61.36
31.66
40.76
52.05
60.08
63.17
63.63
63.42
63.07
62.90
c=0.3
62.96
23.91
32,65
45.70
58.25
63.62
64.90
64.92
64.80
64.59
c04.
64.32
24.88
34.52
48.19
60.47
65.14
66.55
66.72
66.46
66.41
c0.55
65.85
18.07
25.30
39.24
56.10
64.30
66.50
67.12
67.15
67.21
c=0.6
66.75
16.58
23.81
37.84
56.00
67.53
68.09
67.85
67.71
007
68.49
10.58
15.74
28.33
49.67
63.67
67.64
68.77
68.98
69.03
=08.8
69.03
13.51
20.17
33.17
53.63
65.21
68.59
69.32
69.51
69.45
-09
69.70
7.92
11.19
18.76
36.40
59.36
67.75
69.62
69.89
70.02
28

---

Under review as a conference paper at ICLR 2023
Table S7: Influence of different slope e with the quasi-latency N 4.
Slope
ANN Acc.
T=1
T=2
T=4
T=8
T=16
T=32
T=64
T=128
T=256
VGG-16 CIFAR-10
c=0.1
93.24
68.87
83.64
89.31
92.14
93.03
93.32
93.41
93.47
93.47
c=0.2
92.68
60.15
80.03
86.37
90.43
92.17
92.62
92.71
92.75
92.71
c=0.3
93.55
47.52
83.30
88.85
92.07
93.18
93.54
93.65
93.68
93.67
c=0.4
93.94
51.52
80.67
88.13
91.89
93.20
93.81
93.98
94.03
94.01
c=0.5
94.54
65.47
83.46
90.13
93.09
94.25
94.61
94.62
94.63
94.58
c=0.6
94.95
50.73
83.07
90.25
93.37
94.61
94.91
95.06
95.05
95.02
c=0.7
95.02
32.67
79.63
89.69
93.55
94.84
95.14
95.04
95.02
95.05
c=0.8
95.52
21.08
76.27
89.69
93.73
94.98
95.47
95.53
95.61
95.60
c=0.9
95.60
11.37
75.18
88.80
93.54
95.20
95.66
95.65
95.66
95.67
ResNet-
CIFAR-10
c=0.1
96.01
88.01
90.96
93.34
95.12
95.86
96.02
96.13
96.16
96.16
c=0.2
96.31
86.16
89.82
93.00
95.02
95.90
96.27
96.43
96.44
96.45
c=3.3
96.15
86.52
90.78
93.84
95.48
96.10
96.12
96.22
96.15
96.19
c=0.4
96.27
87.18
90.76
93.66
95.29
95.90
96.13
96.25
96.21
96.25
c=0.5
96.38
84.76
89.29
92.89
94.99
95.82
96.27
96.33
96.36
96.36
c=0.6
96.29
79.25
85.25
90.26
94.05
95.68
96.30
96.39
96.42
96.41
c=0.7
96.68
74.78
82.30
89.16
93.86
95.89
96.46
96.60
96.66
96.69
c-8.8
96.53
73.43
80.72
88.15
93.28
95.70
96.23
96.45
96.55
96.58
c=0.9
96.67
56.79
68.00
81.08
90.61
95.08
96.31
96.53
96.52
96.59
Ne on CIFAR-10
c=0.1
91.42
66.51
75.99
84.62
89.58
91.24
91.80
91.89
91.97
92.01
0-2.2
91.82
60.30
71.25
82.44
89.05
91.79
92.27
92.36
92.35
92.28
0=3.3
91.81
60.66
72.13
82.62
89.40
91.74
92.36
92.46
92.53
92.51
c=0.4
92.07
61.96
72.57
82.27
88.68
91.45
92.38
92.46
92.55
92.55
c=0.5
92.91
44.08
54.50
71.27
86.16
91.66
93.14
93.24
93.32
93.21
c=0.6
92.96
45.87
57.82
73.17
86.66
92.13
93.23
93.36
93.29
93.19
c=0.7
93.30
41.65
54.13
70.79
85.97
91.79
93.28
93.61
93.55
93.46
=08
93.33
29.14
39.35
59.35
80,90
90.65
92.76
93.14
93.26
93.24
c=0.9
93.37
15.29
21.55
41.27
75.60
90.45
92.95
93.49
93.52
93.52
VGG-16
CIFAR-100
c=0.1
71.78
22.27
28.83
38.88
51.68
63.38
69.68
71.64
72.04
71.92
c=0.2
72.16
20.01
26.24
35.41
47.79
60.82
68.78
71.67
72.47
72.59
c=0.3
73,40
26.37
33.29
42.97
55.27
65.80
71.30
73.26
73.55
73.69
c=0.4
73.18
18.13
24.70
34.32
47.98
61.11
69.59
72.83
73.65
73.51
c=0.5
73.25
15.29
20.91
30.29
43.65
58.57
68.41
72.33
73.28
73.53
c=0.6
74.26
18.37
24.09
32.89
46.76
61.41
70.50
73.83
74.42
74.50
c=0.7
74.94
19.95
26.09
35.86
49.68
63.12
71.18
74.05
74.91
75.03
=08
74.50
9.07
13.70
22.44
37.92
57.41
69.20
73.02
74.33
74.69
-=9.9
75.25
13.62
21.23
32.24
47.76
63.32
71.84
74.61
75.13
75.15
Res.Net-18 18 on CIFAR-100
c=0.1
76.71
46.54
56.14
66.28
73.07
75.93
76.72
77.11
77.20
77.15
c=0.2
77.82
45.71
55.72
66.18
74.02
77.19
77.96
78.15
78.21
78.25
c=0.3
77.85
42.74
53.62
64.77
73.37
76.95
78.06
78.26
78.26
78.26
c=0.4
78.28
44.72
55.01
65.50
73.58
77.36
78.61
78.54
78.76
78.78
c=0.5
77.69
38.73
50.81
63.20
72.84
76.69
77.95
77.94
77.77
77.74
c=0.6
78.30
29.83
40.41
55.37
69.27
76.07
77.94
78.66
78.61
78.59
c=0.7
78.56
25.33
35.45
51.41
68.22
75.46
77.79
78.24
78.55
78.75
c00.8
77.96
21.54
30.90
45.51
64.42
77.14
78.16
78.34
78.33
c=0.9
78.00
13.54
20.76
33.67
57.43
72.09
76.43
77.60
77.98
78.14
Res Ne
CIFAR-100
c=0.1
66.37
15.69
23,85
40.99
58,23
65.42
67.39
67.68
67.51
67.33
c=0.2
66.91
19.33
27.89
43.62
59.79
66.51
68.16
68.40
68.42
68.45
c=0.3
67.39
15.92
22.44
38.25
57.74
66.47
68.29
68.70
68.59
68.45
c0.4
68.40
16.52
23.79
37.94
57.20
66.61
68.76
69.04
69.09
68.96
c0..5
68.86
11.14
15.27
26.79
49.58
64.77
68.70
69.63
69.75
69.69
c=0.6
68.83
9.15
23.57
45.74
68.25
69.08
69.24
69.32
007.
69.45
6,90
10.56
19.27
40,90
62.26
68.71
70.06
70.29
70.13
=08.8
69.59
7.09
9.00
14.88
32.93
59.35
68.05
69.61
70.08
69.94
=09
70.18
4.79
7.11
12.04
25.98
53.20
66.77
69.95
70.54
70.46
29


Reviews:
[{"review_id": "rssv6lt4Rb", "reviewer": "Reviewer_FzYt", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a conversion algorithm for deploying SNNs.", "summary_of_the_review": "In summary, this paper's attempt is worthy to be credited but it really lacks good motivation and novelty.", "strength_and_weaknesses": "-Not novel and results are not good. Today, many works are focused on direct training precisely to reduce the overall timestep count in SNNs that can lead to very expensive timestep computations during inference[1]. The authors work still gives best conversion accuracy at T>100. This is not acceptable anymore. I suggest the authors to go over recent conversion works that have strived to created conversion SNNs at T<10 timesteps [2].\n\n[1] Yin, Ruokai, et al. \"SATA: Sparsity-Aware Training Accelerator for Spiking Neural Networks.\" arXiv preprint arXiv:2204.05422 (2022).\n\n[2] Zheng, Hanle, et al. \"Going deeper with directly-trained larger spiking neural networks.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "h8ulFHdKOz", "reviewer": "Reviewer_FPAL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an efficient ANN-SNN conversion mechanism based on the SlipReLU activation function. It replaces the traditional ReLU to improve the accuracy of the converted SNN. The results look promising and competitive compared to the related works.", "summary_of_the_review": "Good paper where few concerns should be clarified.", "strength_and_weaknesses": "Strengths:\n1. The tackled problem is relevant to the scope of ICLR.\n2. The contributions look solid.\n3. The results look promising.\n\nWeaknesses:\n1. It would be interesting to have more details about more intuitions about the design decision made when developing the proposed SlipReLU activation function.\n2. The description of the proposed method in Section 4 looks vague and hard to follow. It is recommended to use schemes, and examples to ease the discussion.\n3. Please describe the experimental setup and tool flow used to generate the results shown in Section 6.\n4. It would be useful to provide the source code for reviewers' inspection during the rebuttal.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "YiW6ITvT7V", "reviewer": "Reviewer_7AYG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper improves upon the conversion error of two step ANN to SNN conversion based technique to yield sota accuracy.  In particular, it presents an efficient ANN-SNN conversion mechanism based on the SlipReLU and shifted SlipReLU activation function replacing the traditional ReLU to improve the accuracy of the converted SNN. The results look promising and competitive compared to the related works. The theoretical analysis of error strengthens the paper.", "summary_of_the_review": "Overall, this is a well written well motivated paper that requires further experiments to clarify the impact of the proposed non linearity in SNN paradigm.\n\nPost rebuttal: the contributions are new with detailed additional results, major concerns are addressed with results on further complex datasets. ", "strength_and_weaknesses": "### Strengths\n=============\n\nWell written.\n\nWell motivated.\n\nResult is good and inspiring.\n\nAnalysis of the various conversion error is present in details. Some are taken from earlier research though.\n\n### Weakness:\n===============\n\nThe experimental results section is weak. The authors should provide results on larger datasets to show the efficacy.\n\nThe author should do an ablation study with SlipReLU and shifted SlipReLU.\n\nIt would be interesting to see how such ReLU performs under various noisy inputs as discussed in [1-2], particularly, whether they can also maintain some inherent robustness with such non-linear functions, under two-step conversion based framework.\n\nPlease put some light on to the selection of input type (direct, rate etc.) on the performance with  SlipReLU.\n\nlack of reproducibility and missing details of experimental setup.\n\n[1] Inherent adversarial robustness of deep spiking neural networks: Effects of discrete input encoding and non-linear activations, ECCV 2020.\n\n[2] HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training With Crafted Input Noise, ICCV 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F78j-c4-8s", "reviewer": "Reviewer_rSjR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors devise a two-step conversion framework for ANN-SNN conversion. Unlike previous methods which consider the ANN-SNN conversion error, this work considers the error between source ANN and altered ANN as well as altered ANN with SNN. The idea is generally sound and simple to understand. Yet in my view, the two-step conversion naturally has some deficiencies with than one-step conversion (see weakness section for details). \n\n\n", "summary_of_the_review": "My reasons for the weak rejection are insufficient analysis and the lack of large-scale experiments. Were they addressed by the authors, I can increase my score. ", "strength_and_weaknesses": "Strength:\n\n+ The structure and presentation of this paper are neat and easy to understand.\n\n+ The perspective of the trade-off between the distance to ANN and the distance to SNN is novel and an interesting activation function is proposed (SlipReLU). \n\nWeakness:\n\n- The major concern in this work is the first term in Eq. 5, which hypothesize that the output between ReLU-based ANN and customized ANN should be considered, is not fully verified in experiments. As far as I interpret Eq. 5, the SlipReLU should have higher ANN accuracy but also higher conversion error than QCFS, because it has less distance to ANN while more distance to SNN compared with QCFS. \nAnd ideal result in SlipReLU is that it shows higher ANN accuracy and despite the higher conversion error, SlipReLU can still achieve higher SNN accuracy than QCFS. However, the results in paper does not verify the tradeoff or did not explicitly discuss the tradeoff in experiments. Ablation studies on the weight of combination, and more analysis on ANN, SNN accuracy tradeoff should be provided. \n\n- Two-step conversion certainly has its pros, but it also has cons. Two step method must train their own ANN models, while one-step models can utilize directly-trained checkpoints in open-sourced projects. Some large-scale datasets and models, are require a lot resource to train the ANN, for example, ImageNet. Thus, ImageNet results are also anticipated. \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject