SEENN: Towards Temporal Spiking Early Exit Neural Networks
Authors: Yuhang Li
Tamar Geller
Youngeun Kim
Priyadarshini Panda
Venue: NeurIPS 2023 poster (2023)
Keywords: Spiking Neural Networks
ANN-SNN Conversion
Conditional Computing

Abstract: Spiking Neural Networks (SNNs) have recently become more popular as a biologically plausible substitute for traditional Artificial Neural Networks (ANNs). SNNs are cost-efficient and deployment-friendly because they process input in both spatial and temporal manner using binary spikes. However, we observe that the information capacity in SNNs is affected by the number of timesteps, leading to an accuracy-efficiency tradeoff. In this work, we study a fine-grained adjustment of the number of timesteps in SNNs. Specifically, we treat the number of timesteps as a variable conditioned on different input samples to reduce redundant timesteps for certain data. 
We call our method Spiking Early-Exit Neural Networks (**SEENNs**). To determine the appropriate number of timesteps, we propose SEENN-I which uses a confidence score thresholding to filter out the uncertain predictions, and SEENN-II which determines the number of timesteps by reinforcement learning. 
Moreover, we demonstrate that SEENN is compatible with both the directly trained SNN and the ANN-SNN conversion. 
By dynamically adjusting the number of timesteps, our SEENN achieves a remarkable reduction in the average number of timesteps during inference. For example, our SEENN-II ResNet-19 can achieve **96.1**\% accuracy with an average of **1.08** timesteps on the CIFAR-10 test dataset. Code is shared at https://github.com/Intelligent-Computing-Lab-Yale/SEENN.

SEENN: Towards Temporal Spiking Early-Exit Neural Networks
Yuhang Li Yale University New Haven, CT, USA yuhang. li@yale.eeee edu
Tamar Geller Yale University New Haven, CT, USA tamar. www.eeeyallee....mmm edu
Youngeun Kim Yale University New Haven, CT, USA youngeun. kim@yale. edu
Priyadarshini Panda
Yale University New Haven, CT, USA priya. panda@yal. edu
Abstract
Spiking Neural Networks (SNNs) have recently become more popular as biologi- cally plausible substitute for traditional Artificial Neural Networks (ANNs). SNNs are cost-efficient and deployment-friendee friinddl because they process input both spatial and temporal manner using binary spikes. However, we observe that the informa- tion capacity in SNNs is affected by the number of timesteps, leading to an accuracy- efficiency tradeoff. In this work, we study fine-graine adjustment of the number of timesteps in SNNs. Specifically, we treat the number of timesteps as a variable conditioned on different input samples to reduce redundant timesteps for certain data. We call our method Spiking Early-Exit Neural Networks (SEENNs). To determine the appropriate number of timesteps, we propose SEENN-I which uses a confidence score thresholding to filter the uncertain predictions, and SEENN-II which determines the number of timesteps by reinforcement learning. Morcover, we demonstrate that SEENN is compatible with both the direetly trained SNN and the ANN-SNN conversion. By dynamically adjusting the number of timesteps, SEENN achieves a remarkable reduction in the average number of timesteps during inference. For example, our SEENN-II ResNet-19
achieve 96.1% accuracy with an average of 1.08 timesteps on the CIFAR-10 10 test dataset. Code is shared at Computing Lab-inn------------ Yale/SEENN.
https://githieii /ttp:/gggtttbbtt// wwt.//wwo./eeeeeeeeeeeeeeeeeeeee wawpllliitntt--ttttnnnggg
Introduction
Deep learning has revolutionized a range of computational tasks such computer vision and natural language processing [1] using Artificial Neural Networks (ANNs). These successes, however, have come at the cost of tremendous computational demands and high lateney [2]. In recent years. Spiking Neural Networks (SNNs) have gained traction an energy meery-efficient alternative to ANNs [3, 4, 5].. OSNNs infer inputs across a number of timesteps as opposed ANNs, which infer over what is essentially a single timestep. Morcover, during each timestep. the neuron in an SNN either fires spike or remains silent, thus making the output of the SNN neuron binary and sparse. Such spike-based computing produces calculations that substitute multiplications with additions.
In the field of SNN research, there are main approaches getting an SNN: (1) directly training SNNs from scratch and (2) converting ANNs to SNNs. Direct training seeks to optimize an SNN using methods such as spike timing-based plasticity [6] surrogate gradient based optimization [7.8]. In contrast, the ANN-SNN conversion approach [9, 10, 11, 12, 13, 14] uses the feature representation
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

of a pre-trained ANN and aims to replicate it in the corresponding SNN. Both methods have the potential to achieve regh-peforrance SNNs when implemented correctly.
Despite the different approaches, both training-based and conversion-based SNNs are limited by binary activations. As result, the key factor that affects their information processing capacity is the number of timesteps. Expanding the number of timesteps enables SNNs to capture more features in the temporal dimension, which can improve their accuracy in conversion and training. However, larger number of timesteps increases latency and computational requirements, resulting in lower acceleration ratio, and yielding tradeoff between accuracy and time. Therefore, current efforts to enhance SNN accuracy often involve finding ways to achieve it with fewer number of timesteps.
In this paper, propose novel approach to improve the tradeoff between accuracy and time in SNNs. Specifically, our method allows each input sample to have a varying number of timesteps during inference, increasing the number of timesteps only when the current sample is hard classify, resulting in an early exit in the time dimension. We refer to this approach as Spiking Early-Exit Neural Networks (SEENNs). To determine the optimal number of timesteps for each sample, we propose two methods: SEENN-I, which uses confidence score thresholding to output confident prediction as fast as possible; and SEENN-II, which employs reinforcement learning to find the optimal policy for the number of timesteps. Our results show that SEENNS can be applied to both conversion- based and direct training-basse based approaches, achieving new state-of-the-and performance for SNNs. In summary, our contributions are threefold:
1. We introduce new direction optimize SNN performance by treating the number of timesteps as a variable conditioned to input samples.
2. We propose Spiking Early-Exit Neural Networks (SEENNs) and two methods to determine which timestep to exit: confidence score thresholding and reinforcement learning optimized with policy gradients.
3. We evaluate our SEENNS on both conversion-based and training-basee models with large-scale datasets like CIFAR and ImageNet. For example, our SEENNS can use ~111 timesteps to achieve similar performance with 6-timestep model on the CIFAR-10 dataset.
2 Related Work
2.1 Spiking Neural Networks
Converting ANNs to SNNs utilizes the knowledge from pre-trained ANNs and replaces the ReLU activation in ANNs with a spike activation SNNs [10, 12, 15, 16]. The conversion-basee based method, therefore, seeks to match the features in two different models. For example, [10, 15] studies how to select the firing threshold to cover all the features in an ANN. [17] studies using smaller threshold and [13] proposes to use bias shift to better match the activation. Based on the error analysis, [14] utilizes a parameter calibration technique and [18] further changes the training scheme in ANN.
ANN-SNN Conversion
Direct Training of SNN Direct training from scratch allows SNNs to operate within extremely few timesteps. In recent years, the number of timesteps used to train SNNs has been reduced from more than 100 [19, 20] to less than [21]. The major success is based on the spatial-temporal backpropa- gation [7,22] and surrogate gradient estimation of the firing function [23]. Through gradient- based learning, recent works [24, 25, 27, 28, 29] propose to optimize not only parameters but also firing threshold and leaky factor. Moreover, loss function [30]. surrogate gradient estimation [31]. batch nor- malization [32], activation distribution [33], membrane potential gormalization/regulaizzzz
[34,35] also factors that affect the learning behavior in direct training and have been investigated properly. Our method, instead, focuses on the time dimension, which fully complementary to both conversion and training.
2.2 Conditional Computing
Conditional computing models can boost the representation power by adapting the model architec- tures, parameters, or activations to different input samples [36]. BranchyNet [37] and Conditional Deep Learning [38] add multiple classifiers different layers to apply spatial early exit to ANNs. SkipNet [39] and BlockDrop [40] use dynamic computation graph by skipping different blocks

---

CIFAR10 (Training)
CIFAR100 (Training)
ImageNet (Training)
CIFAR10 (Conversion)
94.5
NET=2.228
74.5
75.0
AET 3.033
74.72
68
AET=2.065
67.11
68.3
96
AET NET-1.484
50.2
Accuracy (%)
94.4
65.66
94.3
73.5
01.47
94.1
73.0
62
72.87
60.75
94.0
72.5
60
#Timesteps
#Timesteps
#Timesteps
#Timesteps
Figure 1: The accuracy of spiking ResNet under different numbers of timesteps CIFARIO, CIFARI00, and ImageNet datasets, either by direct training [30] or by conversion [18].
based different samples. CondConv [41] and Dynamic Convolution [36] use the attention mecha- nism to change the weight in convolutional layers according to imput features. Squeeze-and Squeeze-and-Exxe and-Excitation Networks [42] proposes reweight different activation channels based on the global context of the input samples. To the best of our knowledge, our work is the first to incorporate conditional computing into SNNs.
3 Methodology
In this section, we describe our overall methodology and algorithm details. We start by introducing the background on fixed exed-timestep spiking neural networks and then, based on strong rationale, we introduce our SEENN method.
3.1 Spiking Neural Networks
SNNs simulate biological neurons with Leaky Integrate- and--tte-and-Fiig (LIF) layers [43]. In each timestep, the input current in the ê-th layer charges the membrane potential in the LIF neurons. When the membrane potential exceeds threshold, spike s will fire to the next layer, as given by:
u [t 1| ru [t] + -t) [t).
[t + = (u []
(2)
where 7 E (0, is the leaky factor, mimicking natural potential decay. H-)) the Heaviside step
function and V is the firing threshold. If spike fires, the membrane potential will be reset to 0:
(ult+ + 1] ult 1] (1 s(t + 1])). Following existing baselines, direct training, we 7 0.5 while in conversion we use т = 1.0, transforming to the Integrate-and-Fing (IF) model. Moreover, the
reset conversion is done by subtraction: (uft + = ult + 1] Va,s(t + 1]) as suggested by [12]. Now, denote the overall spiking neural network function fr (x), its forward propagation can be formulated
T Σ t=1
fr(x) = T
Chogoog o --1 o 2 (x),
where ht-) denotes the final linear classifier, and (-) denotes the b-th block of backbone networks. L represents the total number of blocks in the network. A block contains a convolutional layer to compute input current (''s'-1), a normalization layer [21], and LIF layer. In this work, we use direct encoding method, i.e. using g' (x) to encode the input tensor into spike trains, as done in recent SNN works [24,30,4 (4). In SNNs, we repeat the inference process for T times and average the output from the classifier to produce the final result.
3.2 Introducing Early-Exit to SNNs
Conventionally, the number of timesteps T is set as fixed between accuracy and efficiency. Here, we also provide several accuracy T curves of spiking ResNets Fig. 1. These models are trained from scratch or converted 4, 6, timesteps and evaluate the performance with the available numbers of timesteps. For the CIFAR-10 dataset,
causing a tradeoff

---

increasing the number of timesteps from 2 to 6 only brings 0.34% top-1 accuracy gai, at the cost of 300% more latency. In other words, the majority of correct predictions can be inferred with much fewer timesteps.
The observation in Fig. motivates interested in an SNN that can adjust T based on the characteristics of different input images, Hy- pothetically, each image is linked to an difficulty factor, and this SNN can identify this difficulty to decide how many timesteps should be used, thus eliminating unnecessary timesteps for easy images. spiking early-exit neural network (SEENN).
explore more fine tieeggrinedd adjustment of T. We
We refer to such a model
To demonstrate the potential of SEENN, we propose a metric that calculates the minimum timesteps needed to perform correct prediction averaged on the test dataset, i.e. the lowest timesteps we can achieve without comprising the accuracy, and it is hased on the following assumption:
Assumption 3.1. Given a spiking neural network fr, if it can correctly predict with timesteps. then it always outputs correct prediction for any l' such that & < l' T.
This assumption indicates the inclusive property of different timesteps. In Appendix A, we provide empirical evidence to support this assumption. Let C, be the set of correct predicted input samples for timestep 4, we have C1 S C2 S .... S Cr based on Assumption 3.1. Also denote W = CT as the wrong prediction set, we propose the averaged earliest timestep (AET) metric, given by
AET = N |CI|+T| + Σ ((C1 -IC-
(4)
where l |rettrns the cardinal number of the set, and - -Cal returns the number of samples that belong to Ct yet not to C.-1. N = =|Cr| + [W] is the total number of samples in the validation dataset. The AET metric describes an ideal scenario where correct predictions are always inferred using the minimum number of timesteps required, while still preserving the original accuracy. It's worth noting that incorrect samples are inferred using the maximum number of timesteps, as it is usually not possible to determine if a sample cannot be correctly classified before inference.
In Fig. 1, we report the AET in each case. For models directly trained on CIFAR 10 or CIFAR100, the AET remains slightly higher than (note that the minimum number of timesteps is 2). With merely 11 % more latency added to the 2-timestep SNN on CIFAR10 (T 2.228), we can achieve an accuracy equal to 6-timestep SNN. The converted SNNs also only need a few extra time steps. This suggests the huge potential for using early exit in SNNs.
Despite the potential performance boost from the early exit, it is impossible to achieve the AET effect in practice since we cannot access the label in the test set. Therefore, the question of how to design an efficient and effective predictor that determines the number of timesteps for each input is non-trivial. In the following sections, we propose two methods, SEENN-I and SEENN-II, to address this challenge.
3.3 SEENN-I
In SEENN-I, we adopt a proxy signal to determine the difficulty of the input sample-confidene confidence score. Formally, let the network prediction probability distribution be p softmax (f (x)) = "M. where M is the number of object classes, the confidence score (CS) is defined as
CS = max(p),
(5)
which means the maximum probability in p. The CS is signal that measures the level of uncertainty. If the CS is high enough (e.g. CS= 0.99), the prediction distribution will be highly deterministie; otherwise (e.g. CS= X ), the prediction distribution is uniform and extremely uncertain.
line of work [37,45] has shown that the level of uncertainty highly correlated with the accuracy of the neural networks. For input samples with deterministic output prediction, the neural network typically achieves high accuracy, which corresponds to relatively easy samples. Hence, we can utilize this property as an indicator of how many time steps should be assigned to this sample. We adopt simple thresholding mechanism, i.e. given a preset threshold a, we iterate each timestep and once the confidence score is higher than the accumulated output is used for prediction. Appendix B demonstrates the distribution of confidence scores changes with different numbers of timesteps. diagram describing this method can be found in Fig, 2(a).

---

Policy Network
Accumalated
Rg.R
Output
Spiking
Spiking Network
(xc
Accumulated Output
Entropy
SEENN-I
(b) SEENN-11
Figure 2: The frameworks our proposed SEENNs. (a): SEENN-I uses confidence score thresholding for determining the optimal number of timestops, (b): SEENN-II leverages a policy network to predict the number of timesteps, optimized by reinforcement learning.
3.4 SEENN-II
Our SEENN-I is a post-training method that can be applied an off-the-shelf SNN. It is easy to use, but the SNN is not explicitly trained with the early exit, so the full potential of early exit is not exploited through SEENN-I. In this section, we propose an "early-exit-awe training" method for SNNs called SEENN-II.
SEENN-II is built from reinforcement learning directly predict the difficulty of the image. Specifi- cally, we define an action space T {11, 12, ... ). which contains the candidates for the number of timesteps that be applied to input sample To determine the optimal timestep candidates, we develop a policy network that generates an n--dimnnsional policy vector to sample actions. During training, reward function calculated based on the policy and the prediction result, which generated by running the SNN with the number of timesteps suggested by the policy vector. Unlike traditional reinforcement learning [46], our problem does consider state transition the policy can predict all actions at once. We also provide a diagram describing the overall method in Fig. 2(b).
Formally, consider input sample and policy network fp with parameter 0, define the policy of selecting the timestep candidates as an n-dim categorical distribution:
n
V sfftmax(ff 0)), To (z(
II V
(6)
where v is the probability of the categorical distribution, obtained by inferring the policy networks with softmax function. Thus represents the probability of choosing & as the number of timesteps in the SNN. An action z E {0, 1)" sampled based on the policy v. Here, z is one-hot vector since only one timestep can be selected. Note that the policy network architecture is made sufficiently small such that the cost of inferring the policy is negligible compared SNN (see architecture details in Sec. 4.1).
Once we obtain an action vector z, we can evaluate the prediction using the target number of timesteps, i.e. fx(x). Our objective is to minimize the number of timesteps we used while not sacrificing accuracy. Therefore, we associate the actions taken with the following reward function:
if correct prediction
R(z)
2h -B
if incorrect prediction
Here, 1e/x==1 represents the number of timesteps selected by z. Here, the reward function is determined by whether the prediction is correct or incorrect. If the prediction is correct, then we incentivize early exit by assigning larger reward to a policy that uses fewer timesteps. However, the prediction is wrong, we penalize the reward with B, which serves the role to balance the accuracy and the efficiency, As an example, a large 3 leads to more correct predictions but also more timesteps.
Gradient Calculation in the Policy Network To this end, our objective for training the policy network is to maximize the expected reward function, given by:
max Exaor [[(z)]
(8)
5

---

In order to calculate the gradient of the above objective, we utilize the policy gradient method [46] to compute the derivative of the reward function w.r.t. 6, given by
[oB[R(2)] = E[R(z) log (2]x) = B [( log II = Σ
2x log vul- (9)
Moreover, unlike other reinforcement learning which relies on Monte-Carlo sampling to compute the expectation, our method can compute the exact expectation. During the forward propagation of the SNN, we can store the intermediate accumulated output each fgg and calculate the reward function using the stored accumulated output ft. (x). Since To Fz(z)x)is categorical distribution, we can rewrite Eq. (9) as
V [eE[
= Σ
V log Vk. Vk:
(10)
k=1
where R(1),,=1) is the reward function evaluated with the output prediction using ly timesteps.
3.5 Training SEENN
In this section, describe the training methods for our SEENN. To train the model for SEENN-I, explicitly add a cross-entropy function to each timestep, thus making the prediction in early timesteps better, given by
min W 71
Σ
W.y),
(11)
where LCg denotes the cross-entropy loss function and y is the label vector. This training objective is essentially Temporal Efficient Training (TET) loss proposed in [30]. We find this function can enhance the performance from every l compared to LCE applied to only the maximum number of timesteps. As for conversion-based SEENN-I, we do not modify any training function. Instead, we directly apply the confidence score thresholding to the converted model.
To train the SEENN-II, we first employ TET loss to train the model for several epochs without involving the policy network. This can avoid the low training accuracy in the early stage of the training which may damage the optimization of the policy network. Then, we jointly optimize the SNN and the policy network by
min Ezwro (-R(z) + LOE (frr (x). W. y)]. W.8
(12)
Note that we do not train SEENN-II for the converted model since in conversion, pre-trained ANN is used to obtain the converted SNN and does not warrant any training.
4 Experiments
To demonstrate the efficacy and the efficiency of our SEENN, we conduct experiments on popular image recognition datasets, including CIFAR10, CIFAR100 [47], ImageNet [48], and an event-stream dataset CIFARIO-DYS [49]. Moreover, to show the compatibility our method, we compare SEENN with both training-based and conversion-based state-of-the-art methods. Finally, we also provide some hardware and qualitative evaluation on SEENN.
4.1 Implementation Details
In order to implement SEENN-I, we adopt the code framework of TET [30] and QCFS [18], both of which provide open-source implementation. All models are trained with a stochastic gradient descent optimizer with momentum of 0.9 for 300 epochs. The learning rate is 0. and decayed following cosine annealing schedule [50]. The weight decay is set to 5e 4. For ANN pre-training in QCFS, set the step to 4. We Cutout [51] and AutoAugment [52] for better accuracy adopted in [31.
To implement SEENN-II, we first take the checkpoint of pre-trained SNN in SEENN-I and initialize the policy network. Then, we jointly train the policy network as well as the SNN. The policy network we take for the CIFAR dataset is ResNet-8, which contains only 0.547% computation of ResNet- 19.

---

Table 1: Accuracy-T comparison of direct training SNN methods on CIFAR datasets.
CIFAR-10
CIFAR-100
Method
Model
Acci(T))
Acoz(Ts)
Accs(Ts)
Acci
Aoca(T2)
Accs(Ts)
ResNet-20 Dspike [31]
93.13 (2)
93.16 (4)
94.25 (6)
71.28 (2)
73.35 (4)
74.24 (6)
wBN [21]
92.34 (2)
92.92 (4)
93.16
TET [30]
94.16 (2)
94.44 (4)
(6)
72.87 (2)
74.47 (4)
74.72 (6)
RecDis-SNN [33]
93.64 (2)
95.53 (4)
95.55 (6)
74,10 (4)
IM-Loss [53]
93.85 (2)
95,40 (4)
95,49 (6)
ResNet-19
TEBN [32]
95.45 (2)
95.58 (4)
95.60 (6)
78.07 (2)
78.71 (4)
78.76 (6)
SEENN-I (Ours)
96.07 (1.09)
96.38 (1.20)
96.44 (1.34)
79.56 (1.19)
81.42 (1.55)
81.65 (1.74)
SEENN-II (Ours)
96.01 (1.08)
80.23 (1.21)
Diet-SNN [25]
92.70 (5)
69.67 (5)
Temporal prune [54]
93.05 (1)
93.72 (2)
93.85 (3)
63.30 (1) 64.86 (2)
66.16 (3)
VGG-16
SEENN-I (Ours)
94.07 (1.08)
94.41 (1.20)
94.56 (1.46)
71.87 (1.12) 73.53 (1.46) 74.30 (1.84)
SEENN-II (Ours)
94.36 (1.09)
72.76 (1.15)
Table 2: Accuracy-1 comparison of ANN-SNN conversion SNN methods on CIFAR datasets.
CIFAR-10
CIFAR-100
Method
Model
(ccy(TT)
(T2)(T2)
(T3)
Acc, (Ti)
(Tay(T2)
(Ta)
ResNet-20 Opt. [13]
92.41 (16)
93.30 (32)
93.55 (64)
63.73 (16)
68.40 (32)
69.27 (64)
Calibration [14]
94.78 (32)
95.30 (64)
95.42 (128)
OPI [30]
75.44 (8)
90.43 (16)
94.82 (32)
23.09 (8)
$2.34 (16)
67.18 (32)
QCFS [18]
75.44 (2)
90.43
94.82 (8)
19.96 (2)
34.14 (4)
55.37 (8)
ResNet-18
SEENN-I (Ours) 91.08 (1.18) 93.63 (1.40)
95.08 (2.01)
39.33 (2.57)
56.99 (4.41) 65.48 (6.19)
We also provide the details of architecture in Appendix E and measure the latency/energy of the policy network in Sec. 4.3. For the ImageNet dataset, we downsample the image resolution to 1 112x 112 for the policy network. Both the policy network and the SNN are finetuned for 75 epochs using learning rate of 0.01. Other training parrmmtarsmsters are kept the same as SEENN-I.
4.2 Comparison to SOTA work
CIFAR Training We first provide the results on the CIFAR- and CIFAR-100 datasets [47]. We test the architectures the ResNet family [55] and VGG-16 [56]. We summarize both the direct training comparison as well as the ANN-SNN conversion comparison in Table 1 and Table 2. respectively. Since the T in our SEENN can be variable based on the input, we report the average number of timesteps in the test dataset. As we can see from Table 1, existing direct training methods usually use 2, 4, and 6 timesteps in inference. However, increasing the number of timesteps from 2 to 6 brings marginal improvement in accuracy, for example, 95.45 % to 95.60% in TEBN [32]. Our SEENN-I can achieve 96.07% with only 1.09 average timesteps, which is 5,5x lower than the state of the art. SEENN-II gets similar performance with SEENN-I in the case of CIFAR-10, but it can achieve 0.7% higher accuracy than SEENN-I on the CIFAR-100 dataset.
Comparison with 1- 1-Timestep SNN Reducing the number of timestep to 1 is another way towards low latency SNN [54,57]. However, this method ignores the sample- wise difference and may lead to larger accuracy drop. In Table 1, we compare VGG-16 16 with even I timestep [54]. Our SEENN-1 achieves 8.5% accuracy improvement at the cost of 12% more latency on the CIFAR- 100 dataset.
CIFAR Conversion We also include the results of ANN-SNN conversion using SEENN-I in Table 2. Here, the best existing work is QCFS [18], which can convert the SNN in lower than 8 timesteps. We directly run SEENN-I with different choices of confidence score threshold a. Surprisingly, on the CIFAR-I0 dataset, our SEENN-I can convert the model with 1.4 timesteps, and get 93.63% accuracy. Instead, the QCFS only gets 75.44% accuracy using 2 timesteps for all input images. By selecting higher threshold, we can obtain 95.08% accuracy with 2.01 timesteps, which uses 4x lower number of timesteps than QCFS under the same accuracy level.
7

---

Table 3: Accuracy-T comparison ImageNet dataset.
Model
Method
T
Acc. Model
Method
T
Acc.
Direct Training of SNNs
ANN-SNN Conversion
ddBN [21]
6
63.72
32
33,01
Opt [13]
TET [30]
6
64.79
64
59.52
TEBN [32]
6
64.29
32
64.34
ResNet-34
Calibration [14]
2.28
63.65
64
71.12
SEENN-I (Ours)
3.38
64.66
ResNet-34
16
59.35
SEENN-II (Ours)
2,40
64.18
QCFS [18]
32
69.37
SEW [58]
4
67.04
64
72.35
TET [30]
4
68.00
23.47
70.18
SEENN-I (Ours)
TEBN [32]
4
68.28
29.53
71.84
SEW-ResNet-34
1.66
66.21
SEENN-I (Ours)
2.35 67.99
SEENN-II (Ours)
1.79 67.48
ImageNet We compare the direct training and conversion methods the ImageNet dataset. The results are sorted Table 3. For direct training, we use two baseline networks: vanilla ResNet-34 [55] and 3EW-ResNet-34 [58]. For SEENN-I, directly use the pre-trained checkpoint from TET [30].
can be observed that our SEENN-I can achieve the same accuracy using only 50% of original number of timesteps. For example, SEENN-I ResNet-34 reaches the maximum accuracy in 2.35 timesteps. SEENN-II can obtain a similar performance by using smaller T, i.e. 1.79. For conversion experiments, again compare against QCFS using ResNet-34. Our SEENN-I obtains 70.2% accuracy with 23.5 timesteps, higher than 32-timestep QCFS model.
event-stream dataset, CIFARIO-DVS [49]. Following existing baselines, we train the SNN with fixed number of timesteps. From Table 4 we find that SEENN-I can surpass all existing work except TET with only 2.53 timesteps, amounting to nearly 4x faster inference. Moreover, SEENN-II can get an accuracy of 82.6 using 4.5 timesteps.
CIFARIO-DVS Here, compare SEENN-I
SNN
96.46, T
Model
Method
T Acc.
96.01,
Direct Training of SNNs
SEENN
96.07, T-1.09
ResNet- 19
tdBN [21]
67.8
SNN
Acc 95.01,
VGGSNN
PLIF [24]
20
74.8
150
175
ResNet-18
Dspike [31]
10
75.4
Throughout (image per second)
Re
RecDis- -NNiss--NNNNNN [33]
10
72.4
SNN
Acc 96.46, T-d
VGGSNN
TET [30]
10
83.1
SEENN-I1
Act 95.01, T=1.08
SEENN-4
96.38,
T=1.20
VGG-SNN SEENN-I (Ours)
2.53
77.6
5.17
82.7
0.0
a2
6.4
0.5
0.8
1.0
15
SEENN-II (Ours)
4.49
82.6
Energy per sample (m))
Table 4: Accuracy-T comparison on CIFAR10-DVS
Figure 3: Comparison of latency (throughput) and
dataset.
energy consumption between SNN and SEENN.
4.3 Hardware Efficiency
In this section, we analyze the hardware efficiency of our method, including latency and energy consumption. Due to the sequential processing nature of SNNs. the latency is generally proportional T' most hardware devices. Therefore, we directly use GPU (NVIDIA Tesla V100) to evaluate the latency (or throughput) of SEENN. For energy estimation, we follow rough measure to count only the energy of operations that is adopted in previous work [18, 25, 31], as SNNs are usually deployed on memory-cheap devices (detailed in Appendix D). Fig. 3 plots the comparison of inference throughput and energy. It can be found that our SEENN-I simultaneously improves inference speed while reducing energy costs. Meanwhile, the policy network in SEENN-II only brings marginal effect and does not impact the overall inference speed and energy cost, demonstrating the efficiency of our proposed method. We show ImageNet results in Appendix C.

---

4.4 Ablation Study
In this section, we conduct the ablation study on our SEENN. In particular, allow SEENN to flexibly change their balance between the accuracy and the number of timesteps. For example, SEENN-I can adjust the confidence score threshold and SEENN-II can adjust the penalty value defined in the reward function. To demonstrate the impact of different utilize SEENN-I evaluated with different a. Fig. 4 shows the comparison. The yellow line denotes the trained SNN with a fixed number of timesteps while the blue line denotes the corresponding the ImageNet dataset and the ResNet-19 19 on the CIFAR 10 dataset. For both datasets, our SEENN-I has a higher accuracy - T curve than the vanilla SNN, which confirms that our SEENN improves the iccuracy-efffiiiess tradeoff. Moreover, our SEENN-I largely reduces the distance between the AET coordinate and the accuracy- T curve, meaning that our method is approaching the upper limit of the early exit. On the right side of the Fig. 4, we additionally draw the composition pie charts of SEENN-I, which shows how many percentages of inputs are using 1, 2, or 4 timesteps, respectively. can be shown that, as we gradually adjust a from 0.4 to 0.9 for the JEW-ResNet-34 334, the percentage of inputs using I timestep decreases (73.4% to 30.7%). For the CIFAR 10 dataset, we find images have a higher priority in using the first timestep, ranging from 95.4% to 56.2%.
selection, we
SEENN-I with 6 different thresholds. We test the NeW-ResNet-34
68
67
67.99
Accuracy (%)
22
67.11
66
65.66
Ang 1:333
L:49
Ang L.98
64
ImageNet SEW ResNetts $4
62-
AET. 2.055
660.75
60
2.00
ANg 2:23
96.5
05.44
96.45
3,599
Accuracy (%6
96.0
96.27
a= 0.999
16.26
3577
95.5
35:64
CIFAR-10 Reshel-119
95.0
95.01
AETT 1.131
Figure 5: Qualitative assessment using in- put images from ImageNet dataset. We se- Figure 4: Left: Accuracy vs. the number of timesteps curve. Right: lect images from orange, cucumber, bubble, Pie charts indicating the composition of input images using different brocooli, aireraft carrier, and torch classes numbers timesteps for inference.
3
Number of Timesteps
and separate them according to SEENN.
4.5 Qualitative Assessment
In this section, we conduct a qualitative assessment of SEENN by visualizing the input images that separated by our SEENN-II, the policy network. Specifically, we take the policy network and let it output the number of timesteps for each image in the ImageNet validation dataset. In principle, the policy network can differentiate whether the image is easy or hard so that easy images can be inferred with less number of timesteps and hard images can be inferred with more number of timesteps. Fig. 5 provides some examples of this experiment, where images are chosen from orange, cucumber. bubble, broccoli, aircraft carrier, and torch classes in the ImageNet validation dataset. We can find that T 1 (easy) images and T' an example, the orange, cucumber, and broccoli images from T 1 row are indeed easier to be identified, as they contain single objects in a clean background. However, in the case of 4, there are many irrelevant objects overlapped with the target object, or there could be many small samples of target objects which makes it harder to identify. For instance, in the cucumber case, there are other vegetables that increase the difficulty of identifying them as cucumbers. These results confirm our hypothesis that visually simpler images are indeed easier and can be correctly predicted using a fewer number of timesteps.
(hard) images have huge visual discrepancies. As

---

Conclusion
In this paper, we introduce SEENN, a novel attempt to allow a varying number of timesteps on an input-dependent basis. Our SEENN includes both a post-training approach (confidence score thresholding) and an early-exit-aware training approach (reinforcement learning for selecting the appropriate number of timesteps). Our experimental results show that SEENN is able find sweet spot that maintains accuracy while improving efficiency. Moreover, we show that the number of timesteps selected by SEENNS is related the visual difficulty of the image. By taking an input-by-input approach during inference, SEENN is able to achieve wate-of-theeane accuracy with less computational resources.
Acknowledgment
This work was supported in part by CoCoSys, a JUMP2.0 center sponsored by DARPA and SRC, the National Science Foundation (CAREER Award, Grant #2312366, Grant #2318152), TII (Abu Dhabi), and the DoE MMICC center SEA-CROGS (Award #DE-SC0023199)
References
[1] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 2015.
[2] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint arXiv: /.0iv::550..00 150.000/99 2015.
[3] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike- based machine intelligence with neuromorphic computing. Nature, 575(7784):607-667,, 2019.
[4] Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothee Masquelier, and Anthony Maida. Deep learning in spiking neural networks. Neural Networks, 111:47-63, 2019.
[5] Yufei Guo, Xuhui Huang, and Zhe Ma. Direct learning- -based deep spiking neural networks: a review. Frontiers in Neuroscience, 17:1 17:1209795, 2023.
[6] Laxmi R Iyer and Yansong Chua. Classifying neuromorphic datasets with tempotron and spike timing dependent plasticity. In 2020 International Joint Conference on Neural Networks (ICNNN, pages 1-8. JEEE, 2020.
[7] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training nigh-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
[8] Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer eлror reassignment in time, In Advances in Neural Information Processing Systems, pages 1412-1421, 2018.
[9] Peter U. Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, and Shih Chii Liu. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In Neural Networks (IJCNN), 2015 International Joint Conference on, 2015.
[10] Bodo Rueckauer, AlexanAl Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion of analog to spiking convolutional neural networks. arXiv: Statistics/Machine Learning, (1612.04052):0 0, 2016.
[11] Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in Neuroence, 13, 2018.
[12] Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. European Conference on Computer Vision, 2020.
[13] Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. arXiv preprini arXiv:2
2021.
[14] Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In International Conference on Machine Learning, pages 6316-6325. PMLR, 2021.
10

---

[15] Bodo Rueckauer, aulia-Alexandra Lungu, Yubuang Hu, Michael Pfeiffer, and Shih- Chih-Chii Liu. Conversion of continuous- valuud deep networks efficient event-driven networks for image classification. Frontiers in neuroscience, 1:682, 2017.
[16] Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy, Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13:95, 2019.
[17] Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13558-13567, 2020.
[18] Tong Bu, Wei Fang, Jianhao Ding, PENGLIN DAI, Zhaofei Yu, and Tiejun Huang. Optimal ANN-SNN conversion for high-accuracy and ultra-low-latens -laenncyy spiking neural networks. In International Conference on Learning Reprexentations, 2022.
[19] Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations, 2019.
[20] Jiangrong Shen, Qi Xu, Jian K Liu, Yueming Wang, Gang Pan, and Huajin Tang. Esl-snns: An evolutionary structure learning strategy for spiking neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 86-93, 2023.
[21] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly- larger spiking neural networks. arXiv preprint arXiv:201 1.05280, 2020.
[22] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1311-1318. 2019.
[23] Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems, 2017
[24] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 2661-2671, 2021.
[25] Nitin Rathi and Kaushik Roy. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[26] Youngeun Kim and Priyadarshini Panda. Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in neuroscience, page 1638, 2021.
[27] Youngeun Kim and Priyadarshini Panda. Optimizing deeper spiking neural networks for dynamic vision sensing. Neural Networks, 2021.
[28] Qi Xu, Jiangrong Shen, Xuming Ran, Huajin Tang, Gang Pan, and Jian K Liu. Robust transcoding sensory information with neural spikes. IEEE Transactions on Neural Networks and Learning Systems, 3335):193511946 2021.
[29] Qi Xu, Yaxin Li, Jiangrong Shen, Jian K Liu, Huajin Tang, and Gang Pan. Constructing deep spiking neural networks from artificial neural networks with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7886-7895, 2023.
[30] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re- weighting. arXiv preprint arXiv: 200.:2222./1446,, 2022.
[31] Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differ- entiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021.
[32] Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
11

---

[33] Yufei Guo, Xinyi Tong, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Zhe Ma, and Xuhui Huang. Recdis-snn: Rectifying membrane potential distribution for directly training spiking neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 326-335, 2022.
[34] Yufei Guo, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Yinglei Wang. Xuhui Huang, and Zhe Ma IM-loss: Information maximization loss for spiking neural networks. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[35] Yufei Guo, Yuhan Zhang, Yuanpei Chen, Weihang Peng, Xiaode Liu, Liwen Zhang, Xuhui Huang, and Zhe Ma. Membrane potential batch normalization for spiking neural networks. arXiv preprint arXiv:2308.088559 2023.
[36] Yizeng Han, Gao Huang, Shiji Song. Le Yang. Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
[37] Surat Teerapittayanon, Bradley McDanel, and Hsiang- Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 1464-2465 IEEE, 2016.
[38] Priyadarshini Panda, Abhronil Sengupta, and Kaushik Roy. Conditional deep learning for energy-efficient and enhanced pattern recognition. In 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE), pages 475-480. IEEE, 2016.
[39] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 409-424, 2018.
[40] Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8817-8826, 2018
[41] Brandon Yang, Gabriel Bender, Quoe V Le, and Jiquan Ngiam. Condconv: Conditionally parameterized convolutions for efficient inference. Advances in Neural Information Processing Systems, 32, 2019.
[42] Jie Hu, Li Shen, and Gang Sun. srncezz--nddexciieee networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[43] Anthony Burkitt. review of the integrate-ang and---nnd-e neuron model: homogeneous synaptic input. Biological cybernetics, 95(1):1-19, 2006.
[44] Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, and Priyadarshini Panda. Neural architecture search for spiking neural networks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 36-56. Springer, 2022.
[45] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
[46] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press. 2018.
[47] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 (canadian institute for advanced research). URI. http://www. cs. toronto. edu/kriz/cifar html, 5(4):1, 2010.
[48] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[49] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifarl 0-dvs: an event- stream dataset for object classification. Frontiers in neuroscience, 11:309, 2017.
[50] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprins arXiv: 06600.00833... 2016.
[51] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1758.00000.4552, 2017.
12

---

[52] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 113-123, 2019.
[53] Yufei Guo, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Yinglei Wang, Xuhui Huang, and Zhe Ma. Im-loss: information maximization loss for spiking neural networks. Advances in Neural Information Processing Systems, 35: 35:156-166, 2022.
[54] Sayeed Shafayet Chowdhury, Nitin Rathi, and Kaushik Roy. Towards ultra low latency spiking neural networks for vision and sequential tasks using temporal pruning. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XI, pages 709-726. Springer, 2022.
[55] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Sun Jian. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[56] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
[57] Gourav Datta, Zeyu Liu, and Peter A Beerel. Hoyer regularizer is all you need for ultra low-latency spiking neural networks. arXiv preprint arXiv:22/2.10170, 2022.
(58) Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34:21056-21069, 2021.
13

---

A Verification of Assumption 3.1
In Assumption 3.1, we conjecture that a correct prediction made by fx),,, then for all 1' > 1 that fe (x) is also correct. And based this assumption, propose the equation compute AET, i.e Eq. (4). Here, to validate if our assumption holds, we propose another metric: empirical AET, given by
N Σ 4
AET
(13)
N
where & is the actual earliest number of timesteps that predicts the correct class for the i-th sample. Note that, similar to AET, if the network cannot make correct predictions in any timesteps, then we set & to the maximum number of timesteps.
Consider an example that does not satisfy Assumption 3.1, for example, the prediction results for the first 4 timesteps are (False, True, False, True). The empirical AET will obtain 2 while the AET will obtain 2
= Therefore, by comparing the difference between the AET and the empirical AET we can verify if Assumption 3.1 holds. Table 5, as shown below, demonstrates the AET and the empirical AET comparison across different models and datasets. can find that difference is very small, often less than 0.05. The only dataset that creates slightly larger difference is CIFARIO-DY Indeed, the event-strean dataset may provide more variations over time. Nevertheless, the potential for early exit is still high.
Table Comparison between ATE and empirical AET across models and datasets.
Model
Dataset
max T
AET
AET
Direct Training of SNNs
ResNet-19
CIFAR-10
4
1.1309
1.1188
ResNet-19
CIFAR-100
1,6075
1.5616
ResNer-34
ImageNet
6
2.9026
2.7781
SEW-ResNet-34 ImageNet
4
2,0646
2.0092
VGGSNN
CIFARIO-DVS
10
3.096
2.774
B Distribution of Confidence Scores
In Fig. 2, we demonstrate the distribution of confidence scores of ResNet-19 on the CIFAR-10 validation dataset. Obviously, with a higher number of timesteps more input data have higher confidence scores. This aligns with our target to filter out easy input data.
6000
4000
T=1
T=1
T=2
T=2
T=3
T=3
#samples
3000
T=4
#samples
4000
T=4
2000
2000
1000
0
0.00
0.25
0.50
0.75
1.00
0.9999
1.0
Confidence score
Confidence score
Figure 6: The distribution of confidence scores of ResNet-19 19 on CIFAR- 1000 left: |0, 0.9999), right: (0.9999, 1]
14

---

C Hardware Efficiency on ImageNet
We demonstrate the results in Fig. 7.
SNN
Acc 68.00, T=4
SEENN-II
Acc 67.48, T=1.79
SEENN-I
Acc 66.21, T=1.66
SNN
Acc 60.75, T=1
500
1000
1500
2000
2500
3000
Energy per sample (mJ)
Figure 7: Hardware efficiency ImageNet dataset.
D Hardware Evaluation
For latency measurement, we directly tested on GPU with the Pytorch framework. The latency measurement is computed across the whole test dataset. For example, the throughput calculation is given by
Throughput = Test set inference time
(14)
Number of test samples'
For energy estimation, we adopt the conventional way to measure the addition and multiplication operations for the entire SNN inference. The addition costs 0.9p.J and the multiplication costs 4.6p.,, respectively.
E
Architecture Details
The major network architecture we adopt in this work is ResNet-series architecture. There are many variants of ResNets used in the field of SNNs, e.g. ResNet-18, 18, ResNet-19, and ResNet-20. They also mixedly referred to in existing literature [18, 21, 10,0,,55) Therefore, to avoid confusion these the details of the network configurations.
ResNet architectures, in this section,
Table 6 summarizes the different ResNets we used in our CIFAR experiments. For ImageNet models, the standard ResNet-34 [55] and SEW-ResNet-3e [58] which are well-defined. The basic difference between the four ResNets the channel configurations and the block configurations. Therefore, their actual FLOPs difference lot larger than the difference suggested by their names. For example, our policy network (ResNet-8) only contains 0.547% number of operations of ResNet- 19, Therefore, the cost of rumning SEENN-II is almost negligible.
15

---

Table 6: The architecture details of ResNets. denotes that the first residual block contains downsample layer reduce feature resolution.
ResNet-8
ResNet-18
conv 1
3 16
3 3, 64, sl
3 x 3, 16
x 3, 64
stagel
3 x 3, 16
x
x 64
x
32 3 x 3,32
3 x 3, 128
stage2
1
2
3 x 3, 128
3 x 3, 64 x 3, 64
3 x 3, 256
stage3
x 1
x 2
3 x 3, 256
3 x 3, 512
stage4
N/A
2
3 3, 512)
pooling
Global average pooling
classifier
(64, O) FC
(512, O) FC
ResNet-19
ResNet-20
conv1 1
3 x 3, 128, s1
3 x 3, 16, s1
3 3, 128 3 x 3, 128
3 x 3, 16 3 x 3, 16
stagel
3
3 3, 256 3,256
3 3, 32
stage2
X 3
2
3 3, 32
3 512 3 3, 512
3 3, 64
stage3
x 2
x 2
3 3, 64
pooling
Global average pooling
classifier
(512, O) FC
(64, O) FC
16


Reviews:
[{"review_id": "bEZ6dMV5b6", "reviewer": "Reviewer_ZThW", "summary": "This paper proposes a novel manner to improve the efficiency of the spiking neural network. Specifically, the SEENN is proposed to determine the appropriate number of timesteps, which therefore reduce the inference time-cost. The proposed method is evaluated on CIFAR-10 and Imagenet, achieving good performance.", "questions": "See weakness.", "limitations": "None.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The main contribution of this paper is to treat the number of timesteps as a variable in SNN model. Accordingly, several variations of early-exit manner is deigned for better accuracy-efficiency tradeoff.\n+ The paper writing and organization is good.", "weaknesses": "- The proposed method for determining the best timesteps is not novel. The first manner is to simply set the right number with the confidence score, while the second one introduces a policy network for better prediction. Although above two manners could work, these would inevitably introduce extra computation cost or human-based prior, which is not expected. In addition, the proposed SNN model still rely heavily on the ANN backbones, e.g., Resnet or VGGnet.\n- As for the hardware efficiency, the adopted nvidia V 100 maybe not proper, which is not designed for SNN application.\n- The ablation study is not clear, it seems the SEEN adopts different backbones with SNN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts in reviewing our article and providing constructive feedback. We’d like to reply to your concerns in detail. \n\nQ1: The proposed method for determining the best timesteps is not novel. The first manner is to simply set the right number with the confidence score, while the second one introduces a policy network for better prediction. Although above two manners could work, these would inevitably introduce extra computation cost or human-based prior, which is not expected. In addition, the proposed SNN model still rely heavily on the ANN backbones, e.g., Resnet or VGGnet.\n\nA1: Thanks for your question. Our proposed methodology aims to minimize the extra computation cost of SEENN. For SEENN-I, calculating the confidence score is quite fast, only involving a softmax and a max function to determine the score, which is negligible compared to the whole network inference (Eq. 5). For SEENN-II, we use an extremely tiny network (ResNet-8) with 8x fewer channels than ResNet-19 (See details in our appendix C), which only occupies 0.5% FLOPs of the ResNet-19. In our experiments, we compared the energy/latency with static SNN baselines and found that SEENN incurs much lower hardware costs even with these extra computations.\n\nAs for the human prior problem, we should clarify that this is the same problem with **all** SNNs works. Under the static time steps setting, it is required to pre-set the number of timesteps for the SNN before the inference. As such, the human prior is needed to choose more efficiency or more accuracy by changing the number of timesteps. In our SEENN, this is similar to setting the $\\alpha$ or $\\beta$ to balance efficiency and accuracy. \n\nFor the architecture concern, we’d like to emphasize that our work adjusts the timesteps of any SNN, agnostic to the SNN architecture used. For example, we can use the SNAS-Net [1] on the CIFAR10, a work that searches for unique SNN network architecture. We apply our early exit mechanism to it. The results are shown below: \n\n| SNASNet |       | SEENN SNASNet  |       |\n|---------|-------|----------------|-------|\n| T=1     | 73.69 | T=1.39         | 84.59 |\n| T=2     | 84.79 | T=1.99         | 90.85 |\n| T=3     | 90.87 | T = 2.36       | 92.21 |\n| T=4     | 92.66 | T = 2.76       | 93.05 |\n| T=5     | 93.39 | T = 3.14       | 93.37 |\n\nIn addition to NAS-based networks, we refer you to our reply to Reviewer DV4t where we provide the results on the Transformer based network architecture. \n\nQ2: As for the hardware efficiency, the adopted nvidia V100 maybe not proper, which is not designed for SNN application.\n\nA2: Thanks for your question. It is true that Nvidia GPUs are not optimized for SNNs. Therefore, most SNN acceleration work cannot be tested on GPUs. However, in this paper, we show that reducing the number of timesteps of SNNs can lead to acceleration on any inference platform, including GPUs that are not even optimized for SNNs. Our SEENN can be inherently accelerated on other platforms like neuromorphic hardware and in-memory computing (IMC) architectures. Here, we show the implementation of SEENN on a publicly available IMC architecture simulator [2]. The experiments use ResNet-19 architecture on the CIFAR10 dataset. Besides the accuracy metric, we use the relative Energy-Delay-Product (EDP) with respect to Static SNN at 4 timesteps as the hardware metric: \n\nThe results are shown in the table below. It can be found that our SEENN also shares a significant amount of acceleration on other hardware platforms.\n\n| Method     | T    | Accuracy | Relative EDP (%) |\n|------------|------|----------|------------------|\n| Static SNN | 1    | 95.01    | 9.6%             |\n| Static SNN | 2    | 95.64    | 30.0%            |\n| Static SNN | 3    | 96.26    | 57.3%            |\n| Static SNN | 4    | 96.46    | 100%             |\n| SEENN-I    | 1.09 | 96.07    | 11.8%            |\n| SEENN-I    | 1.20 | 96.38    | 13.8%            |\n\nQ3: The ablation study is not clear, it seems the SEEN adopts different backbones with SNN.\n\nA3: We apologize for any confusion in the ablation study. But we do use identical backbones when comparing SNNs and SEENNs. Therefore, the acceleration effect brought by our proposed method and the comparison is fair and clear. We will change our description and clarify it in the revised version of the manuscript. \n\n*Reference*\n\n[1] Kim, Youngeun, et al. \"Neural architecture search for spiking neural networks.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Moitra, Abhishek, et al. \"Spikesim: An end-to-end compute-in-memory hardware evaluation tool for benchmarking spiking neural networks.\" IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (2023).\n\n"}, {"review_id": "CGxvk6Ih7b", "reviewer": "Reviewer_J7UY", "summary": "This paper introduces a new inference scheme for spiking neural networks (SNNs), the early exit on the time dimension. To make sure that relatively easy images can be predicted with less number of timesteps, this work build two frameworks to identify the appropriate timestep to minimize the latency while maintaining decent performance. The first method is an on-the-fly approach and the second one is more complicated that requires finetuning.  The authors test their SEENN on various recognition benchmarks and obtain quite good accuracy with even lower number of timesteps compared to existing papers. ", "questions": "Can SEENN-II be applied to the conversion approach? if not, why is that, please elaborate it.\n\nI suggest the following response about this problem is discussed into the revision.", "limitations": "see above.\n", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Compared to other SNNs papers that focus on either conversion and training, what is work studied is a universal approach (i.e. time) that can be applied to any types SNNs.\n2. The methodology is technically sound and covers different user resources (whether finetune or just plug-and-play)\n3. Experiments results is thorough and solid and verify the effectiveness of this approach.", "weaknesses": "1. There is no empirical evidence showing that the confidence score will increase along with the number of timesteps. The authors are suggested to add a figure to show the confidence score evolution.\n\nThe above limitations has been explained in the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback on our work. Please check our response to your questions and concerns. \n\nQ1: There is no empirical evidence showing that the confidence score will increase along with the number of timesteps. The authors are suggested to add a figure to show the confidence score evolution.\n\nA1: Thanks for your suggestion. We agree with the reviewer that a visualization of the confidence score would help the readers understand our SEENN-I mechanism. Here, we draw the confidence score distribution on test images. We split the range into [0, 0.9999] and [0.9999, 1.0], otherwise the figure would distort too much. We put the figure into the rebuttal PDF file, please refer to it there. \n\nIt can be clearly observed that the number of test samples moving to [0.9999, 1.0]  has increased if we increase the number of timesteps. This means that the network prediction is getting more confident as we increase the number of timesteps. \n\nMoreover, we also measure the mean/variance of confidence scores over the test batches, as shown below. We can find the mean/variance continues to increase/decrease as we increase the number of timesteps.\n\n| Timesteps   | 1     | 2     | 3     | 4     |\n|-------------|-------|-------|-------|-------|\n| CS Mean     | 0.816 | 0.923 | 0.953 | 0.964 |\n| CS Variance | 0.059 | 0.024 | 0.014 | 0.010 |\n\nQ2: Can SEENN-II be applied to the conversion approach? if not, why is that, please elaborate it.\n\nA2: Thanks for this question. In SEENN-II, we jointly optimize the policy network and the SNNs. If we apply SEENN-II to converted SNNs, we can focus on training the policy network only and keep the converted SNNs frozen. \n\nHowever, many computation resources and the whole training data are required to train the policy network. This setup breaks the assumption that ANN-SNN conversion is done when there are limited computation resources and training data. Therefore, we did not include the SEENN-II to ANN-SNN conversion methods in our initial draft because we think the comparison is unfair. \n\nTo demonstrate that SEENN-II can work effectively with the converted model, we’d like to provide a result for QCFS-based (Bu et al., 2022b) converted ResNet-18 on the CIFAR-10 dataset. We train a policy network and the average predicted number of timesteps is 1.35, achieving 94.43% accuracy. The improvement is consistent with the results shown in our paper. \n\n\n"}, {"review_id": "EVfbz7yfH4", "reviewer": "Reviewer_DV4t", "summary": "This paper proposes an accuracy-efficiency tradeoff method by adjusting the number of timesteps in SNNs, which is new and interesting. The authors accomplish this idea with two methods that uses a confidence score thresholding and reinforcement learning. These methods can be applied to directly training SNN and the ANN-SNN both, and results show these methods can save energy greatly but with negligible accuracy decreasing. I like the work that has potential and would provide a new direction for the following SNN work. ", "questions": "1.Can this method be used to other backbones, like transformer?\n2.What is the performance of the method using vanilla CE loss?", "limitations": "1.I find no potential negative societal impact.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. well-written, easy to read.\n2.The idea to adjust the number of timesteps to balance accuracy-efficiency is new and interesting.\n3. The method is simple and easy to follow.\n4. Experimental results are really good. The method can keep similar performance while reduce cost.", "weaknesses": "To show the effectiveness of the method. A similar timesteps for other methods should be provided, for example, 2 or 3 timesteps for TET on ImageNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback on our work. Please check our response to your questions and concerns. \n\nQ1: To show the effectiveness of the method. A similar timesteps for other methods should be provided, for example, 2 or 3 timesteps for TET on ImageNet.\n\nA1: Thanks for the suggestion. We report the TET SEW-ResNet-34 (denoted as Static SNN) accuracy from T=1 to T=4 on the ImageNet dataset as well as our SEENN accuracy. The results are shown in the table below. \n\n| Method     | T     | Accuracy |\n|------------|-------|----------|\n| Static SNN | 1     | 60.78    |\n| Static SNN | 2     | 65.74    |\n| Static SNN | 3     | 67.11    |\n| Static SNN | 4     | 68.00    |\n| SEENN-I    | 1.66  | 66.20    |\n| SEENN-I    | 2.35  | 67.99    |\n| SEENN-II   | 1.79  | 67.48    |\n\nQ2: Can this method be used to other backbones, like transformer?\n\nA2: Yes, our method can be seamlessly integrated into any SNN backbones with any number of timesteps. For example, we train a SpikeFormer [1] on the CIFAR10 dataset and demonstrate the improvement from our SEENN-I in the following table. \n\n| SpikeFormer |       | SEENN SpikFormer |       |\n|-------------|-------|------------------|-------|\n| T=1         | 89.39 | T=1.23           | 93.47 |\n| T=2         | 93.96 | T=1.35           | 93.91 |\n| T=3         | 94.30 | T=1.82           | 94.28 |\n| T=4         | 94.51 | T=2.11           | 94.49 |\n\nQ3: What is the performance of the method using vanilla CE loss? \n\nA3: This is a good question. When using vanilla CE loss, the performance of early timesteps will inevitably decrease, thus damaging the early exit performance. We refer you to our reply to Reviewer ZThW where we show the SEENN performance on a pretrained SNASNet checkpoint, which is trained with vanilla CE loss function. It can be found that the acceleration of SEENN is still effective. However, the reduction in timesteps is relatively smaller, presumably brought by a higher AET value (i.e., Eq. 4 in the main manuscript). \n"}, {"review_id": "0y6VXJGnYO", "reviewer": "Reviewer_Sp8z", "summary": "This paper considers to adaptively determine inference time steps of spiking neural networks to improve the tradeoff between accuracy and time. Two methods are proposed. The first one uses confidence score thresholding. The second one introduces an additional policy network to predict the number of timesteps by reinforcement learning. Experiments show the effectiveness of the proposed methods for direct SNN training and ANN-SNN conversion methods.", "questions": "Can the authors discuss more about conceptually how to determine the inference of (deep) SNNs (in small time steps) with confidence signals from the top layer? Note that the common time steps of SNNs simply refer to time steps for each layer and do not consider propagation across layers. However, if we consider asynchronous neurons, e.g. on neuromorphic hardware, it also takes time to propagate signals across layers, and the actual delay may be layer number plus time steps per layer. For deep SNNs (e.g. >18 layers in this paper), it takes much time to obtain the confidence score from the top layer, which may be longer than the small time steps per layer. Then how can it effectively control such small time steps?", "limitations": "The authors do not discuss limitations and societal impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper considers early-exit of spiking neural networks, which can effectively improve the efficiency of SNNs. The idea of dynamically determining inference time steps based on the difficulty of inputs is interesting.\n\n2. Extensive experiments on static and neuromorphic datasets as well as qualitative assessment are conducted.\n", "weaknesses": "1. Dynamic inference time steps of SNNs, especially based on confidence scores, were also explored by (possibly concurrent) recent works [1,2], which can be discussed. The idea of confidence score is very simple and straight-forward.\n\n2. For SEENN-II, an additional policy network is required for inference, and it is unclear how this can be deployed. From the descriptions (and codes), the policy network is an ANN rather than SNN, so it is not compatible with the main SNN for hardware deployment. This poses challenges and also arouses questions of why considering such hybrid architectures. Additionally, SEENN-II seems not to be flexible, i.e. after training it cannot make tradeoff between accuracy and time/energy.\n\n3. From the descriptions, it is not clear enough whether the energy consumption estimation of SEENN-II consider policy networks. This should be included and more detailedly discussed. For example, on ImageNet, is the policy network an ANN ResNet-34 (as shown in code)? Then it may consume more than the SNN part. The energy result on ImageNet is missing.\n\n[1] Wu, D., Jin, G., Yu, H., Yi, X., & Huang, X. (2023). Optimising Event-Driven Spiking Neural Network with Regularisation and Cutoff. arXiv preprint arXiv:2301.09522.\n\n[2] Li, C., Jones, E., & Furber, S. (2023). Unleashing the Potential of Spiking Neural Networks by Dynamic Confidence. arXiv preprint arXiv:2303.10276.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts in reviewing our article and providing constructive feedback. We’d like to reply to your concerns in detail. \n\nQ1: Dynamic inference time steps of SNNs were also explored by (possibly concurrent) recent works [1,2], which can be discussed. \n\nA1: Thank you for pointing out the concurrent works [1] and [2]. We were unaware of these papers during our manuscript preparation, and we appreciate the opportunity to clarify the distinctions between our work and theirs.\n\nWhile [1] and [2] concentrate on converted SNNs with early exit using post-training methods, our SEENN approach is applied to both conversion and direct training of SNNs. Their methods rely on metrics like spike differences and dynamic confidence scores, whereas we utilize confidence score metrics. Beyond the post-training approach, we introduce a training-aware method (SEENN-II) that leverages reinforcement learning to optimize early exit performance. This innovation enhances accuracy within the same number of timesteps. Our work provides a comprehensive empirical comparison with existing SNN literature, encompassing both conversion and direct training methodologies.\n\nWe acknowledge the relevance of [1] and [2] and will incorporate references to these papers in our revised manuscript. This inclusion will enrich the context and highlight the unique contributions of our SEENN approach.\n\nQ2: For SEENN-II, an additional policy network is required for inference, and it is unclear how this can be deployed. Additionally, SEENN-II seems not to be flexible. \n\nA2: Thanks for the question. Choosing the policy network architecture is open to discussion. Technically, any neural network would suffice this role. Indeed, if we use two SNNs it might have better hardware compatibility. Then, the question is whether we should use the early exit on policy network architecture and so on. We have avoided this recursive paradox and used an extremely small ANN (ResNet-8 with much fewer channels), which only occupies 0.5% of the FLOPs of the ResNet-19. \n\nMeanwhile, we would also like to point out existing work that tries to involve both ANNs and SNNs to enhance the representation ability and efficiency. It is possible to use two types of neural networks on the edge devices collaboratively, e.g. [3]. \n\nAs for the tradeoff problem, we agree with this notice. Here, we want to clarify that a training-aware approach for SNNs has always been fixed to certain timesteps. To be more specific, in the direct training of SNNs, all existing works report the test accuracy when using the same $T$ in training. Even though they can test under lower timesteps, their reported accuracy on the different numbers of timesteps always uses different networks trained with the corresponding numbers of timesteps. We argue that this is a common problem for all training-aware approaches. Post-training algorithms, like our SEENN-I and conversion methods, can flexibly trade off between accuracy and time. In conclusion, our SEENN-I and SEENN-II align with the existing SNNs method in terms of practicality. \n\n\nQ3: From the descriptions, it is not clear enough whether the energy consumption estimation of SEENN-II consider policy networks. The energy result on ImageNet is missing.\n\nA3: We included all costs from policy networks and in line 277 we described that “Meanwhile, the policy network in SEENN-II only brings marginal effect and does not impact the overall inference speed and energy cost”. This is due to the extremely efficient design of the policy network architecture. \n\nTo measure the energy, we used the same energy per operation times the number of operations as in [27], where the ANN policy network architecture uses multiplications and additions as well. In our code, we use ResNet-8 with an 8x fewer channel size (when compared to ResNet-19) for the policy network, occupying 0.5% FLOPs of the ResNet-19. Hence, the actual cost induced by the policy network has been made as minuscule as possible. Here, we report the energy results on the ImageNet dataset in the rebuttal PDF file. \n\nQ4: Can the authors discuss more about conceptually how to determine the inference of (deep) SNNs (in small time steps) with confidence signals from the top layer?  How can it effectively control such small time steps?\n\nA4: This is a good question. To the best of our understanding, the reviewer is asking about an asynchronous pipeline inference implementation, where later timesteps start getting processed before the current timestep reaches the top layer and causes unnecessary energy waste.  \n\nFirst, we would like to clarify that not all hardware devices implement this technique, since it requires lots of area to map the full network to the hardware. For example, Loihi2 [4] only has 8192 neurons in one core, which is not enough for an 18-layer network. \n\nSecond, if we have to use SEENN on this type of hardware. We can avoid unnecessary energy waste with several methods. (1) Instead of sending the next timestep of current input to the pipeline, we can send the next input data. The next timestep of the current input will be processed only when the confidence score is computed. We put a figure in the rebuttal PDF to show this scheme. \n(2) Determine the confidence score at early layers. This can be done by training a classifier in the early layers (similar to ANN early exit) and using the signal from early layers to determine the exit timesteps. (3) Predict timesteps before the inference as we did in SEENN-II.\n\nQ5: The authors do not discuss limitations and societal impact.\n\nA5: We apologize for not mentioning the limitations and broader impact. Please check our general response. \n \nReference:\n\n[3] Zhao, Rong, et al. \"A framework for the general design and computation of hybrid neural networks.\" Nature communications 13.1 (2022): 3427.\n[4] Davies, Mike. \"Taking neuromorphic computing to the next level with Loihi2.\" Intel Labs’ Loihi 2 (2021): 1-7.\n\n\n \n\n\n"}]

Author rebuttal:
We’d like to thank all reviewers for their constructive feedback and suggestions on our work. We will address each reviewer’s questions and concerns point-to-point. And we welcome any further discussion on our paper.  We have attached a PDF file for demonstrating 4 figures. A detailed description can be found in each rebuttal thread. 

Here, we'd like to reply to a concern that we did not include limitations and potential negative social impact. Our work focuses on reducing the inference cost of SNNs that can potentially benefit their deployment on edge hardware. We think it does not bring potential negative impacts on society. For limitations, SEENN uses hyper-parameter $\alpha, \beta$ to control the accuracy-efficiency tradeoff, which is less straightforward to users and may require certain trial and error. 

Meta-review: This paper received all four positive review scores. In the first round, reviewers raised some critical questions about the method's novelty, energy consumption, and experimental analyses. The authors provided detailed one-to-one responses to answer these questions. After the rebuttal and discussion, reviewers think the concerns are addressed and the opinions are converged to accept. The authors are encouraged to include the rebuttal materials in the revision, and the final recommendation is Accept.
Decision: Accept (poster)