SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Accuracy-  $T$  comparison of ANN-SNN conversion SNN methods on CIFAR datasets.
Method	Model	CIFAR-10	CIFAR-100
Acc1(T1)	Acc2(T2)	Acc3(T3)	Acc1(T1)	Acc2(T2)	Acc3(T3)
ResNet-20	Opt. [13]	92.41 (16)	93.30 (32)	93.55 (64)	63.73 (16)	68.40 (32)	69.27 (64)
Calibration [14]	94.78 (32)	95.30 (64)	95.42 (128)	-	-	-
OPI [30]	75.44 (8)	90.43 (16)	94.82 (32)	23.09 (8)	52.34 (16)	67.18 (32)
ResNet-18	QCFS [18]	75.44 (2)	90.43 (4)	94.82 (8)	19.96 (2)	34.14 (4)	55.37 (8)
SEENN-I (Ours)	91.08 (1.18)	93.63 (1.40)	95.08 (2.01)	39.33 (2.57)	56.99 (4.41)	65.48 (6.19)
We also provide the details of architecture in Appendix E and measure the latency/energy of the policy network in Sec. 4.3. For the ImageNet dataset, we downsample the image resolution to  $112 \times 112$  for the policy network. Both the policy network and the SNN are finetuned for 75 epochs using a learning rate of 0.01. Other training hyper-parameters are kept the same as SEENN-I.
4.2 Comparison to SOTA work
CIFAR Training We first provide the results on the CIFAR-10 and CIFAR-100 datasets [47]. We test the architectures in the ResNet family [55] and VGG-16 [56]. We summarize both the direct training comparison as well as the ANN-SNN conversion comparison in Table 1 and Table 2, respectively. Since the  $T$  in our SEENN can be variable based on the input, we report the average number of timesteps in the test dataset. As we can see from Table 1, existing direct training methods usually use 2, 4, and 6 timesteps in inference. However, increasing the number of timesteps from 2 to 6 brings marginal improvement in accuracy, for example,  $95.45\%$  to  $95.60\%$  in TEBN [32]. Our SEENN-I can achieve  $96.07\%$  with only 1.09 average timesteps, which is  $5.5 \times$  lower than the state of the art. SEENN-II gets a similar performance with SEENN-I in the case of CIFAR-10, but it can achieve  $0.7\%$  higher accuracy than SEENN-I on the CIFAR-100 dataset.
Comparison with 1-Timestep SNN Reducing the number of timestep to 1 is another way towards low latency SNN [54, 57]. However, this method ignores the sample-wise difference and may lead to a larger accuracy drop. In Table 1, we compare VGG-16 with even 1 timestep [54]. Our SEENN-I achieves  $8.5\%$  accuracy improvement at the cost of  $12\%$  more latency on the CIFAR-100 dataset.
CIFAR Conversion We also include the results of ANN-SNN conversion using SEENN-I in Table 2. Here, the best existing work is QCFS [18], which can convert the SNN in lower than 8 timesteps. We directly run SEENN-I with different choices of confidence score threshold  $\alpha$ . Surprisingly, on the CIFAR-10 dataset, our SEENN-I can convert the model with 1.4 timesteps, and get  $93.63\%$  accuracy. Instead, the QCFS only gets  $75.44\%$  accuracy using 2 timesteps for all input images. By selecting a higher threshold, we can obtain  $95.08\%$  accuracy with 2.01 timesteps, which uses  $4\times$  lower number of timesteps than QCFS under the same accuracy level.
Table 3: Accuracy-  $T$  comparison on ImageNet dataset.
Model	Method	T	Acc.	Model	Method	T	Acc.
Direct Training of SNNs	ANN-SNN Conversion
ResNet-34	tdBN [21]	6	63.72	ResNet-34	Opt [13]	32	33.01
TET [30]	6	64.79	64	59.52
TEBN [32]	6	64.29	Calibration [14]	32	64.54
SEENN-I (Ours)	2.28	63.65	64	71.12
3.38	64.66	QCFS [18]	16	59.35
SEENN-II (Ours)	2.40	64.18	32	69.37
SEW-ResNet-34	SEW [58]	4	67.04	64	72.35
TET [30]	4	68.00	SEENN-I (Ours)	23.47	70.18
TEBN [32]	4	68.28	29.53	71.84
SEENN-I (Ours)	1.66	66.21
2.35	67.99
SEENN-II (Ours)	1.79	67.48
ImageNet We compare the direct training and conversion methods on the ImageNet dataset. The results are sorted Table 3. For direct training, we use two baseline networks: vanilla ResNet-34 [55] and SEW-ResNet-34 [58]. For SEENN-I, we directly use the pre-trained checkpoint from TET [30]. It can be observed that our SEENN-I can achieve the same accuracy using only  $50\%$  of original number of timesteps. For example, SEENN-I ResNet-34 reaches the maximum accuracy in 2.35 timesteps. SEENN-II can obtain a similar performance by using a smaller  $T$ , i.e. 1.79. For conversion experiments, we again compare against QCFS using ResNet-34. Our SEENN-I obtains  $70.2\%$  accuracy with 23.5 timesteps, higher than a 32-timestep QCFS model.
CIFAR10-DVS Here, we compare our SEENN-I on an event-stream dataset, CIFAR10-DVS [49]. Following existing baselines, we train the SNN with a fixed number of 10 timesteps. From Table 4 we can find that SEENN-I can surpass all existing work except TET with only 2.53 timesteps, amounting to nearly  $4 \times$  faster inference. Moreover, SEENN-II can get an accuracy of 82.6 using 4.5 timesteps.
Figure 3: Comparison of latency (throughput) and energy consumption between SNN and SEENN.
Model	Method	T	Acc.
Direct Training of SNNs
ResNet-19	tdBN [21]	10	67.8
VGG5NN	PLIF [24]	20	74.8
ResNet-18	Dspike [31]	10	75.4
ResNet-19	RecDis-SNN [33]	10	72.4
VGG5NN	TET [30]	10	83.1
VGG-SNN	SEENN-I (Ours)	2.53	77.6
5.17	82.7
SEENN-II (Ours)	4.49	82.6
Table 4: Accuracy-  $T$  comparison on CIFAR10-DVS dataset.
4.3 Hardware Efficiency
In this section, we analyze the hardware efficiency of our method, including latency and energy consumption. Due to the sequential processing nature of SNNs, the latency is generally proportional to  $T$  on most hardware devices. Therefore, we directly use a GPU (NVIDIA Tesla V100) to evaluate the latency (or throughput) of SEENN. For energy estimation, we follow a rough measure to count only the energy of operations that is adopted in previous work [18, 25, 31], as SNNs are usually deployed on memory-cheap devices (detailed in Appendix D). Fig. 3 plots the comparison of inference throughput and energy. It can be found that our SEENN-I simultaneously improves inference speed while reducing energy costs. Meanwhile, the policy network in SEENN-II only brings marginal effect and does not impact the overall inference speed and energy cost, demonstrating the efficiency of our proposed method. We show ImageNet results in Appendix C.
In this section, we conduct the ablation study on our SEENN. In particular, we allow SEENN to flexibly change their balance between the accuracy and the number of timesteps. For example, SEENN-I can adjust the confidence score threshold  $\alpha$  and SEENN-II can adjust the penalty value  $\beta$  defined in the reward function. To demonstrate the impact of different hyper-parameters selection, we utilize SEENN-I evaluated with different  $\alpha$ . Fig. 4 shows the comparison. The yellow line denotes the trained SNN with a fixed number of timesteps while the blue line denotes the corresponding SEENN-I with 6 different thresholds. We test the SEW-ResNet-34 on the ImageNet dataset and the ResNet-19 on the CIFAR10 dataset. For both datasets, our SEENN-I has a higher accuracy -  $T$  curve than the vanilla SNN, which confirms that our SEENN improves the accuracy-efficiency tradeoff. Moreover, our SEENN-I largely reduces the distance between the AET coordinate and the accuracy -  $T$  curve, meaning that our method is approaching the upper limit of the early exit. On the right side of the Fig. 4, we additionally draw the composition pie charts of SEENN-I, which shows how many percentages of inputs are using 1, 2, 3, or 4 timesteps, respectively. It can be shown that, as we gradually adjust  $\alpha$  from 0.4 to 0.9 for the SEW-ResNet-34, the percentage of inputs using 1 timestep decreases (73.4% to 30.7%). For the CIFAR10 dataset, we find images have a higher priority in using the first timestep, ranging from 95.4% to 56.2%.
Figure 4: Left: Accuracy vs. the number of timesteps curve. Right: Pie charts indicating the composition of input images using different numbers of timesteps for inference.
Figure 5: Qualitative assessment using input images from ImageNet dataset. We select images from orange, cucumber, bubble, broccoli, aircraft carrier, and torch classes and separate them according to SEENN.
4.5 Qualitative Assessment
In this section, we conduct a qualitative assessment of SEENN by visualizing the input images that are separated by our SEENN-II, or the policy network. Specifically, we take the policy network and let it output the number of timesteps for each image in the ImageNet validation dataset. In principle, the policy network can differentiate whether the image is easy or hard so that easy images can be inferred with less number of timesteps and hard images can be inferred with more number of timesteps. Fig. 5 provides some examples of this experiment, where images are chosen from orange, cucumber, bubble, broccoli, aircraft carrier, and torch classes in the ImageNet validation dataset. We can find that  $T = 1$  (easy) images and  $T = 4$  (hard) images have huge visual discrepancies. As an example, the orange, cucumber, and broccoli images from  $T = 1$  row are indeed easier to be identified, as they contain single objects in a clean background. However, in the case of  $T = 4$ , there are many irrelevant objects overlapped with the target object, or there could be many small samples of target objects which makes it harder to identify. For instance, in the cucumber case, there are other vegetables that increase the difficulty of identifying them as cucumbers. These results confirm our hypothesis that visually simpler images are indeed easier and can be correctly predicted using a fewer number of timesteps.
In this paper, we introduce SEENN, a novel attempt to allow a varying number of timesteps on an input-dependent basis. Our SEENN includes both a post-training approach (confidence score thresholding) and an early-exit-aware training approach (reinforcement learning for selecting the appropriate number of timesteps). Our experimental results show that SEENN is able to find a sweet spot that maintains accuracy while improving efficiency. Moreover, we show that the number of timesteps selected by SEENNs is related to the visual difficulty of the image. By taking an input-by-input approach during inference, SEENN is able to achieve state-of-the-art accuracy with less computational resources.
This work was supported in part by CoCoSys, a JUMP2.0 center sponsored by DARPA and SRC, the National Science Foundation (CAREER Award, Grant #2312366, Grant #2318152), TII (Abu Dhabi), and the DoE MMICC center SEA-CROGS (Award #DE-SC0023198).
[1] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-444, 2015.
[2] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
[3] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607-617, 2019.
[4] Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothee Masquelier, and Anthony Maida. Deep learning in spiking neural networks. *Neural Networks*, 111:47–63, 2019.
[5] Yufei Guo, Xuhui Huang, and Zhe Ma. Direct learning-based deep spiking neural networks: a review. Frontiers in Neuroscience, 17:1209795, 2023.
[6] Laxmi R Iyer and Yansong Chua. Classifying neuromorphic datasets with tempotron and spike timing dependent plasticity. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2020.
[7] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
[8] Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems, pages 1412-1421, 2018.
[9] Peter U. Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, and Shih Chii Liu. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In Neural Networks (IJCNN), 2015 International Joint Conference on, 2015.
[10] Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion of analog to spiking convolutional neural networks. arXiv: Statistics/Machine Learning, (1612.04052):0-0, 2016.
[11] Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in Neuroence, 13, 2018.
[12] Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In European Conference on Computer Vision, 2020.
[13] Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. arXiv preprint arXiv:2103.00476, 2021.
[14] Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In International Conference on Machine Learning, pages 6316-6325. PMLR, 2021.
[52] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 113–123, 2019.
[53] Yufei Guo, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Yinglei Wang, Xuhui Huang, and Zhe Ma. Im-loss: information maximization loss for spiking neural networks. Advances in Neural Information Processing Systems, 35:156-166, 2022.
[54] Sayeed Shafayet Chowdhury, Nitin Rathi, and Kaushik Roy. Towards ultra low latency spiking neural networks for vision and sequential tasks using temporal pruning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XI, pages 709-726. Springer, 2022.
[55] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Sun Jian. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[56] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[57] Gourav Datta, Zeyu Liu, and Peter A Beerel. Hoyer regularizer is all you need for ultra low-latency spiking neural networks. arXiv preprint arXiv:2212.10170, 2022.
[58] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34:21056-21069, 2021.
A Verification of Assumption 3.1
In Assumption 3.1, we conjecture that if a correct prediction is made by  $f_{t}(\mathbf{x})$ , then for all  $t' \geq t$  that  $f_{t'}(\mathbf{x})$  is also correct. And based on this assumption, we propose the equation to compute AET, i.e. Eq. (4). Here, to validate if our assumption holds, we propose another metric: empirical AET, given by
$$
\widetilde {\mathrm {A E T}} = \frac {1}{N} \left(\sum_ {i = 1} ^ {N} \tilde {t} _ {i}\right), \tag {13}
$$
where  $\tilde{t}_i$  is the actual earliest number of timesteps that predicts the correct class for the  $i$ -th sample. Note that, similar to AET, if the network cannot make correct predictions in any timesteps, then we set  $\tilde{t}_i$  to the maximum number of timesteps.
Consider an example that does not satisfy Assumption 3.1, for example, the prediction results for the first 4 timesteps are {False, True, False, True}. The empirical AET will obtain 2 while the AET will obtain  $2 - 3 + 4 = 3$ . Therefore, by comparing the difference between the AET and the empirical AET we can verify if Assumption 3.1 holds. Table 5, as shown below, demonstrates the AET and the empirical AET comparison across different models and datasets. We can find that difference is very small, often less than 0.05. The only dataset that creates a slightly larger difference is CIFAR10-DVS. Indeed, the event-stream dataset may provide more variations over time. Nevertheless, the potential for early exit is still high.
Table 5: Comparison between ATE and empirical AET across models and datasets.
Model	Dataset	maxT	AET	AET
Direct Training of SNNs
ResNet-19	CIFAR-10	4	1.1309	1.1188
ResNet-19	CIFAR-100	4	1.6075	1.5616
ResNet-34	ImageNet	6	2.9026	2.7781
SEW-ResNet-34	ImageNet	4	2.0646	2.0092
VGGSNN	CIFAR10-DVS	10	3.096	2.774
B Distribution of Confidence Scores
In Fig. 2, we demonstrate the distribution of confidence scores of ResNet-19 on the CIFAR-10 validation dataset. Obviously, with a higher number of timesteps more input data have higher confidence scores. This aligns with our target to filter out easy input data.
Figure 6: The distribution of confidence scores of ResNet-19 on CIFAR-100, left: [0, 0.9999], right: [0.9999, 1]
C Hardware Efficiency on ImageNet
We demonstrate the results in Fig. 7.
Figure 7: Hardware efficiency on ImageNet dataset.
D Hardware Evaluation
For latency measurement, we directly tested on GPU with the Pytorch framework. The latency measurement is computed across the whole test dataset. For example, the throughput calculation is given by
$$
\text {T h r o u g h p u t} = \frac {\text {T e s t s e t i n f e r e n c e t i m e}}{\text {N u m b e r o f t e s t s a m p l e s}}. \tag {14}
$$
For energy estimation, we adopt the conventional way to measure the addition and multiplication operations for the entire SNN inference. The addition costs  $0.9pJ$  and the multiplication costs  $4.6pJ$ , respectively.
E Architecture Details
The major network architecture we adopt in this work is ResNet-series architecture. There are many variants of ResNets used in the field of SNNs, e.g. ResNet-18, ResNet-19, and ResNet-20. They are also mixedly referred to in existing literature [18, 21, 30, 55]. Therefore, to avoid confusion in these ResNet architectures, in this section, we sort out the details of the network configurations.
Table 6 summarizes the different ResNets we used in our CIFAR experiments. For ImageNet models, we use the standard ResNet-34 [55] and SEW-ResNet-34 [58] which are well-defined. The basic difference between the four ResNets is the channel configurations and the block configurations. Therefore, their actual FLOPs difference is a lot larger than the difference suggested by their names. For example, our policy network (ResNet-8) only contains  $0.547\%$  number of operations of ResNet-19. Therefore, the cost of running SEENN-II is almost negligible.
Table 6: The architecture details of ResNets. * denotes that the first residual block contains downsample layer to reduce the feature resolution.
ResNet-8	ResNet-18
conv1	3 × 3, 16, s1	3 × 3, 64, s1
stage1	(3 × 3, 16) × 1	(3 × 3, 64) × 2
(3 × 3, 16) × 2
stage2	(3 × 3, 32) * × 1	(3 × 3, 128) * × 2
(3 × 3, 128) × 2
stage3	(3 × 3, 64) * × 1	(3 × 3, 256) * × 2
(3 × 3, 256) × 2
stage4	N/A	(3 × 3, 512) * × 2
(3 × 3, 512) × 2
pooling	Global average pooling
classifier	(64, O) FC	(512, O) FC
ResNet-19	ResNet-20
conv1	3 × 3, 128, s1	3 × 3, 16, s1
stage1	(3 × 3, 128) × 3	(3 × 3, 16) × 2
(3 × 3, 16) × 3
stage2	(3 × 3, 256) * × 3	(3 × 3, 32) * × 2
(3 × 3, 32) × 3
stage3	(3 × 3, 512) * × 2	(3 × 3, 64) * × 2
(3 × 3, 512) × 2
pooling	Global average pooling
classifier	(512, O) FC	(64, O) FC