SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To verify the learning ability for long-term dependencies of different spiking neurons, we evaluated their performance on classifying sequential CIFAR10 and CIFAR100. In these tasks, the image will be sent to the SNN column by column, which is similar to how humans read from left to right. The sequential image classification task is widely used to evaluate the learning ability of a network for long-term dependencies. The network structure is modified from [15] by replacing 2D convolutional/max pooling layers with 1D convolutional and average pooling layers, and removing
Neuron
Dataset	PSN	masked PSN	sliding PSN	GLIF[16]	KLIF[18]	PLIF[15]	LIF	LIF wo reset
Sequential CIFAR10	88.45	85.81	86.70	83.66	83.26	83.49	81.50	79.50
Sequential CIFAR100	62.21	60.69	62.11	58.92	57.37	57.55	55.45	53.33
Table 1: The test accuracy (%) of different spiking neurons on sequential CIFAR.
(a) Sequential CIFAR10
(b) Sequential CIFAR100
Figure 4: The accuracy curves and firing rates of different neurons for classifying sequential CIFAR.
the voting layers. Due to the column-by-column input,  $T$  is 32, which is the width of the image. The compared neurons include PSN, 32-order masked PSN, 32-order sliding PSN, GLIF with channelwise parameters, KLIF, PLIF, and the LIF neuron w/wo reset. We use the optimal reset options determined by ablation experiments for different neurons, which are using soft reset and detach reset [53] for the PLIF and LIF neurons, and using soft reset for the KLIF neuron. The GLIF neuron uses learnable gates to control all options, which do not need to be set manually. For the masked PSN,  $\lambda = \min(1, 8 \cdot \text{epoch} / (\text{epochs} - 1))$ , where epoch denotes the current training epoch, and epochs denotes the total number of epochs.
Tab.1 shows the accuracy of all neurons on sequential CIFAR. It can be found that the rank of accuracy is  $\mathrm{PSN} >$  sliding  $\mathrm{PSN} >$  masked  $\mathrm{PSN} >$  GLIF  $>$  PLIF  $>$  LIF  $>$  LIF w reset, which is as expected. The PSN exploits information from all time-steps and has the highest learning ability, but it may be criticized for unfair comparison because it can read all columns at once. Therefore, the accuracy of the masked PSN and the sliding PSN is more convincing because they can still run in a step-by-step mode. Due to the introduction of high-order information, they work better than any other traditional spiking neurons. Meanwhile, the GLIF neuron is the best among traditional spiking neurons, indicating that fusing different kinds of charging, decaying, and resetting increases learning ability. The LIF neuron without reset has lower accuracy than the LIF neuron, indicating that the direct removal of reset drops the learning ability of the vanilla spiking neurons. Thus, using high-order information as the PSN family is necessary to fix this decline.
To make clear comparisons, we plotted the accuracy curves during training and firing rates of each spiking neuron layer of trained SNNs with different spiking neurons, as shown in Fig.4. These accuracy curves are the moving averages of 8 epochs, and the light-colored curves show the original accuracy. The curves of the PSN family are almost always above other curves, which clearly show their superior performance on convergence speed and final accuracy. Note that there is a sharp drop
Dataset	Method	Spiking Network	Time-steps	Accuracy(%)
CIFAR10	Dspike[38]	Modified ResNet-18	6	94.25
TET[39]	ResNet-19	6	94.50
TDBN[30]	ResNet-19	6	93.16
TEBN[33]	ResNet-19	6	94.71
PLIF[15]	PLIF Net	8	93.50
KLIF[18]	Modified PLIF Net	10	92.52
GLIF[16]	ResNet-19	6	95.03
PSN	Modified PLIF Net	4	95.32
ImageNet	Dspike[38]	ResNet-34	6	68.19
VGG-16	5	71.24
TET[39]	SEW ResNet-34	4	68.00
TDBN[30]	ResNet-34 with double channels	6	67.05
TEBN[33]	SEW ResNet-34	4	68.28
GLIF[16]	ResNet-34	4	67.52
SEW ResNet[40]	SEW ResNet-18	4	63.18
SEW ResNet-34	4	67.04
PSN	SEW ResNet-18	4	67.63
SEW ResNet-34	4	70.54
CIFAR10-DVS	Dspike[38]	ResNet-18	10	75.40
TET[39]	VGG	10	83.17
TDBN[30]	ResNet-19	10	67.80
TEBN[33]	VGG	10	84.90
PLIF[15]	PLIF Net	20	74.80
KLIF[18]	Modified PLIF Net	15	70.90
GLIF[16]	Wide 7B Net	16	78.10
SEW ResNet[40]	Wide 7B Net	16	74.40
sliding PSN (k=2)	VGG	4,8,10	82.30, 85.30, 85.90
Table 2: Comparison of the PSN family and other methods.
in the curves of the masked PSN around epoch 32, which is caused by the mask becoming completely binary. The firing rates indicate that uninterrupted firing does not occur with the removal of neuronal resetting. In general, the PSN and the masked PSN are more easily to be activated and show higher firing rates than others. Meanwhile, the firing rates of the sliding PSN and the LIF neuron without reset are slightly higher than the vanilla spiking neurons. Thus, we conclude that removal of reset does increase firing rates, but the increment degree depends on the neuron's structure. On the other hand, considering the huge improvement in simulation efficiency and long-term dependencies learning ability, the slightly higher firing rate is acceptable.
Fig.5 shows the accuracy-order curve on the sequential CIFAR100 dataset for the masked PSN and sliding PSN, with the highest accuracy marked by a red  $\star$ . It can be seen that when the order increases from 1 to 4, the accuracy improves quickly. When we continue to increase the order, the improvement is marginal and the accuracy even drops. The highest accuracy is obtained by a large, but not the largest order, which is 20 and 31 for the masked PSN and sliding PSN, respectively. But in most cases, high accuracy can be guaranteed with  $k = T$ . Remarkably, the curve of sliding PSN is almost always above the curve of masked PSN, and excels that of the PSN when  $k = 21$  and  $k = 31$ , indicating that the time-invariant parameters have better generalization ability to process temporal information.
Figure 5: The accuracy-order curve on the sequential CIFAR100.
4.3 Static and Neuromorphic Data Classification
Static and neuromorphic data classification tasks are also common benchmarks for SNNs. We evaluated the PSN family on the static CIFAR10, ImageNet datasets, and the neuromorphic CIFAR10-DVS [54] dataset. The number of time-steps for static datasets is small, and the latency is not the main issue. Thus, we use the PSN in CIFAR10 and ImageNet. According to the results on sequential CIFAR, the sliding PSN uses fewer parameters but gets higher accuracy, and we will use it for CIFAR10-DVS. The results are listed in Tab.2.
CIFAR10 We modified the network structure of [15], or the PLIF Net, by using average pooling and removing voting layers. Our PSN uses the shortest time-steps  $T = 4$  and achieves the highest accuracy of  $95.32\%$ .
ImageNet We used the SEW ResNet [40] for classifying the ImageNet dataset. We verified the performance of the SPN on SEw ResNet-18/34 and got a stable  $3 + \%$  accuracy improvement over the original SEW ResNet with the IF neuron. Our method achieves  $70.54\%$  accuracy and is only second to the Dspike method using one more time-step and VGG-16, which has a number of parameters 6.3 times as many as ours.
CIFAR10-DVS We used the VGG structure from [39]. Considering the fact that the DVS data contains temporal features and  $T$  is larger than the static datasets, we use the 2-order sliding PSN. We achieve  $82.3\%$  accuracy with  $T = 4$ , which is the first work to get  $80 + \%$  accuracy in such few time-steps. When increasing time-steps to  $T = 8$ , we achieve  $85.30\%$  accuracy, which is  $2 + \%$  higher than that of the previous SOTA method [33] and we use two fewer time-steps. With  $T = 10$ , the accuracy is further improved to  $85.90\%$ .
In this paper, we remove reset from neuronal dynamics of vanilla spiking neurons and propose the PSN family, including the PSN, the masked PSN, and the sliding PSN. The PSN family can be simulated in parallel, which greatly accelerates the training of deep SNNs. Weights of inputs in the PSN family can be fully connected, or masked/shared with a custom order. Experimental results of simulation speed and temporal/static data classification verify the efficiency and accuracy of the PSN family when compared with previous spiking neurons. Our work may be a milestone as well as a cornerstone for modern spiking neurons.
This work is supported by grants from the National Natural Science Foundation of China (62027804, 61825101, 62088102, and 62176003), the major key project of the Peng Cheng Laboratory (PCL2021A13), and the Agence Nationale de la Recherche under Grant ANR-20-CE23-0004-04 DeepSee.
[1] Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural Networks, 10(9):1659-1671, 1997.
[2] Paul A Merolla, John V Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science, 345(6197):668-673, 2014.
[3] Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, Yuyun Liao, Chit-Kwan Lin, Andrew Lines, Ruokun Liu, Deepak Mathaikutty, Steven McCoy, Arnab Paul, Jonathan Tse, Guruguhanathan Venkataramanan, Yi-Hsin Weng, Andreas Wild, Yoonseok Yang, and Hong Wang. Loihi: A neuromorphic mancore processor with on-chip learning. IEEE Micro, 38(1):82-99, 2018.
[4] Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):106-111, 2019.
[5] Chris Eliasmith, Terrence C. Stewart, Xuan Choo, Trevor Bekolay, Travis DeWolf, Yichuan Tang, and Daniel Rasmussen. A large-scale model of the functioning brain. Science, 338(6111):1202-1205, 2012.
[6] Marcel Stimberg, Romain Brette, and Dan FM Goodman. Brian 2, an intuitive and efficient neural simulator. eLife, 8:e47314, 2019.
[56] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976-11986, 2022.
[57] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
[58] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
[59] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6023-6032, 2019.
[60] Samuel G Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 774-782, 2021.
[61] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence (AAAI), volume 34, pages 13001-13008, 2020.
[62] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pages 2818-2826, 2016.
[63] Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Training high-performance low-latency spiking neural networks by differentiation on spike representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12444–12453, 2022.
[64] Nitin Rathi and Kaushik Roy. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[65] Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations (ICLR), 2020.