SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Combined with the fusion method in 4.3.1, the proportion of excitatory mechanism and inhibitory mechanisms in the biological neural network is adjusted at the same time. We propose an idea: build a network model, set some layers as inhibitory mechanisms and some layers as excitatory mechanisms, and train the two mechanisms in this network model. At the same time, the ratio of excitatory and inhibitory mechanisms in this network model is also 4:1[18], that is, the excitatory mechanism accounts for about  $80\%$ , and the inhibitory mechanism accounts for about  $20\%$ . The method of the inhibitory mechanism we choose in this paper is the STDP method, and the STDP method has some limitations: it can only perform temporal logic training on the convolutional layer and the linear layer. Since the STDP method is used to train the last linear layer of the network structure separately, the structure of the network model ensures that the ratio of excitatory and inhibitory mechanisms reaches an ideal ratio is small. So it is not necessarily possible to train to produce the best results. We use the STDP method to train the convolutional layer and combine the framework of the Resnet network model to build our own network model. The model consists of basic convolutional layers, BN layers, pooling layers, residual blocks, and linear layers. Each residual block contains two sets of convolutional layers, maximum pooling layers, BN layers, and spike neurons. We train the convolutional layers with an inhibitory mechanism, and the linear, pooling, and BN layers with an excitatory mechanism. Finally, the constructed model reached  $77\%$  of the excitatory mechanism and  $23\%$  of the inhibitory mechanism, which is close to the ratio of excitation and inhibition in the human neural network.
Figure 1: The figure above presents the framework of the excitatory inhibition network. On the left side, we can observe the residual blocks of the network, while on the right side, the overall network architecture is depicted. The design of the network framework ensures a ratio of 8:2 between excitatory and inhibitory mechanisms.
In this section, we demonstrate the effectiveness of our proposed EICIL model. We first compare our results with other existing state-of-the-art, then carry out ablation studies to evaluate different aspects of our proposed method.
We conduct experiments on a server equipped with a 16-core Intel(R) Xeon(R) Xeon(R) Gold 6330 2.80GHz CPU and 20 NVidia GeForce RTX 3090 Ti GPUs. The training of SNN is based on the spikingjelly framework [9]. Our experiments mainly use the three network frameworks of Resnet, VGG, and the network we built, and conduct experiments on three image data sets of MNIST[5], CIFAR10 [19] and CIFAR100[20]. In this experiment, we adopt different training methods for the same experimental framework to better verify the training advantages of iterative learning.
MNIST.MNIST is a picture dataset of handwritten digits. It counts pictures of handwritten digits from 250 different people. The dataset is relatively simple and can initially demonstrate the performance of our model. When training, we use Resnet and VGG11 for training to get the results of different training methods.
CIFAR 10 & CIFAR 100. The CIFAR dataset was collected from 80 million RGB miniature image datasets. The CIFAR dataset can be divided into CIFAR-10 and CIFAR-100 according to the number of classified objects involved. The dataset is relatively complex and can more realistically verify the performance of our proposed framework. When training, we need to first encode the static images into spike sequences, and then input them into the SNN, here the first spike neuron layer in the network is used as the encoding layer.
5.2 Evaluation on STDP-BW Method
Experimental results have demonstrated that the incorporation of backpropagation into the STDP model (STDP-BW) achieves an accuracy of 90.39 percent the CIFAR10 dataset, which is a 1percent
Table 1: Test accuracies of EICIL on CIFAR 10, CIFAR 100, and MNIST.
SNN Model	Database	GS (Acc. %)	STDP-BW (Acc. %)	STDP-BW-GS (Acc. %)	GSI (Acc. %)	GSGI (Acc. %)	Improvement (Acc. %)
Spiking Resnet 18[17]	CIFAR10	89.39	90.39	89.16	90.05	90.34	0.95
CIFAR100	56.28	56.20	57.37	63.03	63.47	7.19
MNIST	99.17	99.35	99.26	99.36	99.44	0.27
Spiking VGG 11[22]	CIFAR10	83.96	84.06	83.96	84.11	85.31	1.35
CIFAR100	54.01	54.63	52.44	55.44	55.71	1.70
MNIST	98.2	99.09	95.85	98.55	98.77	0.55
Excitatory inhibition Net	CIFAR10	88.88	89.14	89.64	89.23	89.43	0.55
CIFAR100	53.61	53.92	54.03	54.61	53.86	0.25
MNIST	99.33	99.34	99.30	99.33	99.35	0.02
improvement compared to the GS method. Comparable accuracy was observed on the CIFAR100 and MNIST datasets when compared to the GS method. Furthermore, a comparison with other network models such as Spiking Resnet18, Spiking VGG11, and Excitatory Inhibition Net reveals that the Spiking Resnet18 model outperforms the other two in terms of performance.
5.3 Evaluation on Iteration-based Methods
In the experiment, we continue to use Spiking Resnet18, Spiking VGG11 and the spiking form of our proposed Excitatory inhibitor Net to test the two training methods of GSI and GSGI, and then analyze the model.
For the MNIST data set, as shown in Table 1, when using Spiking Resnet18, GSI is improved by  $0.19\%$  compared to the original GS method, GSGI is improved by  $0.28\%$ , when using Spiking VGG 11, GSI is compared to the original The GS method has been improved by  $0.35\%$ , and the GSGI has been improved by  $0.57\%$ , which shows that the iterative method can improve the expression ability of the enhanced neuron model. To better compare the effectiveness of using an iterative training approach, we visualize a line graph of test accuracy.
For the CIFAR10 data set, when Table 1 uses Spiking Resnet18, GSI improves by  $0.66\%$  compared to the original GS method, and GSGI further improves by  $0.95\%$ , while the model performance of VGG11 is similar. The improvement is  $0.15\%$  and  $1.35\%$ , respectively, compared with the MNIST dataset, the algorithm based on iteration performs better on larger datasets.
For the CIFAR100 dataset, as shown in Table 1, when using Spiking Resnet18, GSI has improved by  $6.75\%$  compared to the original GS method, and GSGI has further improved by  $7.19\%$ , but due to the model structure and complexity The reason for this is that the performance of the VGG11 model is relatively poor, with only an increase of  $1.43\%$  and  $1.70\%$  respectively. The algorithm based on iteration performs better on larger data sets, which to a certain extent shows that the method based on iteration can improve the network's Biological interpretability, which can more effectively use the excitatory and inhibitory information of neurons for collaborative training.
For our Excitatory inhibition Network, as shown in Table 1, the performance of the GSI and GSGI iterative models is slightly better than that of Spiking VGG but slightly lower than that of Spiking Resnet 18. This shows that the iterative method has a stronger matching ability with the Resnet network structure and can greatly improve its expression. Ability, and model accuracy in the MNIST, CIFAR10, CIFAR100 data sets all show a positive improvement effect.
At the same time, comparing the GSGI and GSI results of the three data sets in the three networks, the accuracy of GSGI is higher than that of GSI, which also proves that the STDP-BW-GS method is slightly better than the STDP-BW method. It shows that the combination of excitatory mechanism and inhibitory mechanism proposed by us will lead to the improvement of training accuracy.
(a) Spiking Resnet18 in CIFAR 10
(b) Spiking Resnet18 in CIFAR 100
(c) Excitatory inhibition Net in CIFAR 10
(d) Excitatory inhibition Net in CIFAR 100
Figure 2: Performance of test accuracy of Spiking Resnet 18 and Excitatory inhibition Net in CIFAR 10 and CIFAR 100
Table 2: Test accuracies of EICIL on CIFAR 10, CIFAR 100, and MNIST.
SNN Model	Parameter k	Learning Method	MNIST (Acc.%)	CIFAR 10 Acc. (%)	CIFAR 100 Acc. (%)
Spiking Resnet 18[17]	3	GSI	99.39	89.97	57.74
GSGI	99.42	90.26	62.24
4	GSI	99.33	90.16	62.59
GSGI	99.43	90.21	62.64
5	GSI	99.30	90.46	56.15
GSGI	99.34	90.58	66.36
-3	GSI	99.36	89.97	57.11
GSGI	99.37	89.56	64.28
-4	GSI	99.34	90.38	58.53
GSGI	99.36	90.55	63.16
-5	GSI	99.32	90.34	56.62
GSGI	99.34	90.01	62.86
The effect of the hyperparameter  $\mathbf{k}$  on the expressiveness of the model.
Considering that different  $k$  values have different effects on the model, we conducted ablation experiments and tested the iterative training models GSI and GSGI based on Spiking Resnet 18. The experimental results are shown in Table.2, in the MNIST dataset On the above, the accuracy difference between GSI and GSGI ranges from  $0.03\%$  to  $0.10\%$ . On the CIFAR10 data set, the accuracy difference between GSI and GSGI ranges from  $-0.33\%$  to  $0.29\%$ . On the CIFAR100 data On the set, the accuracy difference between GSI and GSGI ranges from  $0.05\%$  to  $8.62\%$ . We found that, limited by the size of the data set, the accuracy improvement on the MNIST and CIFAR10 data sets is small, while under the CIFAR100 large data set, the expressive ability of the model is significantly improved. At the same time, we found that the larger the K value, the higher the accuracy rate and the better the excitation-inhibition trade-off ability. When the K value reaches 5, the ideal 8:2 ratio of excitation and inhibition of the visual nervous system in the biological brain has been reached, which happens to be the highest accuracy rate of GSI and GSGI at this time. Therefore, it is also proved that our model method is more biologically interpretable while improving accuracy.
Inspired by the idea of neuron excitation-inhibition, this paper proposes a joint learning method based on excitation and inhibition mechanisms. The main idea is to better simulate the functional principles of the nervous system by integrating excitation and inhibition mechanisms in a cohesive manner. Thereby enhancing the biological rationality of SNN and improving the expressive ability and accuracy of spiking neural networks.
Considering the limited expressive power of the model structure, the experimental results show that we can extend the work to more situations and improve the accuracy of the model under specific tasks through iterative training. It also shows that the proposed model and training method exhibit good performance on both small and relatively large datasets.
In our future work, we will extend the joint learning method to exploit the biological properties of excitability and inhibition, continue to explore how the two are combined and paired, and evaluate them on more datasets.
This work was supported in part by National Natural Science Foundation of China (NSFC No.62206037,62306274), National Key Research and Development Program of China (2021ZD0109803), the China Postdoctoral Science Foundation under Grant No. 2023M733067 and No. 2023T160567, and the Fundamental Research Funds for the Central Universities (DUT21RC(3)091)
[1] Henry DI Abarbanel, Nikolai F Rulkov, and Mikhail M Sushchik. Generalized synchronization of chaos: The auxiliary system approach. Physical review E, 53(5):4528, 1996.
[2] Yunus Babacan, Abdullah Yesil, Omer Faruk Tozlu, and First Kacar. Investigation of stdp mechanisms for memristor circuits. AEU-International Journal of Electronics and Communications, 151:154230, 2022.
[3] Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-learn in networks of spiking neurons. Advances in neural information processing systems, 31, 2018.
[4] Guo-qiang Bi and Mu-ming Poo. Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. Journal of neuroscience, 18(24):10464-10472, 1998.
[5] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):141-142, 2012.
[6] Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. arXiv preprint arXiv:2103.00476, 2021.
[7] Jianhao Ding, Tong Bu, Zhaofei Yu, Tiejun Huang, and Jian Liu. Snn-rat: Robustness-enhanced spiking neural network through regularized adversarial training. Advances in Neural Information Processing Systems, 35:24780-24793, 2022.
[8] Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. Advances in Neural Information Processing Systems, 35:34377-34390, 2022.
[9] Wei Fang, Yanqi Chen, Jianhao Ding, Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, 9(40):eadi1480, 2023.
[10] Johannes Friedrich, Robert Urbanczik, and Walter Senn. Learning spike-based population codes by reward and population feedback. *Neural computation*, 22(7):1698–1717, 2010.
[11] Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
[12] Wenzhe Guo, Mohammed E Fouda, Ahmed M Eltawil, and Khaled Nabil Salama. Neural coding in spiking neural networks: A comparative study for robust neuromorphic systems. Frontiers in Neuroscience, 15:638474, 2021.
Intelligence, volume 35, pages 11062-11070, 2021.