SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To validate the effect of the SCBAM on the selection of the key features in the spatial and channel dimensions, we observe the feature map visualization of the RSNN-SCBAM model on the DVS128 GESTURE and CIFAR10-DVS datasets under the situation with and without SCBAM component.
DVS128 GESTURE feature map visualization results are illustrated in Fig. 4, the RSNN module with SCBAM captures more complementary and sparse features than the one without SCBAM. On one hand, the SCBAM possesses fine sparsity in the channel and spatial dimensions through the whole time steps. That benefits from the attention mechanism that captures significant features and eliminates redundant features. On the other hand, most of the features extracted by SCBAM are complementary to each other. Each feature map focuses on one specific area such as the hands and the shoulders. However, some feature maps of the RSNNs without SCBAM are quite similar because of the lack of filtering of redundant information. Based on the proposed SRNN-CBAM, the historical redundancy left in other spiking recurrent models could be discarded sharply, meanwhile, the proposed method maintains good performance by adaptive history reserving.
CIFAR10-DVS feature map visualization results are illustrated in Fig. 5, we visualize the extracted features at several time steps of the proposed RSNNs with and without SCBAM. It shows that the RSNN module with SCBAM captures information as anticipated, exhibiting strong sparsity across both temporal and spatial dimensions throughout the entire time steps while the RSNN module without SCBAM only extracts limited features which may lose a significant amount of crucial information.
We further conducted ablation experiments on dynamic thresholding during training on the DVSGesture dataset, and our RSNN-SCBAM model adopted two different dynamic thresholding methods in LIFNode and MultiStepLIFNode SNN. We set the thresholds of MultiStepLIFNode and LIFNode as trainable parameters and conduct experiments to verify their impact. The experimental results are shown in Table. 3 shown.
Table 3: Accuracy of two different dynamic threshold methods for the DVS128 GESTURE dataset.
Method	Node_threshold	Accuracy (%)
RSNN-SCBAM	LIFNode	91.3
RSNN-SCBAM	MultiStepLIFNode	89.9
Table 4: Accuracy of additional time step values of 10, 15, and 25 on the DVSGesture dataset.
Method	Time step	Accuracy (%)
RSNN-SCBAM	10	89.2
RSNN-SCBAM	15	92.4
RSNN-SCBAM	25	90.3
However, the current results do not show any significant improvement in dynamic thresholds. This threshold adjustment may make the model relatively complex, involving more parameters and
Table 5: Accuracy of solutions for the CIFAR10-DVS dataset (10 classes).
Method	Accuracy (%)
HOTS[29]	27.1
Lightweight Statistical[30]	31.2
Attention Mechanisms[31]	44.1
HATS[32]	52.4
LIF-NET[33]	63.5
LIAF-NET[33]	70.4
Proposed CBAM-SNN	66.3
nonlinear operations. If the parameters or network structure of the model are not adjusted correctly, poor model performance may result. Furthermore, SNNs may require large amounts of training data and longer training times to achieve effective learning and optimization.
Regarding the variation in LSTM time steps, we conduct experiments using additional time step values of 10, 15, and 25 on the DVSGesture dataset. As shown in Table. 4, the results show that the time step of 20 is the optimal choice. This could be attributed to the nature of the DVSGesture data, where gestures are repeated by individuals at specific time intervals. The accuracies of the RSNN-CSAM increase first and then reduce as the growth of the time steps from 10 to 25. Different time steps capture varying temporal features, leading to differences in test results. Fewer time steps could cause the excessive concentration of information and then could not extract enough event features, while the larger time steps may extract superfluous information and could not capture the key information.
4.4 Comparison with other methods.
We conduct a performance comparison on the CIFAR10-DVS dataset to explore the efficiency of the proposed model. The compared models are as follows: The HOTS method proposed in [29] employs spatio-temporal features called temporal surfaces to create an event-based hierarchical pattern recognition architecture, we adopt three layers of time surface prototypes in the experiment. The article [30] represents the visual information captured by DVS as a stream of asynchronous pixel addresses (events) representing relative intensity changes at those locations, and uses a random ferns classifier to classify the stream of pixel events. The network structure consists of the following three modules: a motion detector, a bank of binary feature extractors, and a Random Ferns classifier. The number of ferns for CIFAR10-DVS is (50, 14), and there are 15,000 segment events with a patch size range between 12 and 20. The paper [31] focuses on visual attention, a specific feature of vision, and proposes two attention models for event-based vision. We adopt the p-N-DRAW recognition network, using pLSTM layers as encoders. The paper [32] introduces a novel event-based feature representation and a new machine learning architecture. And use the local memory unit to efficiently utilize past-time information and build a robust event-based representation. In the experiment, a temporal surface representation method with local memory was used to divide all event points within a spatial region into different fixed cells. Within each cell, all event points are accumulated to generate a temporal surface, obtaining features. Classification is achieved using an SVM (Support Vector Machine). The paper [33] proposes a neuron normalization technique to tune neural selectivity and develops a direct learning algorithm for deep SNNs. The network architecture employed in [33] is as follows: LIAF Blocks and Dense Blocks connected end-to-end. LIAF Block has a sequential structure, consisting of ConvLIAF, TD-layer-normalization, TD-activation (ReLU) and TD-AvgPooling, where TD refers to the time-distributed operation.
A detailed comparison of SNN based models for the benchmark is shown in Table 5, the proposed SRNN-CBAM achieves competitive performance among those models. Our proposed SRNN-CBAM method performs better than the LIF-NET model in [33]. Although the classification accuracy of SRNN-CBAM is lower than LIAF-Net [33], the adaptive memory invoking should be emphasized for the further application of spatiotemporal pattern processing.
Different from some classical spiking based models which aim at static pattern recognition, this paper utilized the dynamic spatio-temporal information processing ability of SNNs on event data. This paper proposes a brand new spiking recurrent neural network model with spiking convolutional block attention mechanism (SRNN-SCBAM), which can invoke the temporal history information in spatial and temporal features adaptively. It has prior advantages in efficient memory calling and irrelevant history information elimination. The experiment results show that the proposed model can achieve competitive performance among other state-of-the-art models, and the attention component captures meaningful features according to the sensitivity of the output to the input in the end-to-end training process.
At the same time, we believe that the proposed Spiking ConvLSTM is good at different time-series data and applications, such as object tracking, and it is worth noting that the proposed Spiking ConvLSTM needs the spike-based input, hence, when facing other time-series data, the input data should firstly be encoded into spike trains.
In addition, it is worth noting that we take a further step in processing efficient memory invoking and preserving of spiking neural networks, which has promising potential in processing and memorizing spatio-temporal patterns. This paper rethinks the role that attention plays in spiking recurrent neural networks, which not only provides a new way of constructing an efficient model for dynamic event stream data processing but more importantly that the proposed model could mimic the mechanism of potential information processing in neural circuits. Remember what should be remembered, and forget what should be forgotten.
This work was supported in part by National Key Research and Development Program of China (2021ZD0109803), National Natural Science Foundation of China (NSFC No.62206037 and No. 62306274), the China Postdoctoral Science Foundation under Grant No. 2023M733067 and No. 2023T160567, and the Fundamental Research Funds for the Central Universities (DUT21RC(3)091)
[1] R. Dolmetsch and D. H. Geschwind, "The human brain in a dish: the promise of ipsc-derived neurons," Cell, vol. 145, no. 6, pp. 831-834, 2011.
[2] V. Kukekov, E. Laywell, O. Suslov, K. Davies, B. Scheffler, L. Thomas, T. O'brien, M. Kusakabe, and D. Steindler, "Multipotent stem/progenitor cells with similar properties arise from two neurogenic regions of adult human brain," Experimental neurology, vol. 156, no. 2, pp. 333-344, 1999.
[3] K. Roy, A. Jaiswal, and P. Panda, "Towards spike-based machine intelligence with neuromorphic computing," Nature, vol. 575, no. 7784, pp. 607-617, 2019.
[4] A. Shaban, S. S. Bezugam, and M. Suri, "An adaptive threshold neuron for recurrent spiking neural networks with nanodevice hardware implementation," Nature Communications, vol. 12, no. 1, pp. 1-11, 2021.
[5] W. Zhang and P. Li, "Spike-train level backpropagation for training deep recurrent spiking neural networks," Advances in neural information processing systems, vol. 32, 2019.
[6] M. Zhang, H. Qu, A. Belatreche, Y. Chen, and Z. Yi, “A highly effective and robust membrane potential-driven supervised learning method for spiking neurons,” IEEE transactions on neural networks and learning systems, vol. 30, no. 1, pp. 123–137, 2018.
[7] A. Lotfi Rezaabad and S. Vishwanath, "Long short-term memory spiking networks and their applications," in International Conference on Neuromorphic Systems 2020, pp. 1-9, 2020.
[8] N. Kasabov, K. Dhoble, N. Nuntalid, and G. Indiveri, "Dynamic evolving spiking neural networks for on-line spatio-and spectro-temporal pattern recognition," Neural Networks, vol. 41, pp. 188-201, 2013.
[27] M. Zhang, J. Wang, J. Wu, A. Belatreche, B. Amornpaisannon, Z. Zhang, V. P. K. Miriyala, H. Qu, Y. Chua, T. E. Carlson, et al., "Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks," IEEE transactions on neural networks and learning systems, vol. 33, no. 5, pp. 1947-1958, 2021.
[28] J. Shen, Y. Zhao, J. K. Liu, and Y. Wang, "Hybridsnn: Combining bio-machine strengths by boosting adaptive spiking neural networks," IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 9, pp. 5841 - 5855, 2021.
[29] X. Lagorce, G. Orchard, F. Galluppi, B. E. Shi, and R. B. Benosman, "Hots: A hierarchy of event-based time-surfaces for pattern recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 7, pp. 1346-1359, 2017.
[30] C. Shi, J. Li, Y. Wang, and G. Luo, "Exploiting lightweight statistical learning for event-based vision processing," IEEE Access, vol. 6, pp. 19396-19406, 2018.
[31] M. Cannici, M. Ciccone, A. Romanoni, and M. Matteucci, “Attention mechanisms for object recognition with event-based cameras,” in 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1127–1136, 2019.
[32] A. Sironi, M. Brambilla, N. Bourdis, X. Lagorce, and R. Benosman, “Hats: Histograms of averaged time surfaces for robust event-based object classification,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1731–1740, 2018.
[33] Z. Wu, H. Zhang, Y. Lin, G. Li, M. Wang, and Y. Tang, "Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing," IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 11, pp. 6249-6262, 2022.