Enhancing Adaptive History Reserving by Spiking Convolutional Block Attention Module in Recurrent Neural Networks
Authors: Qi Xu
Yuyuan Gao
Jiangrong Shen
Yaxin Li
Xuming Ran
Huajin Tang
Gang Pan
Venue: NeurIPS 2023 poster (2023)
Keywords: Spiking neural networks (SNNs)
Recurrent spiking neural network (RSNN)
Dynamic Vision Sensor (DVS)
Spiking convolutional block attention module (SCBAM)
TL;DR: Enhancing Adaptive History Reserving by Spiking Convolutional Block Attention Module in Recurrent Neural Networks

Abstract: Spiking neural networks (SNNs) serve as one type of efficient model to process spatio-temporal patterns in time series, such as the Address-Event Representation data collected from Dynamic Vision Sensor (DVS). Although convolutional SNNs have achieved remarkable performance on these AER datasets, benefiting from the predominant spatial feature extraction ability of convolutional structure, they ignore temporal features related to sequential time points. In this paper, we develop a recurrent spiking neural network (RSNN) model embedded with an advanced spiking convolutional block attention module (SCBAM) component to combine both spatial and temporal features of spatio-temporal patterns. It invokes the history information in spatial and temporal channels adaptively through SCBAM, which brings the advantages of efficient memory calling and history redundancy elimination. The performance of our model was evaluated in DVS128-Gesture dataset and other time-series datasets. The experimental results show that the proposed SRNN-SCBAM model makes better use of the history information in spatial and temporal dimensions with less memory space, and achieves higher accuracy compared to other models.

Enhancing Adaptive History Reserving by Spiking Convolutional Block Attention Module in Recurrent
Neural Networks
Qi Xu
Yuyuan Gao! Jiangrong Shen Yaxin Li" Gang Pan
Xuming Ran"
Huajin Tang²
School of Artificial Intelligence, Dalian University of Technology College of Computer Science and Technology, Zhejiang University
Abstract
Spiking neural networks (SNNs) serve as one type of efficient model to process spatio-temporal patterns in time series, such as the Address-Event Representation data collected from Dynamic Vision Sensor (DVS). Although convolutional SNNs have achieved remarkable performance on these AER datasets, benefiting from the predominant spatial feature extraction ability of convolutional structure, they ignore temporal features related to sequential time points. In this paper, we develop a recurrent spiking neural network (RSNN) model embedded with an advanced spiking convolutional block attention module (SCBAM) component to combine both spatial and temporal features of spatio-temporal patterns. invokes the history information in spatial and temporal channels adaptively through SCBAM, which brings the advantages of efficient memory calling and history redundancy elimination. The performance of our model was evaluated in dataset and other time-series datasets. The experimental results show that the proposed SRNN-SCBAM model makes better use of the history information in spatial and temporal dimensions with less memory space, and achieves higher accuracy compared to other models.
Introduction
Especially, the eligibility traces are induced by the fading memory of spike events where the presynaptic neuron fired before the postsynapstic neuron (3] Inspired by the loop behaviors of neurons, Recurrent spiking neural networks (RSNNs), have been proposed and proven to have potential in processing spatio- temporal patterns through learning from the historical context. Many of those existing approaches suggest memory preserving and extraction mechanisms play crucial role in exploiting historical context underlying spatio-temporal dynamics. Although the current recurrent structures RSNNS can capture the spatial and temporal variations at each level of the networks, the importance of different channels at different time points is ignored (4H6) which contributes the memory preservation and extraction can not focus on the small and important channels at specific time points.
Neurons in the human brain maintain preceding activity traces on the molecular level 2
Different recurrent structures of SNNs have been proposed capture the memory. Like Long Short- Term Memory (LSTM) cells in ANNs, the LSTM spiking networks 7 preserve valuable historical information by latching the gradients of hidden state, proficient processing time series data. To enhance the spatial feature extraction ability further of LSTM, its extension, named ConvLSTM, with
Corresponding authors: jrshen @ www..cdu.cn.......
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

convolutional structures in both input-to-state and state-to-state transitions. Meanwhile, the Address- event representation (AER) 8 data is one typical spatio-tempora pattern and has contextual spatial and temporal cues. Considering the advantages of ConvLSTMM over LSTTM, we aim to introduce the ConVLSTM structure into SNNs, to construct the recurrent spiking networks with ConVLSTM structures.
The core benefit of SNNs with recurrent structures is the memory invoking along with sequential time points. However, most of the current recurrent structures of SNNs ignore the influence distinction of different historical contexts on spatial and temporal dimensions. The attention mechanism has been studied and achieved to tell where to focus and improve the representation of interests. The convolutional block attention module (CBAM) extracts the key channel and spatial information by emphasizing the significant features. We adopt the CBAM component to implement the history invoking adaptively in forgetting gates in JSNNs.Because CBAM can conduct Attention on the channel and spatial dimensions. Because CBAM can conduct Attention on the channel and spatial dimensions, we can adjust the weight of forgetring gate information. Therefore, we propose the spiking ConvLSTM with Spiking CBAM component to invoke the history information in spatial and temporal channels adaptively, which brings the advantages of efficient memory calling and history redundancy elimination.
The contributions of the paper are as follows:
The spiking recurrent neural networks model with the spiking convolutional block attention module (CBAM) component (SRNN-SCBAM) is proposed to combine both spatial and temporal features of spatio-temporal patterns.
The proposed model invokes the history information in spatial and temporal channels adaptively through spiking CBAM, which brings the advantages of efficient memory calling and history redundancy elimination.
The experiments are conducted LIFARIO-DV'S and DVS S1551288-Gesss Gesture datasets. The ablation study and comparison results show that our model achieves competitive performance among current RSNNS models.
2 Related Work
Recurrent SNNs. Recurrent structures have manifested great potential in processing spatio-temporal patterns, benefiting from the information integration over time by multiple participant computations of each recurrent neuron. Current RSNNs contain two different kinds of network architectures according to the structural constraints. Firstly, based on the reservoir with randomly connected recurrent structures, The first-order reduced and controlled error algorithm could learn arbitrary firing patterns by stabilizing strong chaotic rate fluctuations in a network of excitatory and inhibitory neurons respecting Dale's 's law. Considering the inefficient and unstable connectivity of separating the excitatory and inhibitory populations, the framework of training the naatinuous-variabee rate RSNNs with biophysical constraints and transferring the learned dynamics to spiking RSNNs, by introducing one parameter to establish the equivalence between rate and spiking RSNNS [10
Another category of the RSNNs is based on learnable recurrent structures. Learning from the training method of artificial neural networks with recurrent structures, the backpropagation through time (BPTT) is applied to the training of RSNNs 12 [3 which requires propagation of gradients backward in time by unfolding recurrent layers. In the view of biologically realistic [4 [5], e- e-prop optimally combines the local eligibility traces and top-down signals and formulates the gradient descent learning in RSNNs [16]. Besides, RSNN models can also be implemented on neuromorphic hardware, such as SpiNNaker [7 or Loihi chips [8]. bringing computing efficiency combined with device properties.
Attention Mechanism. Human's 's visual system exploits the sequence of partial glimpses instead of processing the whole scene at once. Inspired by that property, the attention mechanism concentrates on the selective information in the input according to the sensitivity of the output to the variant inputs 11 22 propose temporal-wise attention SNN (TA-SNN) model to select relevant frames in the training stage and discard irrelevant frames in the inference stage by judging the significance of frames for the final decision. The convolutional block attention module (CBAM) module has been applied to CNNs to capture meaningful features along spatial and channel dimensions. The

---

(b)
+
Tode
Channel Attention Module
UfWode
Spatial
(c)
Figure 1: The framework of the proposed with spiking ConvLSTMs and fully-connected layers. The spiking CBAM can adaptively select the key features both spatial and temporal domains, and its effect on forgetting gating of spiking ConvI.STMs can invoke the history memories efficiently and eliminate the history redundancy, thus improving the processing of the spatiotemporal patterns. (b) The spiking ConvLSTMs are implemented to exploit the spatiotemporal feature through the proposed spiking convolutional component and spiking LSTM module. (c) The spiking CBAM component captures the sparse and complementary key features on the spatial and channel domain.
(a) The network structure of the RSNNs
intermediate feature map in CNNs is adaptively refined through CBAM at each convolutional block (33. However, there is still a lack of combination attention modules to apply in RSNNs both in temporal and spatial dimensions.
3 Method
In this section, we show detailed information on the SRNN-SCBAM model. In order to solve the problem that most SNN recurrent structures ignore the difference in the impact of different historical backgrounds on the spatial and temporal dimensions, we propose that the spiking ConvLSTM with CBAM components can adaptively call the historical information in the spatial and temporal channels. We can model DVS and other time series data, and complete the tasks of classification and prediction. For other data sets such as cifar 1ODar, we additionally add convolution pooling deepen the network structure.
3.1 The framework of the RSNN-CBAM
In the classification task for DVS128 Gesture, we used the network framework shown in Figure 1. The entire network of RSNN-CBAR consists of layer of pooling layer, a layer of Spiking Convlstm with CBAM components, and fully connected layer. Using SRNN, a PyTorch-basse encoder-classinee classifier model is implemented for neuromorphic data sets such as DVS128 Gesture. sort. Use pooling reduce the dimensionality of data and remove redundant information, use Spiking Convistm with CBAM components to extract spatial and channel features from neuromorphic data, and then send them the classifier for classification.
3.2 Spiking ConvL.TT
We introduce ConvLSTM into SNN and use its time and space feature extraction ability to accurately process time series data. Spiking ConVL.STM similar Ann ConvL.STM, and also overcomes the shortcomings of LSTM in processing three-dimensional information. And like the traditional

---

LSTM, the core concept of spiking ConvLSTM lies in the cell state, Ct2 and "gate" structure. The cell state is able to pass on relevant information in the process of sequence processing all the way. The "gate" structure learns which information to keep or forget during training.
Spiking ConvLSTM input is 3D tensor, and the last two dimensions spatial dimensions (Height and Weight), For the data at each timestep T, Spiking ConvLSTY replaces part of the connection operation LSTM with a convolution operation, which has convolution structure in both input-to- state and state-to-state transitions, through the current input and the past state of local neighbors.
Spiking ConvLSTM is more suitable for processing dynamic spatio temporal patterns, such gesture recognition with clear gesture moving trajectories but without preserving the details of fine-gre grained images with nigh-resolution pixels.
More specifically, given set of spiking inputs æ 22, ar, the gates and states are characterized as follows:
ft σi (WF,h h h Wf Wt bj)
i = (wi,h * h + wi * 2t + bi).
9, 02 (Wggh h Wgg T. bg)
(1)
C J, O Ct-1 + ds ④ 91
O, σi (Wo,h h + W Dt + bo)
h = Of ④ Ct:
where @ represents the Hadamard product,* * represents convolution operation, and 01 and d2 are spike activations that map the membrane potential of a neuron, spike if it exceeds the threshold value respectively. Also, w, and b, denote associated weights and biases. For the network, respectively. a, and h respectively correspond the spiking input at the current moment and the hidden state at the previous moment.
The forget gate, denoted by ft, decides which information should be ignored. The input gate ét- which controls the information entering the cell, has another auxiliary input layer gt, modulated by another peak activation sigmaa. Finally, the output of the spiking Convistm is formed based on the output gate o, and the cell state 6,.
3.3 Spiking CBAM
On the basis of spiking Convistm, spiking ConvLSTM with CBAM component is proposed to adaptively call historical information in space and time channels. The forget gate in spiking Convlstm is obtained by splicing the pulse input at the current moment and the hidden state at the previous moment, and then obtaining it through convolution and segmentation. For the CBAM component RSNN-CBAM, the forget gate is used as input, and will sequentially extract the one-dimensional channel attention map and the wwo-dimensional space attention map of the forget gate, thereby extracting the channel features and spatial features of the forget gate, and further, for space and time, the historical information in the channel is adjusted. The overall attention process can be summarized as:
the attention mechanism improve the forget gate. The
f Mc(G.) @ fo r"' M. (C) @ f,
(2)
Where @ denotes clement-wise multiplication. M. is the channel attention module, and M, is the spatial attention module. Mc and Ms sequentially perform channel feature extraction and spatial feature extraction on the forget gate to generate chanmel attention module Mc (fe) and spatial attention module M,(f)) f. is the spiking CBAM output. The calculation process of each attention diagram shown in the figure. The following describes the details of each attention module.
The channel attention component in Spiking CBAM uses balanced pooling and maximum pooling features to generate a one-dimensional channel attention map of forget gate through SNN-MLP. The channel attention is calculated as:

---

The ourput RENN with Spiking CBAM
The ourput of RSNN without Spiling CBAN
Time Soog
Thee Stap>
Vap
Texe Seop
Tixe Skg
Tix: Ekp
The visualization of extract features of RSNNs with and without SCBAM. We draw the feature map in spiking Convlstm layer the time steps of Ozh, 5th, 10th, 15th and 19 The RSNNs with spiking CBAM capture more sparse and complementary features compared with the RSNNs without spiking CBAM.
Figure 2:
Me(G) = Mc(G)==oSMMPP// (SMMIP(AvgPool(()) + SMLP(MaxPPoo((.))
where represents the number of spiking activations, Average pooling, and maximum pooling feature map share SNN-MLP weight.
The spatial attention component in Spiking CBAM applies the average pooling and maximum pooling operations along the channel axis, performs the convolution operation on it, and sends into the spiking activation function to generate the two-dimensional spatial attention feature map of the forget gate. The spatial attention is calculated as:
M, =
(4)
Where a represents the number of spiking activations, and TTX represents a convolution operation with the filter size of 23
Through the above two attention mechanisms, we obtained the one-dimensional channel attention map and two-dimensional spatial attention map of the forget gate, extracted the channel features and spatial features of the forget gate, and used Spiking CBAM adaptively recalls historical information in both spatial and temporal channels, and ablation studies and visualizations are performed to show the effectiveness of the attention component.
3.4 Surrogate Gradient Method
We mainly use the Leaky integrate-and-fire (LIF) node as the basic unit. The LIF model also the most commonly used model to deseribe the dynamics of SNN neurons, which can be described by the following equation:
du(t) dt
-u() + 1(t),
(5)
Where u(l) is the neuron membrane potential (membrane potential) at time t, is the time constant, and (t) represents the presynaptic signal determined by the previous neuron's pulse activity or external stimulus and synaptic weights. enter. When the membrane potential u exceeds a specified threshold the neuron fires pulse and resets its membrane potential to Ureset-
In order to solve the problem that the loss function is not differentiable in the backpropagation in SNN, the surrogate gradient training method 26). The substitution function is used to generate spikes in spiking Convlstm and spiking CBAM.
The principle of the gradient substitution method is to use the Heaviside step function in the forward propagation: y = ((x), While backpropagating, the substitution function is used: dy/ dz = (x).. Where (zz) is a function similar in shape to ((z) but smooth and continuous. We mainly use the
5

---

Time Step
Time Stop
Time Step
Time Step
Time Stop 19
Figure 3: Visualization of input neuromorphic data with a size of 2*1 11128*122 128. We plot the feature maps of the input image time steps of Oeh. 5th 10th- 15th and 19th-
spike function of the gradient of the Gaussian error function (erf) during backpropagation. The corresponding original function is:
g(x) (1 (01-r((-ax))
1 v
dt.
(6)
Where a the parameter that controls the smoothness of the gradient during backpropagation. Backpropagation employs the gradient descent rule according to the derivatives:
g(xx)
Then, the derivatives of the gradient of the variables can be approximated by the above formulations.cr the derivatives of the gradient of the variables can be approximated by the above formulations.
Results
We conduct the following experiments to evaluate the performance the proposed RSNN-CBAM models. We first introduce the experimental settings. Then, the classification accuracy of RSNN- CBAM is recorded and compared to the current state-of-the-art models. The ablation study and visualization are followed to indicate the effectiveness of the attention component.
4.1 Experimental Setup
In order to further show the performance of the proposed method dynamic spatio-temporal information processing, we evaluate the classification accuracy on CIFARIO-DE and DVS128 Gesture datasets. 10-DDS is event-based version of the classical CIFARI0 dataset which also contains 10 classes with 10,000 event streams converted from images. Similar to CIFARIO-DVS. DVS128-GE 221eGerreeereeeeee is an event-based spiking dataset that contains 11 hand gestures from 29 subjects. Based on these spatio-temporal event streams 22 28), the proposed method was evaluated for dynamic information processing and its adaptive history reserving ability.
Detailed parameter settings are listed Table T The learning rate scheduler for RSNN-CBAM is set to be le-3 (selected from linear warm-up and decay step decay). The learning epoch is 200 and the batch size is 32. The integration time constants 7 of LIF neurons are set be 4.0. a is the parameter that controls the smoothness of the gradient during backpropagation for surrogate gradient learning process, which set to be 4.0 in this paper. Ureset the reset voltage of the neuron and set be The above content will be added to the revision.

---

Table 1: Experimental parameter settings.
Table 2: Accuracy of different solutions for the DVS128 GESTURE dataset (11 classes).
Parameters
Gesture
Cifarlo-DVS
Method
f
i
Accuracy (%)
Epuoc
200
1024
Spiking Convistm
0
U
0
93.3
spiking Convistm+CBAM
1
0
0
92.4
Batchsize
32
Spiking Convistm+Spikking CBAM
0
1
1
86.1
Learning rale
1e-3
1e-3
Spiking Convlsm=-Spiking CBAM
1
88.8
LIF, au
2.0
2.0
Spiking ConvlstmmSppkking
0
95.5
Spiking Convlstm-Spikking CBAM
1
1
92.0
Mreset
Spiking Convistm--Spiking CBAM
1
91.3
ALL the following experiments were conducted on either an A100-SMX GPU. The network structure is composed of one layer of Convlstm neurons and three FCLIFID layers. The results of the experiments are given in Table 2
as shown in this table, most of the accuracies among the various structures are increased based on the proposed method.
4.2 Ablation Study
In this section, the ablation study conducted to validate the effectiveness of each component proposed in this paper.
The effectiveness of SCBAM in SRNN. The spiking CBAM component employed in SRNN to implement selective memory preservation. As illustrated in Tab. 2 we conduct the ablation study on the DVS128 GESTURE dataset. Firstly, the Spiking ConvLSTM with spiking CBAM module achieves the highest test accuracy of 95%, which performs better than the one without any CBAM component over 1.7% accuracy gap. indicates the effectiveness of Spiking CBAM in improving the performance of Spiking ConvLSTM. Secondly, the proposed Spiking CBAM component achieves better classification accuracy than the accuracy drop of 2.6% original CBAM component. It suggests that the proposed spiking CBAM component captures more precise features than the original CBAM the spiking ConvLSTMM module because of better temporal feature selection capability.
Explore the adaption of different gates to SCBAM. We employ the spiking CBAM on the forgetting gate control the memory preserving and invoking for Spiking ConvLSTM. Here we further explore the performance of other gates in Spiking ConvLSTM to show the efficiency of the forgetting gate. As shown in the last five columns Table. 2 the results suggest the priority of the forgetting gate compared with the other two input and output gates. The forgetting gate obtains the largest accuracy and exceeds nearly 10% accuracy than other gatings. It is reasonable that the forgetting gate with spiking CBAM controls the history information invoking and produces marked effect on the effective computation of spiking CSnVLSTM.
70-
33-
3-
o
-3-
-15
(a) The visualization of extract features of RSNNs with SCBAM
(b) The visualization of extract features of RSNNs without SCBAM
Figure 4: The visualization of extract features of RSNNs with and without SCBAM. The RSNNs with spiking CBAM capture more sparse and complementary features compared with the RSNNs without spiking CBAM.

---

Thea Sae
Stap
Time Shep
Tew 11
TIXE
TineSap
(a) The output of RSNN with Spiking CBAM
(b) The output of RSNN without Spiking CBAM
The figure above depicts visualizations dataset. We selected time steps of 0, 5, 10, and 15 to compare the extracted features of RSNNs with and without SCBAM. shows that the RSNN module with SCBAM captures information as anticipated, exhibiting strong sparsity across both temporal and spatial dimensions throughout the entire time steps while the RSNN module without SCBAM only extracts limited features which may lose significant amount of crucial information.
Figure 5:
RSNN output tensors on the CIFAR IO-DVS
4.3 Visualization analysis
To validate the effect of the SCBAM on the selection of the key features in the spatial and channel dimensions, we observe the feature map visualization of the RSNN-SCBAM model on the DVS128 GESTURE and CIFARIO-DVS datasets under the situation with and without SCBAM component.
DVS128 GESTURE feature map visualization results are illustrated in Fig. the RSNN module with SCBAM captures more complementary and sparse features than the one without SCBAM. On one hand, the SCBAM possesses fine sparsity in the channel and spatial dimensions through the whole time steps. That benefits from the attention mechanism that captures significant features and eliminates redundant features. On the other hand, most of the features extracted by SCBAM are complementary each other. Each feature map focuses and the shoulders. However, some feature maps of the RSNNs without SCBAM are quite similar because of the lack of filtering of redundant information. Based on the proposed SRNN-CBAM, the historical redundancy left in other spiking recurrent models could be discarded sharply, meanwhile, the proposed method maintains good performance by adaptive history reserving.
specific area such the hands
CIFARIO-DVS feature map visualization results are illustrated in Fig. 5 we visualize the extracted features at several time steps of the proposed RSNNs with and without SCBAM. It shows that the RSNN module with SCBAM captures information as anticipated, exhibiting strong sparsity across both temporal and spatial dimensions throughout the entire time steps while the RSNN module without SCBAM only extracts limited features which may lose a significant amount of crucial information.
further conducted ablation experiments on dynamic thresholding during training on the DVSGes- ture dataset, and our LIFNode and MultiStepLIFNeo SNN. We set the thresholds of MultiStepLIFNode and LIFNode as trainable parameters and conduct experiments to verify their impact. The experimental results shown in Table. shown.
model adopted two different dynamic thresholding methods in
Table 3: Accuracy of two different dynamic threshold methods for the DVS128 GESTURE dataset.
Table 4: Accuracy of additional time step val- ues of 10, 15, and 25 on the DVSGesture dataset.
Method
threshhre
Accuracy (%)
Method
Time step
Accuracy (%)
RSNN-SCBAM
LIFNode
91.3
RSNN-SCBAM
89,2
RSNN-SCBAM
92.4
SCBAM
MulidStepL.LENodee
89.9
RSNN-SCRAM
90.3
However, the current results do not show any significant improvement in dynamic thresholds. This threshold adjustment may make the model relatively complex, involving more parameters and

---

Table Accuracy of solutions for the CIFARIO-DVS dataset (10 classes).
Method
Accuracy (%)
HOTS 29
27.1
Lightweight Statistical 30
31.2
Attention Mcchanisms 31
44.1
HATS X
52.4
LIF-NET
63.5
LIAF-NETE
70.4
Proposed CBAM-SNN
66.3
nonlinear operations. If the parameters or network structure of the model are not adjusted correctly, poor model performance may result. Furthermore, SNNs may require large amounts of training data and longer training times to achieve effective learning and optimization.
Regarding the variation in LSTM time steps, we conduct experiments using additional time step values of 10, 15, and 25 on the DVSGesture dataset. As shown in Table. the results show that the time step of 20 is the optimal choice. This could be attributed the nature of the DVSGesture data, where gestures are repeated by individuals at specific time intervals. The accuracies of the RSNN-CSAM increase first and then reduce as the growth of the time steps from 10 to 25. Different time steps capture varying temporal features, leading to differences in test results. Fewer time steps could cause the excessive concentration of information and then could not extract enough event features, while the larger time steps may extract superfluous information and could not capture the key information.
4.4 Comparison with other methods.
We conduct performance comparison on the CIFARIO-DVS dataset to explore the efficiency of the proposed model. The compared models employs spatio-temporal features called temporal surfaces to create an event-based hierarchical pattern recognition architecture, adopt three layers of time surface prototypes in the experiment. The article [30] represents the visual information captured by DVS as a stream of asynchronous pixel addresses (events) representing relative intensity changes at those locations, and uses a random ferns classifier to classify the stream of pixel events. The network structure consists of the following three modules: a motion detector, a bank of binary feature extractors, and a Random Ferns classifier. The number of ferns for size range between 12 and 20. The paper (31) focuses on visual attention, specific feature of vision, and proposes two attention models for event-based vision. We adopt the p-N-DRAY AW recognition network, using PL.STM layers as encoders. The paper 322 introduces novel event- based feature representation and new machine learning architecture. And use the local memory unit to efficiently utilize past-time information and build robust event-based representation. In the experiment, temporal surface representation method with local memory was used to divide all event points within a spatial region into different fixed cells. Within each cell, all event points are accumulated to generate temporal surface, obtaining features. Classification is achieved using an SVM (Support Vector Machine). The paper [33] proposes a neuron normalization technique to tune ncural selectivity and develops direct learning algorithm for deep SNNs. The network architecture employed in [33 is as follows: LIAF Blocks and Dense Blocks connected end-to Ld-to--nd.LLAF Block has a sequential structure, consisting of ConvLIAF, TD-layer- normaliiomiiiiiiioonn TD-activation (ReLU) and TD-AvgPooling, where TD refers to the dime-iistr operation.
follows: The HOTS method proposed in (29
10-DVS is (50, 14), and there are 15,000 segment events with patch
A detailed comparison of SNN based models for the benchmark is shown in Table 5 the proposed SRNN-CBAM achieves competitive performance among those models. Our proposed SRNN-CBAM method performs better than the LIF-NET model in 33 Although the classification accuracy of SRNN-CBAM is lower than L.AF-Net (3), the adaptive memory invoking should be emphasized for the further application of spatiotemporal pattern processing.

---

Conclusion
Different from some classical spiking based models which aim static pattern recognition, this paper utilized the dynamic spatio-temporal information processing ability of SNNs on event data. This paper proposes a brand new spiking recurrent neural network model with spiking convolutional block attention mechanism (SRNN-SCBAM) which can invoke the temporal history information in spatial and temporal features adaptively. has prior advantages in efficient memory calling and irrelevant history information elimination. The experiment results show that the proposed model can achieve competitive performance among other oottt--f--he-rrr models, and the attention component captures meaningful features according to the sensitivity of the output to the input in the end-to-end training process.
At the same time, we believe that the proposed Spiking ConvLST is good at different time-series data and applications, such as object tracking, and worth noting that the proposed Spiking ConvLSTM needs the spike-based input, hence, when facing other time-series data, the input data should firstly be encoded into spike trains.
In addition, it is worth noting that we take a further step in processing efficient memory invoking and preserving of spiking neural networks, which has promising potential in processing and memorizing spatio-tempora patterns. This paper rethinks the role that attention plays in spiking recurrent neural networks, which not only provides new way of constructing efficient model for dynamic event stream data processing but more importantly that the proposed model could mimic the mechanism of potential information processing in neural circuits. Remember what should be remembered, and forget what should be forgotten.
Acknowledgement
This work supported in part by National Key Research and Development Program of China (0913),0098033), National Natural Science Foundation of China (NSFC No.62206037 and No. 62306274), the China Postdoctoral Science Foundation under Grant No. 2023M733067 and No. 2023T160567, and the Fundamental Research Funds for the Central Universities (DUT2IRC(3)091)
References
[1] R. Dolmetsch and D. H. Geschwind, "The human brain in a dish: the promise of ipsc-derived neurons," Cell, vol. 145, no. 6, pp. 831-834, 2011.
[2] V. Kukekov, E. Laywell, O. Suslov, K. Davies, B. Scheffler, L. Thomas, T. O'brien, M. Kusakabe, and D. Steindler, "Multipotent stem/progenitor cells with similar properties arise from two neurogenic regions adult human brain," Experimental neurology, vol. 156, no. 2, pp. 333-344, 1999.
[3] K. Roy, A. Jaiswal, and P. Panda, "Towards spike-based machine intelligence with neuromorphic computing," " Nature, vol. 575, no. 7784, pp. 607-617,40 2019.
[4] A. Shaban, S. S. Bezugam, and M. Suri, "An adaptive threshold neuron for recurrent spiking neural networks with nanodevice hardware implementation," Nature Communications, vol. 12, no. 1, pp. 1-11, 2021.
[5] W. Zhang and P. Li. "Spike-train level backpropagation for training deep recurrent spiking neural networks," Advances in neural information processing systems, vol. 32, 2019.
[6] M. Zhang, H. Qu, A. Belatreche, Y. Chen, and Z. Yi, "A highly effective and robust membrane potential-driven supervised learning method for spiking neurons," IEEE transactions on neural networks and learning systems, vol. 30, no. 1, pp. 123-137, 2018.
[7] A. Lotfi Rezaabad and S. Vishwanath, "Long short-term memory spiking networks and their applications," in International Conference on Neuromorphic Systems 2020, pp. 1-9, 2020.
[8] N. Kasabov, K. Dhoble, N. Nuntalid, and G. Indiveri, "Dynamic evolving spiking neural networks for on-line spatio-and spectro-temporal pattern recognition," Neural Networks, vol. 41, pp. 188-201. 2013.
10

---

[9] S.-C. Liu and T. Delbruck, "Neuromorphic sensory systems," Current opinion in neurobiology. vol. 20, no. 3, pp. 188-295. 2010.
[10] R. Kim, Li, and J. Sejnowski, "Simple framework for constructing functional spiking recurrent neural networks." Proceedings of the national academy of sciences, vol. 116, no. 45, pp. 2221122222222 122200, 2019.
[11] Q. Xu, Y. Li, J. Shen, J. K. Liu, H. Tang, and G. Pan, "Constructing deep spiking neural networks from artificial neural networks with knowledge distillation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7886-7895, 2023.
[12] J. Shen, J. K. Liu, and Y. Wang, "Dynamic spatiotemporal pattern recognition with recurrent spiking neural network," " Neural Computation, vol. 33, no. 11, pp. 2971-2995, 2021.
[13] Y. Guo, Huang, and 7.. Ma, "Direct learning- based deep spiking neural networks: a review," Frontiers in Neuroscience, vol. 17, p. 1209795, 2023.
[14] Y. Guo, W. Peng. Y. Chen, L. Zhang, X. Liu, X. Huang, and Z. Ma, "Joint a-snn: Joint training of artificial and spiking neural networks via self-distillation and weight factorization," Pattern Recognition, vol. 142, p. 109639, 2023.
[15] T. Bu, J. Ding, Z. Yu, and T. Huang, "Optimized potential initialization for low-latency spiking neural networks," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 11-20, 2022.
[16] G. Bellec, F. Scherr, A. Subramoney, Hajek, D. Salaj, R. Legenstein, and W. Maass, "A solu- tion to the learning dilemma for recurrent networks of spiking neurons," Nature communications, vol. 11, no. 1, pp. 1-15, 2020.
[17] S. B. Furber, F. Galluppi, S. Temple, and L. A. Plana, "The spinnaker project," Proceedings of the IEEE, vol. 102, no. 5, pp. 652-666-6666660666 665, 2014.
[18) M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain, et al., "Loihi: A neuromorphic manycore processor with on-chip learning," leee Micro, vol. 38, no. pp. 82-99, 2018
[19] A. Vaswani, Shazeer, N. Parmar, Uszkoreit, L.. Jones, N. Gomez, Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
[20] Q. Xu, Li, J. Shen, P. Zhang. J. K. Liu, H. Tang, and G. Pan, "Hierarchical spiking- based model for efficient image classification with enhanced feature extraction and encoding." IEEE Transactions on Neural Networks and Learning Systems, 2022.
[21] Q. Xu, J. Peng, J. Shen, H. Tang, and G. Pan, "Deep covdensesnn: A hierarchical event-driven dynamic framework with spiking neurons in noisy environment," Neural Networks, vol. 121, pp. 512-519, 2020.
[22] M. Yao, H. Gao, G. Zhao, D. Wang, Y. Lin, Z. Yang, and G. Li, "Temporal-we wiiee attention spiking neural networks for event streams classification." pp. 10221-10234 2021.
[23] Woo, J. Park, J.-Y. Lee, and I. S. Kweon, "Cbam: Convolutional block attention module," in
Proceedings of the European conference computer vision (ECCV), pp. 3-19, 2018.
[24] E. O. Neftci, H. Mostafa, and F. Zenke, "Surrogate gradient learning in spiking neural networks," 2019.
[25] J. Shen, Q. Xu, K. Liu, Y. Wang, G. Pan, and H. Tang. "Esl-snns: An evolutionary structure learning strategy for spiking neural networks," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 86-93, 2023.
[26] T. Bu, J. Ding, Z. Hao, and Z. Yu, "Rate gradient approximation attack threats deep spiking neural networks," Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7896-7906, 2023.
11

---

[27] M. Zhang. J. Wang, J. Wu, A. Belatreche, B. Amompaisannon, Z. Zhang, V. P. K. Miriyala, H. Qu. Y. Chua, T. Carlson, et al., "Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks." IEFE transactions neural networks and learning systems, vol. 33, no. 5, pp. 1947-1958, 2021.
[28] Shen, Zhao, J. K. Liu, and Y. Wang. "Hybridsmm: Combining bio-machine strengths by boosting adaptive spiking neural networks," IEEE Transactions Neural Networks and Learning Systems, vol. 34, no. 9, pp. 5841 5855, 2021.
[29] X. Lagorce, G. Orchard, F. Galluppi, B. E. Shi, and R. B. Benosman, "Hots: A hierarchy of event-based time-surfaces for pattern recognition," IEEE Trunsactions Pultern Analysis and Machine Intelligence, vol. 39, no. 7, pp. 1346-1359, 2017.
[30] C. Shi, Li, Y. Wang, and G. Luo, "Exploiting lightweight statistical learning for event- based vision processing." " IEEE Access, vol. pp. 19396-194066 2018.
[31] M. Cannici, M. Ciccone, A. Romanoni, and M. Matteucei, "Attention mechanisms for object recognition with event-based cameras," in 2019 IEEE Winter Conference Applications of Computer Vision (WACV), pp. 1127-1136, 2019.
[32] A. Sironi, M. Brambilla, N. Bourdis, X. Lagorce, and R. Benosman, "Hats: Histograms of averaged time surfaces for robust event- based object classification." in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1731-1740, 2018.
[33] Z. Wu, H. Zhang, Lin, G. Li, M. Wang, and Y. Tang, "Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing," IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 11, pp. 6249-6262, 2022.
12


Reviews:
[{"review_id": "h4ePMXYHFT", "reviewer": "Reviewer_9YjJ", "summary": "This article proposes a spike recurrent neural network model with a spiking convolutional block attention mechanism, called SRNN-SCBAM. Its main idea is to adaptively call historical information in the spatio-temporal features of the spatio-temporal pattern, which has advantages in efficient memory calls and eliminating historical redundancy. The experiment on the DVS128 gesture dataset is conducted to verify the effectiveness of the model in utilizing adaptive historical information. The idea of the article is clear, the textual description is clear, and examples are provided to illustrate results.", "questions": "1.The parameter setting about the time constant in the first paragraph of section 2.4 should be explained in detail.\n2.In addition, there are some shortcomings in the experimental aspect of the paper. The visualization experimental results mentioned in the article on the Cifar10-DVS can be supplemented with appropriate experiments.", "limitations": "1.The parameter should be explained further.\n2.More visualization results could be added. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.The paper is easy to read, and generally well written.\n2.The proposed model of RSNN is simple but effective, the motivation is clear.\n3.Experimental evaluations were conducted on two different neuromorphic datasets to demonstrate the performance improvement with the proposed SCBAM under different settings.", "weaknesses": "There is a lack of explanation for some parameter settings. In addition, there are some shortcomings in the experimental aspect of the paper. The experimental results mentioned in the article on the Cifar10-DVS generally can be supplemented with appropriate visualization results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 9YjJ, \n\nWe really appreciate your insightful comments and feedback. We addressed your questions below. We also revised our paper accordingly.\n\n\n**Q1:** \"The parameter setting.\"\n\n **A:**Thanks for the suggestion. All the parameter settings are listed in the following table. \nTogether with the description of section 4.1, the following parameters' setting will be added to the revision: The integration time constants $\\tau$ of LIF neurons are set to be 4.0.\n$\\alpha$ is the parameter that controls the smoothness of the gradient during backpropagation for surrogate gradient learning process, which is set to be 4.0 in this paper.\n$u_{reset}$ is the reset voltage of the neuron and set to be 0. The above content would be added to the revision.\n\n|    Parameters | DVS Gesture | Cifar10-DVS |\n|---------------|-------------|-------------|\n| Epoch         | 200        | 1024        |\n| Batchsize     | 32          | 32          |\n| LIF, $\tau$    | 2.0         | 2.0         |\n| T             | 20          | 20          |\n| $\\alpha$      | 4.0         | 4.0         |\n| $u_{reset}$   | 0           | 0           |\n\n**Q2:**\"the visualization results on CIFAR10-DVS\". \n\n**A:**Thanks for the suggestion, we have supplemented appropriate visual results on CIFAR10-DVS dataset, as shown in the one-page PDF rebuttal file. In detail, we visualize the extracted features at several time stps of the proposed RSNNs with and without SCBAM.  It shows that the RSNN module with SCBAM captures information as anticipated, exhibiting strong sparsity across both temporal and spatial dimensions throughout the entire time steps while the RSNN module without SCBAM only extracts limited features which may lose a significant amount of crucial information.\n\nThanks again for all of your valuable feedback, suggestions, and questions.\n\n"}, {"review_id": "wQJ5HtFyjb", "reviewer": "Reviewer_KuzU", "summary": "The authors proposed a Recurrent spiking neural network (RSNN). The essential component of the proposed RSNN is a spiking conv block attention module (SCBAM), which contains channel and spatial attention blocks. The proposed method is validated with classification tasks on CIFAR10-DVS and DVS128 gesture datasets. ", "questions": "Please see the weaknesses section. ", "limitations": "No limitation is provided. \n\nFrom my perspective, the work follows the thoughts of ANN without an insightful understanding of SNN, especially the temporal processing power of SNNs. SNNs offer compelling temporal feature extraction capability without any special treatment, evidenced by many works already. \n\nIn addition, I did not see the novelty of the proposed approach from either ANN or SNN. Therefore, I think the submission is way below the bar of NeurIPS. ", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "N/A", "weaknesses": "-1. The writing of the submission should be dramatically improved. Many things are hard to follow. In lines 242-243: \"Although the ...is lower than LIAF-Net, ...\". In Table 2, [23] provides lower accuracy than the proposed method. Therefore, I cannot really trust the provided experimental results.\n\n-2. lines 114-115: \"Spiking Convlstm is more suitable for removing  facial expression data....\" Why? \n\n-3. SNN is different from CNN, which can process temporal information without any treatment, especially since many works have proved that, such as [R-1], [R-2]. The process of membrane potential release, accumulation, and triggering spiking is a natural temporal extractor. I do not think using RNN-based architecture is a meaningful way to go in the SNN domain. At the minimum, the authors should provide experimental results of the comparison with these methods, such as [R-1] and [R-2]. \n\n-4. How does the potential threshold of LIF impact the performance of the proposed methods?\n\n-5. How does the number of LSTM steps impact the performance of the proposed methods?\n\n[R-1]: Biologically Inspired Dynamic Thresholds for Spiking neural networks. \n\n[R-2]: Spiking Transformers for Event-based Single object Tracking", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer KuzU,\n\nThank you for the thorough review and constructive criticism. There exist some misunderstanding about our paper, we hope the following the clarification would solve the proposed problems.\n\n**W1:**\"the mismatch about the description about comparison with LIAF-NET\" ?\n\n**A:**Thanks for pointing that. Actually, [23] contains two different model settings of LIF and LIAF with two different accuracies of 63.53% and 70.40% on CIFAR10-DVS dataset, respectively. Because the LIF model is employed in this study, we show the former result with 63.53% accuracy in Table 2 for direct comparison with our study. Meanwhile, we also refer the latter results of 70.40% in lines 242-243, with the desctiption of \" Although the classification accuracy of SRNN-CBAM is lower than LIAF-Net [23], the adaptive memory invoking should be emphasized for the further application of spatiotemporal pattern processing.\" to show the advantages and disadvantage of the proposed RSNN to illustrate the whole view of our study.\n\nWe would rewrite the corresponding description to make it clear, by adding both the results of LIF-NET and LIAE-NET in [23] into Table 2 and supplement the description of \"Our proposed SRNN-CBAM method performs better than the LIF-NET model in [23].\"\n\n**W2:**\"Spiking ConvLSTM is more suitable for removing facial expression data\"?\n\n**A:**Here It means that the Spiking ConvLSTM is more suitable for spike-based coarse-grained data instead of the  fine-grained images such as the facial expression data. In detail, on DVSGesture dataset, each sample is captured by the dynamic vision sensor (DVS), and the facial expression during finishing each gesture process is removed because of the event-driven response property of DVS. Hence, the Spiking ConvLSTM is more suitable to the DVS data instead of the pixel-level fine-grained images.\n\nThanks for pointing that, we would change the description to \"Spiking ConvLSTM is more suitable for processing the dynamic spatiotemporal patterns, such as the gesture recognition with the clear gesture moving trajectories but without preserving the details of fine-grained images with high resolution pixel.\n\"\n\n**W3 and W4:**\"comparison with [R-1] and [R-2]\" ? \"How does the potential threshold of LIF impact the performance of the proposed methods?\"\n\n\n**A:**: Thanks for your suggestion. Here we conducted ablation experiments about the dynamic threshold during training on the DVSGesture dataset, although we could not compare the spiking Transformer in [R-2] for its object tracking framework can not directly applied into our application scene. Two different dynamic threshold methods in SNNs with LIFNode and MultiStepLIFNode are employed in our RSNN-SCBAM model.  We make the thresholds of MultiStepLIFNode and LIFNode trainable parameters and conducted experiments to validate their impact. The results are as follows:\n\n|    Method | Node_ threshold | Acc (%)|\n|---------------|-------------|-------------|\n| RSNN-SCBAM\t| LIFNode\t\t| 91.3        |\n| RSNN-SCBAM\t| MultiStepLIFNode        | 89.9          |\n\nHowever, the current results do not show any significant improvements with the dynamic threshold. This threshold adjustment could make the model relatively intricate, involving an increased number of parameters and non-linear operations. If the model's parameters or network structure are not adjusted correctly, it might lead to suboptimal model performance. Additionally, SNNs might demand a larger amount of training data and an extended training time for effective learning and optimization.\n\n[1] Dayan P, Abbott L. Computational neuroscience: Theoretical neuroscience: Computational and mathematical modeling of neural systems[M]. Cambridge: MIT Press, 2001: 162-166.\n[2] Brunel N, Latham P E. Firing rate of the noisy quadratic integrate-and-fire neuron[J]. Neural Computation, 2003, 15(10): 2281-2306.\n[3] Pellegrini T, Zimmer R, Masquelier T. Low-activity supervised convolutional spiking neural networks applied to speech commands recognition[C]//2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021: 97-103.\n\n**W5:**\"How does the number steps impact the performance of the proposed methods?\"\n\n**A:**Regarding the variation in LSTM time steps, we conduct experiments using additional time step values of 10, 15, and 25 on the DVSGesture dataset. As shown in the following table, the results shows that  the time step of 20 is the optimal choice. This could be attributed to the nature of the DVSGesture data, where gestures are repeated by individuals at specific time intervals. The accuracis of the RSNN-CSAM increase first and then reduce as the growth of the time steps from 10 to 25. Different time steps capture varying temporal features, leading to differences in test results. Less time steps could cause the excessive concentration of information and then could not extract enough event feature, while the larger time steps may extract superfluous information and could not capture the key information.\n\n\n|    Method | Time step | Accuracy (%) |\n|---------------|-------------|-------------|\n| RSNN-SCBAM\t| 10\t\t| 89.2        |\n| RSNN-SCBAM\t| 15        | 92.4          |\n| RSNN-SCBAM\t| 25        | 90.3        |\n"}, {"review_id": "Oh0AMX8uez", "reviewer": "Reviewer_TiEP", "summary": "This study proposed the spiking recurrent neural networks model with a spiking convolutional block attention module component (SRNN-SCBAM). The proposed model invokes the historic information in spatial and temporal channels adaptively through spiking CBAM, which brings the advantages of efficient memory calling and history redundancy elimination. The experimental results show that the proposed SRNN-SCBAM model makes better use of the historic information in spatial and temporal dimensions with less memory space, and achieves higher accuracy compared to other models.", "questions": "1. Table 2: What is the network structure in [23]? The network structures should be presented more clearly to show the components in the network models.\n2. Why the performance of Spiking ConvLSTM with Spiking CBAM is quite higher than that with CBAM in Table 1?\n3. What is the parameter of alpha used in Equation (7)?\n4. It seems the proposed Spiking ConvLSTM is suitable for long-term time-series datasets due to its adaptive memory maintenance. Could the proposed model be applied to other time-series datasets? ", "limitations": "Some details of training the Recurrent SNNs are missing in the paper (only mentioned by a few sentences). It would be better to also include those settings in the appendix.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The learning algorithm for recurrent SNNs is simple and easy-to-understand.\n2. Well-thought-through reuse of standard modeling and algorithmic components to solve the memory calling and history redundancy elimination in recurrent SNNs.", "weaknesses": "Although the motivation is reasonable and the proposed methods are simple and easy to implement, the experimental results could be supplemented to indicate the advantages of the proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:**\"Table 2: What is the network structure in [23]? \"\n\n**A:**The network architecture employed in Table 2 [23] is as follows: 128C3(Encoding)-128C3-AP2-128C3-256C3-AP2-1024FC-Voting.\nWe supplement the network structures in Table 2 as follows:\nTable 2 [19] adopts three layers of time surface prototypes.\nTable 2 [20] consists of the following three modules: a motion detector, a bank of binary feature extractors, and a Random Ferns classifier. The number of ferns for CIFAR10-DVS is (50, 14), and there are 15,000 segment events with a patch size range between 12 and 20.\nTable 2 [21] Table 2 [21] adopts the p-N-DRAW recognition network, using pLSTM layers as encoders.\nTable 2 [22] employs the temporal surface representation method with local memory, which divides all event points within a spatial region into different fixed cells. Within each cell, all event points are accumulated to generate a temporal surface, obtaining features. Classification is achieved using an SVM (Support Vector Machine).\n\n**Q2:**\"Why the performance of Spiking ConvLSTM with Spiking CBAM is quite higher than that with CBAM in Table 1?\n\n**A:**Firstly, the transmission in SNN  is in the form of spike signals. The compatibility between ANN's CBAM and RSNN might not be particularly suitable to process event-based data. Secondly, in spiking attention, the MultiStepLIFNode is utilized, which leads to the higher performance of Spiking ConvLSTM with Spiking CBAM compared to using conventional CBAM.\n\n**Q3:**\"What is the parameter of alpha used in Equation (7)?\"\n\n**A:**$\\alpha$ is the parameter that controls the smoothness of the gradient during backpropagation for surrogate gradient learning process, which is set to be 4.0 in this paper.\nCurrently, it appears that the Spiking ConvLSTM could be applicable to other DVS (Dynamic Vision Sensor) datasets. Exploring its applicability to other time-series datasets is also a focus of our future work.\n\n\n**Q4:**\"It seems the proposed Spiking ConvLSTM is suitable for long-term time-series datasets due to its adaptive memory maintenance. Could the proposed model be applied to other time-series datasets?\"\n\n**A:**Yes, the proposed model can be applied to other time-series datasets. The proposed Spiking ConvLSTM is good at the different time-series data and applications, such as the object tracking, and it is worth noting that the  proposed Spiking ConvLSTM needs the spike-based input, hence, when facing other time-series data, the input data should firstly be encoded into spike trains. \n"}, {"review_id": "2rYrvH9m7A", "reviewer": "Reviewer_NNKj", "summary": "The present study introduces a novel model of spiking recurrent neural networks, which incorporates a spike convolutional block attention mechanism. This model is referred to as SRNN-SCBAM. The primary objective is to effectively incorporate historical information into the spatial and temporal characteristics of spatiotemporal patterns, resulting in improved memory retrieval and elimination of redundant historical data. The efficacy of the proposed model in leveraging historical information and attaining high precision has been validated through experiments conducted on DVS128-Gesture datasets.", "questions": "1.When encounter a new dataset, how the model works and how the architecture should be designed? I don't see discussions of this part in the paper.\n2.More ablations of the different gates being open or closed in Table 1 are required because the influence of ‘i’ is unclear. And if there exist the adaptive parameters to control the possibility of gate open, could it be better for RSNN to achieve better performance? ", "limitations": "The limitations are the same as the weakness.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The practical issue of invoking adaptive memory in spiking recurrent neural networks has been addressed in this paper. The primary objective is to incorporate a Spiking Convolutional Block Attention Module into the gating computation of Spiking ConvLSTM networks. This would enable the selective activation of historical information and the elimination of superfluous history during the training phase. This will provide the iterative calculation with a favorable initial state. The motivation is interesting and the idea of utilizing a spiking convolutional block attention mechanism to acquire historical data in gating computation appears intriguing.", "weaknesses": "1.More ablations of the different gates being open or closed in Table 1 are required because the influence of ‘i’ is unclear. And if there exist the adaptive parameters to control the possibility of gate open, could it be better for RSNN to achieve better performance? \n2.There is a typo in Figure 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful comments and insights regarding our article proposing the SRNN-SCBAM model. We would like to address the points raised and provide further clarification on certain aspects.\n\n\n**W1:**\"More ablations of the different gates being open or closed in Table 1 \"?\n\n**A:**As you suggested, we conduct additional ablation experiments in our study to further explore the effects of different open or closed gates as shown in Table 1. In our subsequent work, we will also investigate the potential of utilizing adaptive parameters or dynamic sub-networks to control gate openings and close.\n\n|    Method | f | i | o | Accuracy |\n|-----------|---|---|---|-----|\n| RSNN-SCBAM | 0 | 1 | 0 | 91.3|\n| RSNN-SCBAM | 1 | 1 | 0 | 92.0|\n\n\nWe will incorporate all feedback and correct all typos and incorporate suggestions.\n\n"}]

Author rebuttal:
We thank the reviewers for their helpful feedback. We are inspired by the fact that they have found our motivation clear [Reviewer 9YjJ], reasonable [Reviewer TiEP] and our proposed approach simple but effective [Reviewer 9YjJ] and novel [Reviewer NNKj]. We appreciate that  [Reviewer TiEP] expresses agreement, stating, "This would enable the selective activation of historical information and the elimination of superfluous history during the training phase. This will provide the iterative calculation with a favorable initial state." We are delighted that everyone agrees our experimental evaluation is clear, and the proposed model is highly suitable for handling spatiotemporal data. We are also pleased that the reviewers found the paper to be well-presented and well-organized [Reviewer 9YjJ], and the ideas presented to be intriguing [Reviewer TiEP].

Here, we emphasize the contributions in this paper:

We introduce the Spiking Recurrent Neural Networks model with the Spiking Convolutional Block Attention Module (SRNN-SCBAM), aiming to leverage both spatial and temporal features of spatio-temporal patterns effectively. The SRNN-SCBAM model utilizes the spiking CBAM component to adaptively incorporate historical information from spatial and temporal channels, thereby benefiting from efficient memory retrieval and eliminating redundancy in historical data.
To validate the model's efficacy, extensive experiments are conducted on CIFAR10-DVS and DVS128-Gesture datasets. The ablation study and comparison results demonstrate that our proposed model achieves competitive performance among the existing RSNN models.

Thanks to the reviews’ suggestions on experimental description [Reviewer 9YjJ], experimental settings [Reviewer KuzU, Reviewer TiEP] and biological plausibility [Reviewer TiEP]. We have noticed the flaws in the origin description from the constructive feedback from the reviewers and have prepared an updated version of the manuscripts. Here we will roughly summarize the changes made. For a more detailed reply and explanation, please refer to the individual responses below.

**About the nolvety:** As [Reviewer TiEP] expresses agreement, stating, "Well-thought-through reuse of standard modeling and algorithmic components to solve the memory calling and history redundancy elimination in recurrent SNNs". We think our study is not the simple combinnation of the existing methods within the same paradigm.

**The comparison with SNNs with learnable threshold**We added the experimental results with the dynamic thresholds of MultiStepLIFNode and LIFNode trainable.

**The parameter settings**We explained all the parameter settings and the network structures.

**More visualization results**We added the visualization results in the one-page PDF rebuttal file.

We will incorporate all feedback and correct all typos and incorporate suggestions. We appreciate the reviewer’s thoughtful comments and insights regarding our article proposing the SRNN-SCBAM model. We would like to address the points raised and provide further clarification on certain aspects.

Thanks again for all reviewers, authors of paper 2180.


Meta-review: This paper got mixed recommendations, 3 accept and 1 reject. On the positive side, most of the reviewers agree that this paper presents clear motivation, novel method and model design and solid experiment results. On the negative side, one reviewer raised several questions on the detailed hyperparameter choice and their impact and also raised concerns on the novelty. The authors have provided detailed response to these questions. The AC has read all the reviews and responses from the authors. The authors have provided detailed answers to the questions that, to the AC, have addressed the questions. Therefore, the AC agree with most of the reviewers and recommend accept for this paper.
Decision: Accept (poster)