Title: Enhancing Adaptive History Reserving by Spiking Convolutional Block Attention Module in Recurrent Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Spiking neural networks (SNNs)

Recurrent spiking neural network (RSNN)

Dynamic Vision Sensor (DVS)

Spiking convolutional block attention module (SCBAM)
Abstract: Spiking neural networks (SNNs) serve as one type of efficient model to process spatio-temporal patterns in time series, such as the Address-Event Representation data collected from Dynamic Vision Sensor (DVS). Although convolutional SNNs have achieved remarkable performance on these AER datasets, benefiting from the predominant spatial feature extraction ability of convolutional structure, they ignore temporal features related to sequential time points. In this paper, we develop a recurrent spiking neural network (RSNN) model embedded with an advanced spiking convolutional block attention module (SCBAM) component to combine both spatial and temporal features of spatio-temporal patterns. It invokes the history information in spatial and temporal channels adaptively through SCBAM, which brings the advantages of efficient memory calling and history redundancy elimination. The performance of our model was evaluated in DVS128-Gesture dataset and other time-series datasets. The experimental results show that the proposed SRNN-SCBAM model makes better use of the history information in spatial and temporal dimensions with less memory space, and achieves higher accuracy compared to other models.

Enhancing Adaptive History Reserving by Spiking Convolutional Block Attention Module in Recurrent Neural Networks

Abstract

Spiking neural networks (SNNs) serve as one type of efficient model to process spatio-temporal patterns in time series, such as the Address-Event Representation data collected from Dynamic Vision Sensor (DVS). Although convolutional SNNs have achieved remarkable performance on these AER datasets, benefiting from the predominant spatial feature extraction ability of convolutional structure, they ignore temporal features related to sequential time points. In this paper, we develop a recurrent spiking neural network (RSNN) model embedded with an advanced spiking convolutional block attention module (SCBAM) component to combine both spatial and temporal features of spatio-temporal patterns. It invokes the history information in spatial and temporal channels adaptively through SCBAM, which brings the advantages of efficient memory calling and history redundancy elimination. The performance of our model was evaluated in DVS128-Gesture dataset and other time-series datasets. The experimental results show that the proposed SRNN-SCBAM model makes better use of the history information in spatial and temporal dimensions with less memory space, and achieves higher accuracy compared to other models.

1 Introduction

Neurons in the human brain maintain preceding activity traces on the molecular level [1, 2]. Especially, the eligibility traces are induced by the fading memory of spike events where the presynaptic neuron fired before the postsynapstic neuron [3]. Inspired by the loop behaviors of neurons, Recurrent spiking neural networks (RSNNs), have been proposed and proven to have potential in processing spatiotemporal patterns through learning from the historical context. Many of those existing approaches suggest memory preserving and extraction mechanisms play a crucial role in exploiting historical context underlying spatio-temporal dynamics. Although the current recurrent structures of RSNNs can capture the spatial and temporal variations at each level of the networks, the importance of different channels at different time points is ignored [4-6] which contributes the memory preservation and extraction can not focus on the small and important channels at specific time points.

Different recurrent structures of SNNs have been proposed to capture the memory. Like Long ShortTerm Memory (LSTM) cells in ANNs, the LSTM spiking networks [7] preserve valuable historical information by latching the gradients of a hidden state, proficient in processing time series data. To enhance the spatial feature extraction ability further of LSTM, its extension, named ConvLSTM, with

convolutional structures in both input-to-state and state-to-state transitions. Meanwhile, the Addressevent representation (AER) [8, 9] data is one typical spatio-temporal pattern and has contextual spatial and temporal cues. Considering the advantages of ConvLSTM over LSTM, we aim to introduce the ConvLSTM structure into SNNs, to construct the recurrent spiking networks with ConvLSTM structures.

The core benefit of SNNs with recurrent structures is the memory invoking along with sequential time points. However, most of the current recurrent structures of SNNs ignore the influence distinction of different historical contexts on spatial and temporal dimensions. The attention mechanism has been studied and achieved to tell where to focus and improve the representation of interests. The convolutional block attention module (CBAM) extracts the key channel and spatial information by emphasizing the significant features. We adopt the CBAM component to implement the history invoking adaptively in forgetting gates in RSNNs.Because CBAM can conduct Attention on the channel and spatial dimensions. Because CBAM can conduct Attention on the channel and spatial dimensions, we can adjust the weight of forgetting gate information. Therefore, we propose the spiking ConvLSTM with Spiking CBAM component to invoke the history information in spatial and temporal channels adaptively, which brings the advantages of efficient memory calling and history redundancy elimination.

The contributions of the paper are as follows:

The spiking recurrent neural networks model with the spiking convolutional block attention module (CBAM) component (SRNN-SCBAM) is proposed to combine both spatial and temporal features of spatio-temporal patterns.

The proposed model invokes the history information in spatial and temporal channels adaptively through spiking CBAM, which brings the advantages of efficient memory calling and history redundancy elimination.

The experiments are conducted on CIFAR10-DVS and DVS128-Gesture datasets. The ablation study and comparison results show that our model achieves competitive performance among current RSNNs models.

2 Related Work

Recurrent SNNs. Recurrent structures have manifested great potential in processing spatio-temporal patterns, benefiting from the information integration over time by multiple participant computations of each recurrent neuron. Current RSNNs contain two different kinds of network architectures according to the structural constraints. Firstly, based on the reservoir with randomly connected recurrent structures, The first-order reduced and controlled error algorithm could learn arbitrary firing patterns by stabilizing strong chaotic rate fluctuations in a network of excitatory and inhibitory neurons respecting Dale's law. Considering the inefficient and unstable connectivity of separating the excitatory and inhibitory populations, the framework of training the continuous-variable rate RSNNs with biophysical constraints and transferring the learned dynamics to spiking RSNNs, by introducing one parameter to establish the equivalence between rate and spiking RSNNs [10, 11].

Another category of the RSNNs is based on learnable recurrent structures. Learning from the training method of artificial neural networks with recurrent structures, the backpropagation through time (BPTT) is applied to the training of RSNNs [12, 13], which requires propagation of gradients backward in time by unfolding recurrent layers. In the view of biologically realistic [14, 15], e-prop optimally combines the local eligibility traces and top-down signals and formulates the gradient descent learning in RSNNs [16]. Besides, RSNN models can also be implemented on neuromorphic hardware, such as SpiNNaker [17] or Loihi chips [18], bringing computing efficiency combined with device properties.

Attention Mechanism. Human's visual system exploits the sequence of partial glimpses instead of processing the whole scene at once. Inspired by that property, the attention mechanism concentrates on the selective information in the input according to the sensitivity of the output to the variant inputs [19-21]. [22] propose temporal-wise attention SNN (TA-SNN) model to select relevant frames in the training stage and discard irrelevant frames in the inference stage by judging the significance of frames for the final decision. The convolutional block attention module (CBAM) module has been applied to CNNs to capture meaningful features along spatial and channel dimensions. The

f

t

Channel Attention

+

Spiking

Neurons

LIFNode

MaxPool

AvgPool

time steps

pooling

F

C

pooling

pooling

Channel Attention

f

t

Spatial Attention

f

t



h t-1

c t-1

i

t

Ct

Ot

h t

c t

t

h

Channel

Attention

Module

t

X

+

Spatial

Attention

Module

+

Channel Attention Module

Spatial Attention Module

(a)

(b)

(c)







intermediate feature map in CNNs is adaptively refined through CBAM at each convolutional block [23]. However, there is still a lack of combination attention modules to apply in RSNNs both in temporal and spatial dimensions.

3 Method

In this section, we show detailed information on the SRNN-SCBAM model. In order to solve the problem that most SNN recurrent structures ignore the difference in the impact of different historical backgrounds on the spatial and temporal dimensions, we propose that the spiking ConvLSTM with CBAMcomponents can adaptively call the historical information in the spatial and temporal channels. We can model DVS and other time series data, and complete the tasks of classification and prediction. For other data sets such as cifar10DVS, we additionally add convolution pooling to deepen the network structure.

3.1 The framework of the RSNN-CBAM

In the classification task for DVS128 Gesture, we used the network framework shown in Figure 1. The entire network of RSNN-CBAM consists of a layer of pooling layer, a layer of Spiking Convlstm with CBAM components, and a fully connected layer. Using SRNN, a PyTorch-based encoder-classifier model is implemented for neuromorphic data sets such as DVS128 Gesture. sort. Use pooling to reduce the dimensionality of data and remove redundant information, use Spiking Convlstm with CBAM components to extract spatial and channel features from neuromorphic data, and then send them to the classifier for classification.

3.2 Spiking ConvLSTM

We introduce ConvLSTM into SNN and use its time and space feature extraction ability to accurately process time series data. Spiking ConvLSTM is similar to Ann ConvLSTM, and it also overcomes the shortcomings of LSTM in processing three-dimensional information. And like the traditional

LSTM, the core concept of spiking ConvLSTM lies in the cell state, c t , and "gate" structure. The cell state is able to pass on relevant information in the process of sequence processing all the way. The "gate" structure learns which information to keep or forget during training.

Spiking ConvLSTM input is a 3D tensor, and the last two dimensions are spatial dimensions (Height and Weight). For the data at each timestep T, Spiking ConvLSTM replaces a part of the connection operation in LSTM with a convolution operation, which has a convolution structure in both input-tostate and state-to-state transitions, through the current input and the past state of local neighbors.

Spiking ConvLSTM is more suitable for processing dynamic spatio temporal patterns, such as gesture recognition with clear gesture moving trajectories but without preserving the details of fine-grained images with high-resolution pixels.

More specifically, given a set of spiking inputs x 1 , x 2 , , x T , the gates and states are characterized as follows:

where ⊙ represents the Hadamard product,* represents convolution operation, and σ 1 and σ 2 are spike activations that map the membrane potential of a neuron, to a spike if it exceeds the threshold value respectively. Also, w, and b, denote associated weights and biases. For the network, respectively. x t and h t -1 respectively correspond to the spiking input at the current moment and the hidden state at the previous moment.

The forget gate, denoted by f t , decides which information should be ignored. The input gate i t , which controls the information entering the cell, has another auxiliary input layer g t , modulated by another peak activation sigma 2 . Finally, the output of the spiking Convlstm is formed based on the output gate o t and the cell state c t .

3.3 Spiking CBAM

On the basis of spiking Convlstm, we use the attention mechanism to improve the forget gate. The spiking ConvLSTM with CBAM component is proposed to adaptively call historical information in space and time channels. The forget gate in spiking Convlstm is obtained by splicing the pulse input at the current moment and the hidden state at the previous moment, and then obtaining it through convolution and segmentation. For the CBAM component in RSNN-CBAM, the forget gate is used as input, and it will sequentially extract the one-dimensional channel attention map and the two-dimensional space attention map of the forget gate, thereby extracting the channel features and spatial features of the forget gate, and further, for space and time, the historical information in the channel is adjusted. The overall attention process can be summarized as:

Where ⊗ denotes element-wise multiplication. M c is the channel attention module, and M s is the spatial attention module. Mc and Ms sequentially perform channel feature extraction and spatial feature extraction on the forget gate to generate channel attention module M c ( f t ) and spatial attention module M s ( f ′ t ) . f ′′ t is the spiking CBAM output. The calculation process of each attention diagram is shown in the figure. The following describes the details of each attention module.

The channel attention component in Spiking CBAM uses balanced pooling and maximum pooling features to generate a one-dimensional channel attention map of forget gate through SNN-MLP. The channel attention is calculated as:

The output of RSNN with Spiking CBAM

The output of RSNN without Spiking CBAM

Time Step 0

Time Step 5

Time Step 10

Time Step 15

Time Step 19

Time Step 0

Time Step 10

Time Step 5

Time Step 15

Time Step 19

where σ represents the number of spiking activations, Average pooling, and maximum pooling feature map share SNN-MLP weight.

The spatial attention component in Spiking CBAM applies the average pooling and maximum pooling operations along the channel axis, performs the convolution operation on it, and sends it into the spiking activation function to generate the two-dimensional spatial attention feature map of the forget gate. The spatial attention is calculated as:

Where σ represents the number of spiking activations, and f 7 × 7 represents a convolution operation with the filter size of 7 × 7 [23].

Through the above two attention mechanisms, we obtained the one-dimensional channel attention map and two-dimensional spatial attention map of the forget gate, extracted the channel features and spatial features of the forget gate, and used Spiking CBAM adaptively recalls historical information in both spatial and temporal channels, and ablation studies and visualizations are performed to show the effectiveness of the attention component.

3.4 Surrogate Gradient Method

We mainly use the Leaky integrate-and-fire (LIF) node as the basic unit. The LIF model is also the most commonly used model to describe the dynamics of SNN neurons, which can be described by the following equation:

Where u ( t ) is the neuron membrane potential (membrane potential) at time t, τ is the time constant, and I ( t ) represents the presynaptic signal determined by the previous neuron's pulse activity or external stimulus and synaptic weights. enter. When the membrane potential u exceeds a specified threshold the neuron fires a pulse and resets its membrane potential to u reset .

In order to solve the problem that the loss function is not differentiable in the backpropagation in SNN, we use the surrogate gradient training method [24-26]. The substitution function is used to generate spikes in spiking Convlstm and spiking CBAM.

The principle of the gradient substitution method is to use the Heaviside step function in the forward propagation: y = θ ( x ) , While backpropagating, the substitution function is used: dy/dx = σ ( x ) ′ . Where σ ( x ) is a function similar in shape to θ ( x ) but smooth and continuous. We mainly use the

Time  Step  5

Time  Step  0

Time

Step

19

Time

Step

10

Time

Step

15

spike function of the gradient of the Gaussian error function (erf) during backpropagation. The corresponding original function is:

Where α is the parameter that controls the smoothness of the gradient during backpropagation. Backpropagation employs the gradient descent rule according to the derivatives:

Then, the derivatives of the gradient of the variables can be approximated by the above formulations. α is the derivatives of the gradient of the variables can be approximated by the above formulations.

4 Results

We conduct the following experiments to evaluate the performance of the proposed RSNN-CBAM models. We first introduce the experimental settings. Then, the classification accuracy of RSNNCBAM is recorded and compared to the current state-of-the-art models. The ablation study and visualization are followed to indicate the effectiveness of the attention component.

4.1 Experimental Setup

In order to further show the performance of the proposed method on dynamic spatio-temporal information processing, we evaluate the classification accuracy on the CIFAR10-DVS and DVS128 Gesture datasets. CIFAR10-DVS is an event-based version of the classical CIFAR10 dataset which also contains 10 classes with 10,000 event streams converted from images. Similar to CIFAR10-DVS, DVS128-Gesture is an event-based spiking dataset that contains 11 hand gestures from 29 subjects. Based on these spatio-temporal event streams [27, 28], the proposed method was evaluated for dynamic information processing and its adaptive history reserving ability.

Detailed parameter settings are listed in Table 1. The learning rate scheduler for RSNN-CBAM is set to be 1e-3 (selected from linear warm-up and decay or step decay). The learning epoch is 200 and the batch size is 32. The integration time constants τ of LIF neurons are set to be 4.0. α is the parameter that controls the smoothness of the gradient during backpropagation for surrogate gradient learning process, which is set to be 4.0 in this paper. u reset is the reset voltage of the neuron and set to be 0. The above content will be added to the revision.

ALL the following experiments were conducted on either an A100-SMX GPU. The network structure is composed of one layer of Convlstm neurons and three FCLIF1D layers. The results of the experiments are given in Table 2, as shown in this table, most of the accuracies among the various structures are increased based on the proposed method.

4.2 Ablation Study

In this section, the ablation study is conducted to validate the effectiveness of each component proposed in this paper.

The effectiveness of SCBAM in SRNN. The spiking CBAM component is employed in SRNN to implement selective memory preservation. As illustrated in Tab. 2, we conduct the ablation study on the DVS128 GESTURE dataset. Firstly, the Spiking ConvLSTM with spiking CBAM module achieves the highest test accuracy of 95% , which performs better than the one without any CBAM component over a 1 . 7% accuracy gap. It indicates the effectiveness of Spiking CBAM in improving the performance of Spiking ConvLSTM. Secondly, the proposed Spiking CBAM component achieves better classification accuracy than the accuracy drop of 2 . 6% original CBAM component. It suggests that the proposed spiking CBAM component captures more precise features than the original CBAM in the spiking ConvLSTM module because of its better temporal feature selection capability.

Explore the adaption of different gates to SCBAM. Weemploy the spiking CBAM on the forgetting gate to control the memory preserving and invoking for Spiking ConvLSTM. Here we further explore the performance of other gates in Spiking ConvLSTM to show the efficiency of the forgetting gate. As shown in the last five columns in Table. 2, the results suggest the priority of the forgetting gate compared with the other two input and output gates. The forgetting gate obtains the largest accuracy and exceeds nearly 10% accuracy than other gatings. It is reasonable that the forgetting gate with spiking CBAM controls the history information invoking and produces a marked effect on the effective computation of spiking ConvLSTM.

20

15

10

5

0

-5

-10

-15

-20

20

10

0

-10

-20

(a) The visualization of extract features of RSNNs with SCBAM

20

15

10

5

0

-5

-10

-15

-20

20

10

0

-10

-20

HIBHI3B

Eli

MH

31

El

HI

SB BB 31 KI

H BB

KB RBI

B

UHBBH R B MBH

EBB BBBB B HHH

I B B BB HI

Time

Step

0

Time

Step

5

Time

Step

15

Time

Step

10

Time

Step

15

Time

Step

10

Time

Step

0

Time

Step

5

HIBHI3B

Eli

MH

31

El

HI

SB BB 31 KI

H BB

KB RBI

B

UHBBH R B MBH

EBB BBBB B HHH

I B B BB HI

Time

Step

0

Time

Step

5

Time

Step

15

Time

Step

10

Time

Step

15

Time

Step

10

Time

Step

0

Time

Step

5

a.

The output

of RSNN with

Spiking CBAM

a.

The output

of RSNN with

b.

The output

of

RSNN without Spiking CBAM

The output

of

RSNN without Spiking CBAM

Figure 5: The figure above depicts visualizations of RSNN output tensors on the CIFAR10-DVS dataset. We selected time steps of 0, 5, 10, and 15 to compare the extracted features of RSNNs with and without SCBAM. It shows that the RSNN module with SCBAM captures information as anticipated, exhibiting strong sparsity across both temporal and spatial dimensions throughout the entire time steps while the RSNN module without SCBAM only extracts limited features which may lose a significant amount of crucial information.

4.3 Visualization analysis

To validate the effect of the SCBAM on the selection of the key features in the spatial and channel dimensions, we observe the feature map visualization of the RSNN-SCBAM model on the DVS128 GESTURE and CIFAR10-DVS datasets under the situation with and without SCBAM component.

DVS128 GESTURE feature map visualization results are illustrated in Fig. 4, the RSNN module with SCBAM captures more complementary and sparse features than the one without SCBAM. On one hand, the SCBAM possesses fine sparsity in the channel and spatial dimensions through the whole time steps. That benefits from the attention mechanism that captures significant features and eliminates redundant features. On the other hand, most of the features extracted by SCBAM are complementary to each other. Each feature map focuses on one specific area such as the hands and the shoulders. However, some feature maps of the RSNNs without SCBAM are quite similar because of the lack of filtering of redundant information. Based on the proposed SRNN-CBAM, the historical redundancy left in other spiking recurrent models could be discarded sharply, meanwhile, the proposed method maintains good performance by adaptive history reserving.

CIFAR10-DVS feature map visualization results are illustrated in Fig. 5, we visualize the extracted features at several time steps of the proposed RSNNs with and without SCBAM. It shows that the RSNN module with SCBAM captures information as anticipated, exhibiting strong sparsity across both temporal and spatial dimensions throughout the entire time steps while the RSNN module without SCBAM only extracts limited features which may lose a significant amount of crucial information.

We further conducted ablation experiments on dynamic thresholding during training on the DVSGesture dataset, and our RSNN-SCBAM model adopted two different dynamic thresholding methods in LIFNode and MultiStepLIFNode SNN. We set the thresholds of MultiStepLIFNode and LIFNode as trainable parameters and conduct experiments to verify their impact. The experimental results are shown in Table. 3 shown.

Table 4: Accuracy of additional time step values of 10, 15, and 25 on the DVSGesture dataset.

However, the current results do not show any significant improvement in dynamic thresholds. This threshold adjustment may make the model relatively complex, involving more parameters and

nonlinear operations. If the parameters or network structure of the model are not adjusted correctly, poor model performance may result. Furthermore, SNNs may require large amounts of training data and longer training times to achieve effective learning and optimization.

Regarding the variation in LSTM time steps, we conduct experiments using additional time step values of 10, 15, and 25 on the DVSGesture dataset. As shown in Table. 4, the results show that the time step of 20 is the optimal choice. This could be attributed to the nature of the DVSGesture data, where gestures are repeated by individuals at specific time intervals. The accuracies of the RSNN-CSAM increase first and then reduce as the growth of the time steps from 10 to 25. Different time steps capture varying temporal features, leading to differences in test results. Fewer time steps could cause the excessive concentration of information and then could not extract enough event features, while the larger time steps may extract superfluous information and could not capture the key information.

4.4 Comparison with other methods.

We conduct a performance comparison on the CIFAR10-DVS dataset to explore the efficiency of the proposed model. The compared models are as follows: The HOTS method proposed in [29] employs spatio-temporal features called temporal surfaces to create an event-based hierarchical pattern recognition architecture, we adopt three layers of time surface prototypes in the experiment. The article [30] represents the visual information captured by DVS as a stream of asynchronous pixel addresses (events) representing relative intensity changes at those locations, and uses a random ferns classifier to classify the stream of pixel events. The network structure consists of the following three modules: a motion detector, a bank of binary feature extractors, and a Random Ferns classifier. The number of ferns for CIFAR10-DVS is (50, 14), and there are 15,000 segment events with a patch size range between 12 and 20. The paper [31] focuses on visual attention, a specific feature of vision, and proposes two attention models for event-based vision. We adopt the p-N-DRAW recognition network, using pLSTM layers as encoders. The paper [32] introduces a novel event-based feature representation and a new machine learning architecture. And use the local memory unit to efficiently utilize past-time information and build a robust event-based representation. In the experiment, a temporal surface representation method with local memory was used to divide all event points within a spatial region into different fixed cells. Within each cell, all event points are accumulated to generate a temporal surface, obtaining features. Classification is achieved using an SVM (Support Vector Machine). The paper [33] proposes a neuron normalization technique to tune neural selectivity and develops a direct learning algorithm for deep SNNs.The network architecture employed in [33] is as follows: LIAF Blocks and Dense Blocks connected end-to-end.LIAF Block has a sequential structure, consisting of ConvLIAF, TD-layer-normalization, TD-activation (ReLU) and TD-AvgPooling, where TD refers to the time-distributed operation.

A detailed comparison of SNN based models for the benchmark is shown in Table 5, the proposed SRNN-CBAM achieves competitive performance among those models. Our proposed SRNN-CBAM method performs better than the LIF-NET model in [33]. Although the classification accuracy of SRNN-CBAM is lower than LIAF-Net [33], the adaptive memory invoking should be emphasized for the further application of spatiotemporal pattern processing.

5 Conclusion

Different from some classical spiking based models which aim at static pattern recognition, this paper utilized the dynamic spatio-temporal information processing ability of SNNs on event data. This paper proposes a brand new spiking recurrent neural network model with spiking convolutional block attention mechanism (SRNN-SCBAM), which can invoke the temporal history information in spatial and temporal features adaptively. It has prior advantages in efficient memory calling and irrelevant history information elimination. The experiment results show that the proposed model can achieve competitive performance among other state-of-the-art models, and the attention component captures meaningful features according to the sensitivity of the output to the input in the end-to-end training process.

At the same time, we believe that the proposed Spiking ConvLSTM is good at different time-series data and applications, such as object tracking, and it is worth noting that the proposed Spiking ConvLSTM needs the spike-based input, hence, when facing other time-series data, the input data should firstly be encoded into spike trains.

In addition, it is worth noting that we take a further step in processing efficient memory invoking and preserving of spiking neural networks, which has promising potential in processing and memorizing spatio-temporal patterns. This paper rethinks the role that attention plays in spiking recurrent neural networks, which not only provides a new way of constructing an efficient model for dynamic event stream data processing but more importantly that the proposed model could mimic the mechanism of potential information processing in neural circuits. Remember what should be remembered, and forget what should be forgotten.

References

R. Dolmetsch and D. H. Geschwind, 'The human brain in a dish: the promise of ipsc-derived neurons,' Cell , vol. 145, no. 6, pp. 831-834, 2011.

V. Kukekov, E. Laywell, O. Suslov, K. Davies, B. Scheffler, L. Thomas, T. O'brien, M. Kusakabe, and D. Steindler, 'Multipotent stem/progenitor cells with similar properties arise from two neurogenic regions of adult human brain,' Experimental neurology , vol. 156, no. 2, pp. 333-344, 1999.

K. Roy, A. Jaiswal, and P. Panda, 'Towards spike-based machine intelligence with neuromorphic computing,' Nature , vol. 575, no. 7784, pp. 607-617, 2019.

A. Shaban, S. S. Bezugam, and M. Suri, 'An adaptive threshold neuron for recurrent spiking neural networks with nanodevice hardware implementation,' Nature Communications , vol. 12, no. 1, pp. 1-11, 2021.

W. Zhang and P. Li, 'Spike-train level backpropagation for training deep recurrent spiking neural networks,' Advances in neural information processing systems , vol. 32, 2019.

M. Zhang, H. Qu, A. Belatreche, Y. Chen, and Z. Yi, 'A highly effective and robust membrane potential-driven supervised learning method for spiking neurons,' IEEE transactions on neural networks and learning systems , vol. 30, no. 1, pp. 123-137, 2018.

A. Lotfi Rezaabad and S. Vishwanath, 'Long short-term memory spiking networks and their applications,' in International Conference on Neuromorphic Systems 2020 , pp. 1-9, 2020.

N. Kasabov, K. Dhoble, N. Nuntalid, and G. Indiveri, 'Dynamic evolving spiking neural networks for on-line spatio-and spectro-temporal pattern recognition,' Neural Networks , vol. 41, pp. 188-201, 2013.

S.-C. Liu and T. Delbruck, 'Neuromorphic sensory systems,' Current opinion in neurobiology , vol. 20, no. 3, pp. 288-295, 2010.

R. Kim, Y. Li, and T. J. Sejnowski, 'Simple framework for constructing functional spiking recurrent neural networks,' Proceedings of the national academy of sciences , vol. 116, no. 45, pp. 22811-22820, 2019.

Q. Xu, Y. Li, J. Shen, J. K. Liu, H. Tang, and G. Pan, 'Constructing deep spiking neural networks from artificial neural networks with knowledge distillation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7886-7895, 2023.

J. Shen, J. K. Liu, and Y. Wang, 'Dynamic spatiotemporal pattern recognition with recurrent spiking neural network,' Neural Computation , vol. 33, no. 11, pp. 2971-2995, 2021.

Y. Guo, X. Huang, and Z. Ma, 'Direct learning-based deep spiking neural networks: a review,' Frontiers in Neuroscience , vol. 17, p. 1209795, 2023.

Y. Guo, W. Peng, Y. Chen, L. Zhang, X. Liu, X. Huang, and Z. Ma, 'Joint a-snn: Joint training of artificial and spiking neural networks via self-distillation and weight factorization,' Pattern Recognition , vol. 142, p. 109639, 2023.

T. Bu, J. Ding, Z. Yu, and T. Huang, 'Optimized potential initialization for low-latency spiking neural networks,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, pp. 11-20, 2022.

G. Bellec, F. Scherr, A. Subramoney, E. Hajek, D. Salaj, R. Legenstein, and W. Maass, 'A solution to the learning dilemma for recurrent networks of spiking neurons,' Nature communications , vol. 11, no. 1, pp. 1-15, 2020.

S. B. Furber, F. Galluppi, S. Temple, and L. A. Plana, 'The spinnaker project,' Proceedings of the IEEE , vol. 102, no. 5, pp. 652-665, 2014.

M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain, et al. , 'Loihi: A neuromorphic manycore processor with on-chip learning,' Ieee Micro , vol. 38, no. 1, pp. 82-99, 2018.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' Advances in neural information processing systems , vol. 30, 2017.

Q. Xu, Y. Li, J. Shen, P. Zhang, J. K. Liu, H. Tang, and G. Pan, 'Hierarchical spiking-based model for efficient image classification with enhanced feature extraction and encoding,' IEEE Transactions on Neural Networks and Learning Systems , 2022.

Q. Xu, J. Peng, J. Shen, H. Tang, and G. Pan, 'Deep covdensesnn: A hierarchical event-driven dynamic framework with spiking neurons in noisy environment,' Neural Networks , vol. 121, pp. 512-519, 2020.

M. Yao, H. Gao, G. Zhao, D. Wang, Y. Lin, Z. Yang, and G. Li, 'Temporal-wise attention spiking neural networks for event streams classification,' pp. 10221-10230, 2021.

S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, 'Cbam: Convolutional block attention module,' in Proceedings of the European conference on computer vision (ECCV) , pp. 3-19, 2018.

E. O. Neftci, H. Mostafa, and F. Zenke, 'Surrogate gradient learning in spiking neural networks,' 2019.

J. Shen, Q. Xu, J. K. Liu, Y. Wang, G. Pan, and H. Tang, 'Esl-snns: An evolutionary structure learning strategy for spiking neural networks,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 37, pp. 86-93, 2023.

T. Bu, J. Ding, Z. Hao, and Z. Yu, 'Rate gradient approximation attack threats deep spiking neural networks,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7896-7906, 2023.

M. Zhang, J. Wang, J. Wu, A. Belatreche, B. Amornpaisannon, Z. Zhang, V. P. K. Miriyala, H. Qu, Y. Chua, T. E. Carlson, et al. , 'Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks,' IEEE transactions on neural networks and learning systems , vol. 33, no. 5, pp. 1947-1958, 2021.

J. Shen, Y. Zhao, J. K. Liu, and Y. Wang, 'Hybridsnn: Combining bio-machine strengths by boosting adaptive spiking neural networks,' IEEE Transactions on Neural Networks and Learning Systems , vol. 34, no. 9, pp. 5841 - 5855, 2021.

X. Lagorce, G. Orchard, F. Galluppi, B. E. Shi, and R. B. Benosman, 'Hots: A hierarchy of event-based time-surfaces for pattern recognition,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 39, no. 7, pp. 1346-1359, 2017.

C. Shi, J. Li, Y . Wang, and G. Luo, 'Exploiting lightweight statistical learning for event-based vision processing,' IEEE Access , vol. 6, pp. 19396-19406, 2018.

M. Cannici, M. Ciccone, A. Romanoni, and M. Matteucci, 'Attention mechanisms for object recognition with event-based cameras,' in 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) , pp. 1127-1136, 2019.

A. Sironi, M. Brambilla, N. Bourdis, X. Lagorce, and R. Benosman, 'Hats: Histograms of averaged time surfaces for robust event-based object classification,' in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1731-1740, 2018.

Z. Wu, H. Zhang, Y. Lin, G. Li, M. Wang, and Y. Tang, 'Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing,' IEEE Transactions on Neural Networks and Learning Systems , vol. 33, no. 11, pp. 6249-6262, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe present study introduces a novel model of spiking recurrent neural networks, which incorporates a spike convolutional block attention mechanism. This model is referred to as SRNN-SCBAM. The primary objective is to effectively incorporate historical information into the spatial and temporal characteristics of spatiotemporal patterns, resulting in improved memory retrieval and elimination of redundant historical data. The efficacy of the proposed model in leveraging historical information and attaining high precision has been validated through experiments conducted on DVS128-Gesture datasets.\n\nSTRENGTHS:\nThe practical issue of invoking adaptive memory in spiking recurrent neural networks has been addressed in this paper. The primary objective is to incorporate a Spiking Convolutional Block Attention Module into the gating computation of Spiking ConvLSTM networks. This would enable the selective activation of historical information and the elimination of superfluous history during the training phase. This will provide the iterative calculation with a favorable initial state. The motivation is interesting and the idea of utilizing a spiking convolutional block attention mechanism to acquire historical data in gating computation appears intriguing.\n\nWEAKNESSES:\n1.More ablations of the different gates being open or closed in Table 1 are required because the influence of ‘i’ is unclear. And if there exist the adaptive parameters to control the possibility of gate open, could it be better for RSNN to achieve better performance? \n2.There is a typo in Figure 4.\n\nQUESTIONS:\n1.When encounter a new dataset, how the model works and how the architecture should be designed? I don't see discussions of this part in the paper.\n2.More ablations of the different gates being open or closed in Table 1 are required because the influence of ‘i’ is unclear. And if there exist the adaptive parameters to control the possibility of gate open, could it be better for RSNN to achieve better performance?\n\nLIMITATIONS:\nThe limitations are the same as the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

comments: SUMMARY:
This study proposed the spiking recurrent neural networks model with a spiking convolutional block attention module component (SRNN-SCBAM). The proposed model invokes the historic information in spatial and temporal channels adaptively through spiking CBAM, which brings the advantages of efficient memory calling and history redundancy elimination. The experimental results show that the proposed SRNN-SCBAM model makes better use of the historic information in spatial and temporal dimensions with less memory space, and achieves higher accuracy compared to other models.

STRENGTHS:
1. The learning algorithm for recurrent SNNs is simple and easy-to-understand.
2. Well-thought-through reuse of standard modeling and algorithmic components to solve the memory calling and history redundancy elimination in recurrent SNNs.

WEAKNESSES:
Although the motivation is reasonable and the proposed methods are simple and easy to implement, the experimental results could be supplemented to indicate the advantages of the proposed model.

QUESTIONS:
1. Table 2: What is the network structure in [23]? The network structures should be presented more clearly to show the components in the network models.
2. Why the performance of Spiking ConvLSTM with Spiking CBAM is quite higher than that with CBAM in Table 1?
3. What is the parameter of alpha used in Equation (7)?
4. It seems the proposed Spiking ConvLSTM is suitable for long-term time-series datasets due to its adaptive memory maintenance. Could the proposed model be applied to other time-series datasets?

LIMITATIONS:
Some details of training the Recurrent SNNs are missing in the paper (only mentioned by a few sentences). It would be better to also include those settings in the appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a Recurrent spiking neural network (RSNN). The essential component of the proposed RSNN is a spiking conv block attention module (SCBAM), which contains channel and spatial attention blocks. The proposed method is validated with classification tasks on CIFAR10-DVS and DVS128 gesture datasets.\n\nSTRENGTHS:\nN/A\n\nWEAKNESSES:\n-1. The writing of the submission should be dramatically improved. Many things are hard to follow. In lines 242-243: \"Although the ...is lower than LIAF-Net, ...\". In Table 2, [23] provides lower accuracy than the proposed method. Therefore, I cannot really trust the provided experimental results.\n\n-2. lines 114-115: \"Spiking Convlstm is more suitable for removing  facial expression data....\" Why? \n\n-3. SNN is different from CNN, which can process temporal information without any treatment, especially since many works have proved that, such as [R-1], [R-2]. The process of membrane potential release, accumulation, and triggering spiking is a natural temporal extractor. I do not think using RNN-based architecture is a meaningful way to go in the SNN domain. At the minimum, the authors should provide experimental results of the comparison with these methods, such as [R-1] and [R-2]. \n\n-4. How does the potential threshold of LIF impact the performance of the proposed methods?\n\n-5. How does the number of LSTM steps impact the performance of the proposed methods?\n\n[R-1]: Biologically Inspired Dynamic Thresholds for Spiking neural networks. \n\n[R-2]: Spiking Transformers for Event-based Single object Tracking\n\nQUESTIONS:\nPlease see the weaknesses section.\n\nLIMITATIONS:\nNo limitation is provided. \n\nFrom my perspective, the work follows the thoughts of ANN without an insightful understanding of SNN, especially the temporal processing power of SNNs. SNNs offer compelling temporal feature extraction capability without any special treatment, evidenced by many works already. \n\nIn addition, I did not see the novelty of the proposed approach from either ANN or SNN. Therefore, I think the submission is way below the bar of NeurIPS.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
This article proposes a spike recurrent neural network model with a spiking convolutional block attention mechanism, called SRNN-SCBAM. Its main idea is to adaptively call historical information in the spatio-temporal features of the spatio-temporal pattern, which has advantages in efficient memory calls and eliminating historical redundancy. The experiment on the DVS128 gesture dataset is conducted to verify the effectiveness of the model in utilizing adaptive historical information. The idea of the article is clear, the textual description is clear, and examples are provided to illustrate results.

STRENGTHS:
1.The paper is easy to read, and generally well written.
2.The proposed model of RSNN is simple but effective, the motivation is clear.
3.Experimental evaluations were conducted on two different neuromorphic datasets to demonstrate the performance improvement with the proposed SCBAM under different settings.

WEAKNESSES:
There is a lack of explanation for some parameter settings. In addition, there are some shortcomings in the experimental aspect of the paper. The experimental results mentioned in the article on the Cifar10-DVS generally can be supplemented with appropriate visualization results.

QUESTIONS:
1.The parameter setting about the time constant in the first paragraph of section 2.4 should be explained in detail.
2.In addition, there are some shortcomings in the experimental aspect of the paper. The visualization experimental results mentioned in the article on the Cifar10-DVS can be supplemented with appropriate experiments.

LIMITATIONS:
1.The parameter should be explained further.
2.More visualization results could be added.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05