Spiking Vision Transformer with Saccadic Attention
Authors: Shuai Wang
Malu Zhang
Dehao Zhang
Ammar Belatreche
Yichen Xiao
Yu Liang
Yimeng Shan
Qian Sun
Enqi Zhang
Yang Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Spiking Neural Networks
Spiking Transformer
Spike-driven Self-attention
Primary area: applications to neuroscience & cognitive science

Abstract: The combination of Spiking Neural Networks (SNNs) and Vision Transformers (ViTs) holds potential for achieving both energy efficiency and high performance, particularly suitable for edge vision applications. However, a significant performance gap still exists between SNN-based ViTs and their ANN counterparts. Here, we first analyze why SNN-based ViTs suffer from limited performance and identify a mismatch between the vanilla self-attention mechanism and spatio-temporal spike trains. This mismatch results in degraded spatial relevance and limited temporal interactions. To address these issues, we draw inspiration from biological saccadic attention mechanisms and introduce an innovative Saccadic Spike Self-Attention (SSSA) method. Specifically, in the spatial domain, SSSA employs a novel spike distribution-based method to effectively assess the relevance between Query and Key pairs in SNN-based ViTs. Temporally, SSSA employs a saccadic interaction module that dynamically focuses on selected visual areas at each timestep and significantly enhances whole scene understanding through temporal interactions.
Building on the SSSA mechanism, we develop a SNN-based Vision Transformer (SNN-ViT). Extensive experiments across various visual tasks demonstrate that SNN-ViT achieves state-of-the-art performance with linear computational complexity. The effectiveness and efficiency of the SNN-ViT highlight its potential for power-critical edge vision applications.

Published as a conference paper at ICLR 2025
SPIKING VISION TRANSFORMER WITH SACCADIC ATTENTION
Shuai Wang'. Malu Zhang': Zhang's Ammar Belatreche', Yichen Xiao', Yu Liang', Yimeng Shan", Qian Sun', Enqi Zhang', Yang Yang'
University of Electronic Science and Technology of China 2 Northumbria University, Liioning Technical University
ABSTRACT
The combination of Spiking Neural Networks (SNNs) and Vision Transform- ers (ViTs) holds potential for achieving both energy efficiency and high perfor- mance, particularly suitable for edge vision applications. However, a significant performance gap still exists between SNN-based ViTs and their ANN counter- parts. Here, we first analyze why SNN-based ViTs suffer from limited perfor- mance and identify mismatch between the vanilla self-attention mechanism and spatio-temporal spike trains. This mismatch results in degraded spatial relevance and limited temporal interactions. To address these issues, we draw inspiration from biological saceadic attention mechanisms and introduce an innovative Sac- cadic Spike Self-Attention (SSSA) method. Specifically, in the spatial domain, method to effectively assess the
SSSA employs a novel spike distribution- relevance between Query and Key pairs in SNN-based ViTs. Temporally, SSSA employs a saccadic interaction module that dynamically focuses on selected vi- sual areas at each timestep and significantly enhances whole scene understanding through temporal interactions. Building on the SSSA mechanism, we develop a SNN-based Vision Transformer (SNN-ViT). Extensive experiments across various visual tasks demonstrate that SNN-ViT ViT achieves ofte-of--he-arr performance with linear computational complexity. The effectiveness and efficiency of the SNN-ViT highlight its potential for power-critical edge vision applications.
INTRODUCTION
revolutionize the traditional computer vision field, achieving higher performance in many vision tasks such as image classification (Chen et al.] [022]] Han et al. 2023) and object detection (Fang et al., 2021c; Touvron et al.) 2021). However, ViTs always demand significant computational and memory resources, which greatly restricts their de- ployment in Consequently, the development of energy-efficient and high- figh-performance solutions remains sig- nificant area of research that necessitates further investigation (Cai et al. 2019 Han et al. 2020b).
Vision Transformers (ViTs) (Dosovitskiy 2020
constrained edge vision environments (Wu et al., 2022 Graham et al. 202T).
Spiking Neural Networks (SNNs), as the third generation of neural networks (Maass 1997: Ger-
stner & Kistler 2002 Izhikevich] 2003] Masquelier et al.) 2008), mimics biological information transmission mechanisms using discrete spikes as the medium for information exchange. Spiking neurons fire spikes only upon activation and remain silent at other times. This event-driven mech- anism (Caviglia al., 2014) promotes sparse synapse operations and avoids multiply-accumulate (MAC) operations, which significantly boost the energy efficiency of these models (Zhang et al.) 2023). However, the architectures of most SNN-based models still revolve around traditional struc- tures such as CNNs (Fang et al.) 2021b, Xing et al. 2019) and ResNets (Fang et al. 2021a Hu et al., 2024), which exhibit a significant performance gap compared VITs.
In recent years, numerous researchers have dedicated efforts to develop SNN-based ViT models. However, most studies (Zhou et al.) [202bb Wang et al. 2023b) retain energy-intensive MAC op- erations in self-attention computational paradigm and not fully take advantage of SNNs' energy
"Corresponding author; maluzhang@@estt..cuu..

---

Published as a conference paper at ICLR 2025
efficiency. Furthermore, these approaches still rely on the Dot-Product operation to measure the spatial relevance between Query (Q) and Key (K) pairs. However, they fail to account for whether the Dot-Products is well-suited to the binary spike characteristics of SNNs. Subsequently, inspired by Metaformer (Yu et al.] 2023), Spike-driven V2 (Yao et al., 2024b) introduces a MAC-free method, and Spikingkesforrer (Shi et al. (0024) combines ResNett-based architecture and self-attention com- putation paradigm to further reduce parameters. These methods ensure the high performance of SNN-based ViTs while achieving full spike-driven manner, offering significant energy savings. Nevertheless, these studies treat self-attentio computational paradigm merely as an efficient token mixer (Yu et all] 2022). without exploring an effective paradigm suited to spike trains. Furthermore, these methods primarily focus on spatial feature extraction, overlooking the temporal dynamics of SNNs. Consequently, exploring spiking self-attention paradigms tailored to the spatio-temporal characteristic of SNNs represents a potential area for improvement.
Biological vision dynamically captures and understands visual scenes through saccadic mecha- nisms (Melcher & Morrone 2003 Binda & Morrone 2018 Guadron et al., 2022). It focuses on specific visual areas at each moment and utilizes dynamic saceadic movements across the temporal
domain to achieve a contextual understanding of the entire visual scene (Hanning al.) 2023). Com- pared to vamilla self-attention mechanisms (Liu et al. 2021b), it offers higher energy and computa- tional efficiency. Additionally, the saccadic process involves intense temporal interactions (Idrees) eet al. 2020), which closely align with the unique temporal characteristics of SNNs. Therefore, we draw inspiration from the saceadic mechanisms design a Saccadic Spike Self-Attention (SSSA) method. The SSSA method adapts to the spatio-temporal characteristics of SNNs, enabling an effi- cient and effective comprehensive understanding of visual scenes. Based on this, we further develop a SNN-based Saccadic Vision Transformer. The summary contributions are as follows:
We thoroughly analyze the reasons for the mismatch between the vanilla self-attention mechanism and SNNs. In the spatial domain, the binary and sparse nature of spikes creates significant magnitude differences between Q and K in SSN-based ViTs, making it difficult for vanilla self-attention to assess spatial relevance. Additionally, vanilla self-attention is designed for ANNs and neglects the temporal interactions among timesteps in SNNs, limiting its ability to explore information in the temporal domain.
We propose a Saccadic Spike Self-Attention (SSSA) mechanism specifically designed for SNNs' spatio-temporal characteristics. In the spatial domain, SSSA introduces a novel spike distribution-basee method to measure relevance between Q and K pairs effectively. Temporally, SSSA introduces saccadic interaction module that dynamically focuses on selected visual areas and achieves a comprehensive understanding of the whole scene.
To further enhance the computational efficiency of SSSA, we introduce linear complexity version called SSSA-V2. It is mathematically linear scaling mapping to SSSA, preserv- ing all performance benefits. Additionally, SSSA-V2 successfully reduces computational complexity to a linear level and works in a fully event-driven manner.
Building on the proposed SSSA mechanisms, we develop a SNN-based Vision Trans- former (ST architecture. Extensive experiments are conducted on various visual tasks demonstrating that SNN-ViT achieves SOTA performance with linear computational complexity. It presents a promising approach for achieving both high-performance and energy-efficient visual solutions.
2 RELATED WORK
Vision Transformers: ViTs segment images into patches and apply self-attention [Vaswani] 2017
Kenton& & Toutanoval 2019 learn inter- -atch relationships, outperforming CNNs across multiple vision tasks (Mei et al. 2021] Bertasius et al. 2021 Guo et al. (2021). Nevertheless, ViTs face chal- lenges like high parameter counts (Liu et al. 202 and increased computational complexity pro- portional to token length (Pan et al. 2020 Liu et al. 2022). To enhance the computational efficiency of ViTs, many researchers (Jie& & Deng 2023; Li et al. (023) are focused on exploring lightweight improvement methods. For example, ViT (Graham al. 202I) incorporates convolutional el- ements to expedite processing, and MobileViT (Mehta & Rastegari (02II) combines lightweight MobileNet blocks with MHSA, achieving lightweight VIT's successfully. However, these enhance-
2

---

Published as a conference paper at ICLR 2025
ments still rely expensive MAC computations which are not suitable for resource- limited devices. This highlights the need for investigating more energy-efficient VIT solutions.
Spiking Neural Networks:
The event-driven mechanism enhances the energy efficiency of SNNs, offering a significant advantage for compute-constraing edge devices. With the introduction of
ANN-SNN (Cao et al. 2015; HHan al., 2020a Wu al. 2021 and direct training (Wu et al. Fang et al., [021b] iZhang et 2021 Wei et al. 2023) algorithm, the difficult associated
2018
with training higg--pefformance SNNs is significantly reduced. Based on these advanced learning algorithms, some research (Hu et al. 2021 |02T) Zheng et al. [021 Hu et al. 2024) propose deep residual
SNNs (Wang et al.] 2024; Shan et al., 2024) and others Yao al. 2023 Zhu et al. [024] [Shan al.
2024) contribute coulti-dimensiin
spike attention mechanisms, achieving competitive performance on many tasks (Zhang et a 2024). These improvements further enhance the application of SNNs in various visual tasks. However, despite rapid advancements, significant performance gap remains between these traditional deep SNN architectures and the latest ViTs.
Vision Transformers Meet Spiking Neural Networks: To explore high-performance and energy- efficient visual solutions, SNN-based ViTs (Zhou et al. 2023b; Wang et al. 2023a) have emerged. Spikformer (Zhou et al.) 2023b a pioneers a bpike-based self-attention computation, establishing the first spiking VIT. However, they still utilize expensive MAC operations and matrix multiplication in self-attention computation, which inefficient for binary spikes. Recently, Spike-driven Trans- former (Yao et al., 2024b) implements Hadamard product in the self-attention module for a fully spike-driven VIT. Additionally, SpikingResformer (Shi et al., (024) integrates Dual Spike self- attention module for improved performance and energy efficiency. However, these models primarily treat self-attention as a token mixer (Yu et al. 2022), without exploring an effective relevance com- putation suited to spike trains. Moreover, they also overlook the temporal dynamics of SNNs. (Zhang al., 2021| Bohte al.) 2000). Therefore, developing spike self-attention mechanisms tailored to the spatio-temporal characteristics of SNNs is essential for further advancements.
PROBLEM ANALYSIS IN SPIKING SELF-ATTENTION
In this section, we analyze the mismatches between vanilla self-attention mechanisms and SNNs in both the spatial and temporal domains. The detailed discussion is provided in the following sections.
3.1 DEGRADED SPATIAL RELEVANCE
Quarr
Quary
1200
10000
3000
459
4000
4000
200
13
25
Mapsiede
Magnitale
Magettado
Magssazi
Figure 1: Distribution of magnitudes for Q and K in ViTs within ANNs and SNNs on CIRAF100. In ANNs, Q and K exhibit similar magnitude distributions, whereas in SNNs. the magnitude differences between Q and K are pronounced.
The vanilla self-attention measures the spatial relevance between Q and K through Dot-Product operation. For given query Q, and key K, vector, the relevance between them are as follows:
D - KiyKeip
Dot-Product
(1)
D is the dimension of both vectors, Qij and K,y refer to the j-th elements of these vectors, respec- tively. Notably, the relevance based on Dot-Produut takes into account both the angle and magnitude of the vectors (Kim et al., 202I). When there is a significant difference in magnitude between vec- tors, the Pot-Product may not accurately measure their spatial relevance.
3

---

Published as a conference paper at ICLR 2025
In ANNs, continuous imput X is first normalized using layer normalization (Dosovitskiy) 2020) and then be processed through linear transformations We and WK to derive the matrices Q and K. This ensures that the magnitudes of Q and K are closely matched (Xu et al., 2019). preventing large variations between vectors. As shown in the left part of Fig the distribution between Q and K across various datasets remains nearly identical, allowing effective measuring of the spatial relevance for attention score in ANNs.
Due to the discrete activation characteristics of spiking neurons, the continuous distribution of the normalized membrane potentials in Q and K is disrupted. As shown in the right part of Fig. [ the magnitude of Q and K in SNNs shows significant variability, which leads to the failure of the Dot- Product in measuring spatial relevance. Moreover, despite Q and K following identical distributions, the sparsity of binary spikes significantly reduces their stability compared to ANNs. We provide a detailed analysis of this assertion in Appendix. A Therefore, developing more effective methods to measure the spatial relevance between spike trains could be viable approach to enhancing the performance of SNN-based ViTs.
(a) Vamilla Self-Autention (VSA)
(b) Spike Self- Soff-Attention (SSA)
(e)
Self-Attention (STSA)
Lizor LN
Coes: & BN
Cose BN
Corx & BN
K
e - #xx D
x
xc
K'
D(V
o
OON
N
AI
62
A1
Dematn
Σ(
MXD
Figure 2: Comparison of three self-attention computation paradigms. (a) VSA employs floating- point matrix multiplication to assess the spatial correlation between Q and K. resulting in a computa- tional complexity of O(N2D) D). (b) SSA lacks a dedicated temporal interaction module, maintaining the same complexity as VSA. (c) In contrast, STSA introduces global ppatial-temporal interactions, increasing the complexity to D2)
3.2 LIMITED TEMPORAL INTERACTION
As shown in Fig. 2 a), vanilla self-attention in ViT' operates independently of timesteps, thereby preventing the need for temporal interaction in self-attention designs. Conversely, SNNs rely on multiple timesteps to enrich their information representation capabilities (Fang et al., 2021b). How- ever, as shown in Fig. 2b), most spike self-attention mechanisms (Yao et al. 2024b) Zhou et al.) 2023b, Shi et al. 2024) lack dedicated modules for the temporal domain. The only temporal in- teraction in those methods is the accumulation of historical information by spiking neurons (LIF neurons), whose dynamics can be described as:
U(t + 1] H + 1]] S(( 1| @T + 1| Va,,,
(2)
(3)
Hit + 1] Vreset Sit + +]+ 1](1 S|t + 1)).
(4)
X|t + 1] denotes the spatial input current, while H H(t and UTt) represent the pre-synaptic and post- synaptic membrane potentials, respectively. The Heaviside function O(-) is employed for spike generation. If spike occurs (S(t + 1] = 1), (Tt)) resets to Vreset: otherwise, Ut + 1) decays with a time constant and feeds into H + 1]. However, due to the reset and decay mechanism, the residual membrane potential cannot sustain long-range dependencies, resulting a significant loss of historical information. To solve this problem. (Wang et al.) 2023b) proposes spatio-temporal spike self-attention method as shown in Fig. c). But this method has O(T2N2 computational complexity, significantly restricting the training efficiency of SNNs and increasing the complexity of deployment. Therefore, achieving more effective spatio-tempont interactions without increasing computational overhead remains a pressing challenge.
4

---

Published as a conference paper at ICLR 2025
4 SACCADIC SPIKING SELF-ATTENTION MECHANISM
We introduce a Saccadic Spiking Self-Attention (SSSA) method tailored for the spatio-tempont characteristic of SNNs. Spatially, SSSA enhances relevance measurement between spike vectors Q and K based on their distribution forms. Temporally, it incorporates dedicated saccadic interaction module for dynamic contextual comprehension of the visual scene. Additionally, we advance SSSA to version V2, which retains the high performance of SSSA and reduces computational complexity from ( to O(D).
(a) Spiking Saccudic Self-attention Block
(b) Dynamics Saccadic
Training
Inference
Cros-Eairspp
Punckp--
Park
Pesckji
Mask
Lineer
Mg
i Adiiion Openaiio
Y
Leumable Weights
Linear BN
LIF Neuron
Saccadic Neuron
Heavisde Farction
Stram
(c) Cross-cntropy (SSSA-VI)
Cross-entropy (SSSA-V2)
R2
(NN
00.0
O(D)
11
ctuu
KT
KT
Marrix Multiply
Matrix Multiply
Element-wise Multiply
Figure 3: Overview SSSA method. (a) SSSA consisting of two key components: cross-entropy relevance computation and saccadic spiking neurons. The latter outputs spike-driven decisions that mask in N-dimensional space. (b) training and inference process for saceadic spiking neurons. (c) the structure of the spatial relevance computation based on spike distribution. (d) the structure of SSSA-V V2 on spatial relevance computation, significantly reducing computational complexity.
4.1 SPATIAL RELEVANCE COMPUTATION FROM SPIKE DISTRIBUTION
To mitigate the issue of degraded spatial relevance caused by Dot-Product operations, we introduce a novel distribution-baat approach. It directly measures the relevance between two vectors using cross-entropy, unaffected by their magnitudes. Further details can be found in Appendix B
For patch E R in either Q or K, it can be treated as D-dimensional {0, 1} random spike train, where represents the spike firing rate. The cross-entropy between patches E Q and k E K
given by:
H (q. - ln log Pv (1 Pq) log (1 pk)],
(5)
where Py and denote the firing rates for vectors q and k, respectively. Since both q and are spike trains, our focus shifts the distribution of spikes rather than silent states. Consequently, we primarily consider the first term of Eq. 5 allowing us to simplify H k) to -Pq log Pa- Given feasible
that both log(x) and maintain the same monotonicity, substituting log(ær) with simplification that preserves the effectiveness of H(q, k), while avoiding nonlinear computations. Detailed analysis provided in Appendix B
Since cross-entropy Hq, k) measures negative relevance, we take its negative as our attention result. As result, the cross- atts-nttennion between Q and denoted as CroAtt (Q, K) -HQQ, K). can be further expressed as:
CroAtt (Q,K) C'K"T, e'= Q, K' I'=
ΣK. KK.QQ.ERTANNO
(6)
5

---

Published as a conference paper at ICLR 2025
As illustrated in Fig. 2(e), Q' and K" represent the sum of spikes across the dimension D. This approximation allows for more efficient parallel computation of spatial relevance between Q and K. By employing this distribution-basee vectors with non-standard distributions, thereby addressing the issue of degraded spatial relevance.
method, we more accurately assess the relevance between
4.2 SACCADIC TEMPORAL INTERACTION FOR ATTENTION
Biological saccadic mechanisms do not process all visual information at once. Instead, they progres- sively focus on key visual areas within a scene Guadron et al. (2022). This ensures that biological systems can efficiently achieve contextual understanding of the entire visual scene. Inspired by this mechanism, we have designed an effective temporal interaction module that incorporates two critical processes: salient patch selection and saccadic context comprehension. The first process selectively computes only a subset of patches at each timestep, while ignoring the others. It can significantly reduce the computational complexity of the SSSA method. This process can be described as:
Palch = Σ CroAu (Q, K), CroAtt (Q, K) E RRNNNN
CroAu(Q, K) represents the spatial relevance between patches in Q and K. By summing the rows of the (roAu(Q, K) matrix, the Patch represents the spatial salience of patches. Subsequently, the saccadic interaction module makes contextual understanding based on Patch. To ensure the asyn- chronous characteristies of SNNs, we aim to integrate the interaction process into spiking neurons. However, the significant historical forgetting caused by the resetting and decay mechanism of LIF neurons prevents efficient interaction. Therefore, introduce plug-and-pla saccadic spiking neuron, whose dynamic during training and inference phases can be described as follows:
H = Patch
{
{ [[[t] = Patch[t] St θ [11 - M M
Training
Inference
(8)
S
θ (H-WV Vin)
Here, H,S,Patch Patch E TTNN represents the data format for parallel training, encompassing the entire temporal dimension. M. is learnable lower triangular matrix that precisely regulates con- tributions from each timestep, facilitating efficient temporal interactions. Utilizing M to compute membrane potentials, saccadic spiking neurons avoid decay or resetting disruptions. As shown in 3 we depict the dynamic process of saccadic spiking neurons. During training, the membrane potential of saccadic spiking neurons is represented as Eg wir Patch[t), Whe E M However, all timesteps are processed simultaneously via matrix multiplication, which requires substantial com- putational resources. To maintain SNÑs' energy efficiency, we propose an asynchronous inference decoupling method. By incorporating the inverse of M into the threshold levels of the saccadic spiking neurons, we ensure temporal decoupling between H and S. The spike firing process depends solely on the current values of H [1] and Vn, Mr (6), eliminating the need for historical information. Thus, saccadic spiking neurons ensure the capability for asynchronous inference. Notably, the tem- poral complexity of saccadic spiking neurons is only O (T') significantly superior to the O (T The dynamics of saccadic spiking neurons are detailed in Appendix C
4.3 LINEAR COMPLEXITY AND SPIKE-DRIVEN COMPUTATION
Building on the aforementioned components, SSSA is specifically designed for the characteristics of SNNs. It enables more effective comprehensive understanding of the entire visual scene with lower time complexity. Its formulation is described as follows:
SSSA (2, K) = e (Mttat ... ,t] Vua) V = θ MM (o' x I- V, (9)
where represents a column vector [1, 1,.....11 1] with dimension N, facilitating the summation of row elements. However, as depicted in Fig. (].). SSSA includes integer multiplication operations within Q' x K', compromising the energy efficiency of the SNNs. Moreover, the quadratic com- plexity of Q' K' still indicates potential for optimization. Given that the matrix multiplications in do do not involve nonlinear operations, they allow for free association of matrices without al- tering the computational sequence. Consequently, to avoid the need for integer multiplication and further reduce computational complexity, we conduct an linear scaling mapping of Eq which can be described as follows:
SSSA (2, V) O ((M,, 2¹) (KTT xL) Vin) .V
(10)

---

Published as a conference paper at ICLR 2025
In SSSA-V2, computations begin with the calculation of Q' and K' based on Q and K, each with complexity of O(D). Then, instead of calculating Q' x SSSA-V2 treats (K*T x L) as a learnable scaling factor a, applied the threshold Ven of the saccadic neuron. Subsequently, M. X Q' as Patch[i [i] input into the saccadic neurons. During the inference process, Mwr can be integrated into the thresholds of saccadic neurons maintain fully spike-driven system.
H|t] -|t 2'tt). 5|1= (H (M [t]
Inference
(11)
Mathematically, SSSA V2 linear scaling mapping to SSSA, preserving all the advantages of SSSA while significantly reducing the need for integer multiplication operations. Additionally, SSSA-V2
achieves a linear spike self-attention mechanism with total computational complexity of O(2D+N), offering significant benefits in resource-comstrained environments.
SNN-BASED SACCADIC VISION TRANSFORMER
As illustrated Fig we introduce novel SNN-ViT based on the proposed SSSA method. primarily consists of GL-SPS blocks and SSSA-based transformer blocks. The following section will provide detailed descriptions these components.
3 WW
X
W
2C
H
W
X
H
X
8
16
16
Classification Head
SPS 19
SSSA-based
34S-19
SSSA-based Transformer
SPS
SSSA-based
8L-SPS
SSSA-based
Transformer
Transforner
Transtormer
Menory Shorizul
GL-SPS Blocks
SSSA-based Blocks
Spiking Saccadie Self-attention Method
LIF Neuron
U;
Seccades Neuron
Conv2D
Liccr
D<com
Dilated Com2D:
D-Cous
Cocs
U2
BN
Sacaades Amention
Linar
Linear Layer
IN
IN
Time Domaai
BNN
Batch Norm
Figure 4: The overall structure SNN-ViT, mainly consisting of GL-SPS blocks and SSSA-based transformer blocks. GL-SPS block combines dilated convolution and standard convolution at differ- ent scales to facilitate multi-scale feature extraction from images. The SSSA-based block, composed of SSSA methods and Linear layers, achieves lower computational complexity.
5.1 GL-SPS: GLOBAL--LOAA SPIKING PATCH SPLITTING MODULE
Currently, existing SPS methods primarily rely on shallow spiking convolution modules to capture local information, which prevents the effective extraction of multi-scale features. This limitation leads to degraded performance in processing wide-field image features. To address this issue, we design the Clobal-Local convolutional SPS block, described as follows:
GL-SPS (X(+)) = BN (Conv(X(t)) BN (DConv (x(+))
(12)
where X[{] is the result of convolution operation with stride of 2. Conv2d and D-Conv2d rep- resents standard and dilated convolution operations. BN(-) is Batch Normalization. The GL-SPS utilizes both Conv2d and D-Comv2d to extract features. Combining layers with different dilation rates effectively gathers context from various visual scales. Consequently, SNN-ViT VIT employs the GL-SPS method its embedding module, enhancing efficiency and scalability in feature extraction.

---

Published as a conference paper at ICLR 2025
5.2 OVERALL ARCHITECTURE
Building upon the pyramid structure (Liu et al., 2021b Yu et al. 2023), we propose a novel SNN- VTT that incorporates the GL-SPS block and the SSSA method. GL-SPS part encodes the input image through downsampling operation and various convolutions operation. Specifically, the down- sampling operation defined as convolution operation with a kemel size of 7 and stride of 2. The GL-SPS method follows the previous section. The whole block is defined as follows:
Up GL-SPS (T)
I TR TXC*XxW
(13)
Subsequently, Uo is inputted into the SSSA-based block, which consists of SSSA method and MLP Layer. To further reduce the computational complexity of the model, we adopt the SSSA-V2 version the paradigm for self-attention computation the architecture. Subsequently, the output from the SSSA-based Transformer blocks is fed into the Global Average Pooling (GAP) module, followed by a Classification Head (FCH) that generates the prediction These parts can be defined as:
U1 = Uo + BN(COV((SSSA(NN((()))
Uo E RTNND
(14)
U/ U1 + BN(Linear(SA((1)))
U1 E RTNND
(15)
Y (CH(GAP(SN(())))
(16)
where Y denotes the predicted outcome. For different types of datasets, we can integrate the GL-SPS component with varying numbers of SSSA decoding blocks. The details of the network architecture and the parameter count are presented Appendix
EXPERIMENTS
6.1 IMAGE CLASSIFICATION
ViT is evaluated on both static and neuromorphic datasets, including CIFARIO, CI- FARI00 (Krizhevsky al. 2009). ImageNet (Deng al.] 2009) and CIFARIO-DVS (Li et al. 2017]. Specifically, for ImageNet, the input image size is 3 x 224 x 224, with batch sizes of 128.
and training epoch is conducted over 310. Our experimental results are summarized Table and
Table 1: Summary of Network Architectures and Parameters across Vary Datasets
CIFARIO
CIFAR100
CIFAR 10-DVS
Method
Param.
Acc.
Param.
Acc.
Param.
Acc.
PLIF Fang et al.
2021b)
93.50
74.8
tdBN Zheng et al.
2021
93.2
67.8
DSpike (Li ct al.
202T
94.25
74.24
75,4
TET DDnng et al.
2022)
12.63
94.44
12.63
74.47
77.33
Spikformer (Zhou et al. Spikingformer (Zhou et 2023a
2023b)
9.32
95.51
9.32
78.21
2.57
80.9
9.32
95.81
9.32
78.21
2.57
81.3
Spike-driven (Yao et al.
2024b)
10.28
95.60
10.28
78.40
2.57
80.0
STSA (Wang et al. 2023b
1.99
79.9
SNN-ViT (Ours)
5.57
96.1
5.57
80.1
1.52
82.3
To facilitate a comprehensive comparison with similar works, we meticulously documented the per- formance of our SNN-ViT ViT across varying model sizes. In the CIFAR100 dataset, the direct training performance of SNN-ViT significantly surpasses the Spike-driven Transformer (Yao et al.) 2024b with fewer parameters. This underscores the high robustness and sensitivity of the SSSA strategy in smaller-scale tasks, closely mirroring biological cognitive processes. Notably, for neuromorphic datasets, the computational complexity SNN-ViT OTDD), while STSA (Wang et al. 2023b) has a complexity of O D2. Despite reducing computational complexity by three orders, SNN- ViT maintains SOTA performance. Furthermore, on the ImageNet-Il K dataset, we conduct a detailed comparison of SNN-ViT with other similar works. The comparison focuses computational com- plexity, parameter count, energy consumption, and accuracy. Experimental results demonstrate that SNN-ViT achieves SOTA performance under linear computational complexity.
8

---

Published as a conference paper at ICLR 2025
Table 2: Detailed comparison with other similar methods ImageNet-1K
Time
Param.
Energy (mJ)
Acc.
Method
Architecture
Complexity
Step
(M)
(%)
VIT
ViT-12-768
O(N²
86M
80.86
77.90
Dosovitskiy
2020
ViT-24-1024
D)
307M
283.36
76.51
Swin Transformer
Swin-T
((ND2)
29M
20.72
81.35
Liu et al. 2021b)
Swin-S
O(ND2)
51M
40.24
83.03
Fatten Transformer
FLatten-Swin-T
O(NDD)
29M
20.72
82.14
Han et al.
2023)
FLatten-Swin-S
O(N2D)
51M
40.24
83.52
Spikformer-8-384
O (N3D)
4
16.8M
7.73
70.24
Spikformer
Zhou 2023b
Spikformer-8-512
O (N3D)
4
29.7M
11.6
73.38
Spikformer-8-760
O (N
4
66.3M
21.5
74.81
SpikingResformer-S
O (N²D)
17.8M
3.37
75.95
SpikingResformer Shi et al. 2024
SpikingBResfrmerr-M
O (N
35.5M
5.46
77.24
SpikingRestormer--L
O (N=D)
4
60.4M
8.76
78.77
Spike-driven
Spike-driven-8-388
0 (ND)
16.8M
3.90
72.28
Yao et al 2024b
Spike-driven---52
0 (ND)
4
29.7M
4.50
74.57
Meta- Meta-Spikel SpikeFormer Former
Meta Meta-SSpikdFormee-3844
(ND2)
33.1M
32.8
74.10
Yao al. 2024a
Meta-SpiikeFormer-512
(ND2)
55.4M
52.4
79.70
V-T-V22-8-256
(
13.7M
14.28
74.66
(OurV)
SNN-ViT-8-384
(
30.4M
20.83
76.87
SNN-ViT-8-512
(
53.7M
35.75
80.23
6.2 REMOTE OBJECT DETECTION
Given the high adaptability of biological saccadic mechanisms to dynamic visual tasks, we aim to apply SNN- VTT to object detection tasks to demonstrate its advantages. As SNNs are often em- ployed resource-constraine edge computing scenarios, we select two remote sensing datasets: NWPU VHR-10 Cheng al. (201)) and SSDD (Wang al.] (019).. The NWPU VHR-10 dataset comprises very high-resolution (VHR) images across ten categories, including airplanes, ships, stor- age tanks, baseball diamonds, tennis courts, basketball courts, ground track fields, harbors, bridges, and vehicles. The SSDD dataset focuses on ship detection using Synthetic Aperture Radar (SAR)
titt
Figure 5: The detection results of SNN-VIT-YOLO on the NWPU-10 dataset are displayed in the first rows. SSSA attention heatmaps showcased in the second rows.
images. We build a SNN-VITI YOLO framework by incorporating the SNN-ViT as the backbone of the YOLO-v3 architecture. The structure details are shown in Appendix G As shown in Table

---

Published as a conference paper at ICLR 2025
to validate the superior performance of SSSA in dynamic visual tasks, present comparison with various deep ANN-based object detection methods, including YOLO-v3 (Liu et al. 2021a), RICNN (Cheng et al., 2016), and Faster RCNN (Fu et al. 2020). Additionally, we also compare our approach with EMS- YOLO (Su et al., 2023), the current SOTA results in the SNN fields. The results indicate that SNN-ViTE SAN--VT--OLA YOLO outperforms other methods on the two datasets. This demon- strates that SNN-ViT offers a viable approach for aviation and satellite image analysis in extreme environments.
Table 3: Performance comparison with ANNs and SNNs NWPU VOR-10 and SSDD.
Dataset
Method
Spike-driven
Timestep
mAP@0.5(%)
YOLO-V3
Liu et al.
20
X
87.3%
Net Chen et al. 2023
x
90.4%
4
86.5%
NWPU VHR-10 EMS-YOLO Su et al.
2023
8
87.9%
88.2% 4
SNN-VIT-YOLO (Ours)
89.4% 8
Faster R-CNN Fu ct al.
2020
X
85.3%
4
94.8%
EMS- Su et al 2023
SSDD
8
95.1 %
4
96.7%
SNN-VIT-YOL. (Ours)
97.0% 8
6.3 ABLATION STUDY
To verify the effectiveness of each component in the SiT, we perform a comprehensive ablation study in the CIFAR 100 dataset. The Spikformer (Zhou et al. 2023b) is selected as the baseline for comparison. Subsequently, we replace the corresponding modules in the base- line with SSSA blocks and GL-SPS blocks assess their impact on performance. As shown in Table replacing our SSSA method im- proves performance by approximately 2.65%, while reducing computational complexity to ((N2). Then we also verify the effectiveness of the GL-SPS blocks. As shown in Table GL-SPS blocks achieve a performance improvement of about 0.93% compared to baseline. This further demon- strates the enhanced compatibility of multi-scale feature maps with the saccadic process. Finally, approximately 3.15% performance improvement.
Table 4: Ablation Study
Param. (M)
Compl exity
Acc.
Model
(%)
Baseline
5.76
O(N2
76.95
+SSSA
5.52
O(D)
79.60 + (2.65)
+GL-SPS
5.81
O(N2D)
77.88 (0.93)
both
5.57
O(D)
80.1 + (3.15)
we replace both SSSA and GL-SPS, achieving Ablation studies validate that the SSSA indeed can significantly enhance performance, confirming its compatibility with apatio-temporal spike trains.
7 CONCLUSION
This work provides a detailed analysis of the mismatch between the vanilla ViT and spatio-temporal spike trains. This mismatch results in degraded spatial relevance and limited temporal interactions. Inspired by the biological saccadic attention mechanism, this work proposes SSSA method tailored to the SNNs. In the spatial dimension, SSSA employs a more efficient distribution- based approach to compute the spatial relevance between Query and Key in SNNs. In the temporal domain, SSSA utilizes a dedicated saccadic interaction module, calculating only subset of patches at each timestep to dynamically understand the context of the entire visual scene. Building on SSSA method, we de- velop a SNN- ViT structure, which achieves state-of-the-an art performance across various visual tasks with linear computational complexity. SNN-ViT effectively integrates advanced biological cogni- tive mechanisms with artificial intelligence techniques, providing a promising avenue for exploring high-performance, energy-efficient edge visual tasks.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGMENT
This work was supported in part by the National Natural Science Foundation of China under grants U20B2063, 62220106008, and 62106038, the Sichuan Science and Technology Program under Grant 2024NSFTD0034 and 2023YFG0259, the Open Research Fund of the State Key Laboratory of Brain-Machine Intelligence, Zhejiang University (Grant No.BM12400020).
REFERENCES
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space- time attention all you need for video understanding? In JCML, volume 2. pp. 4, 2021.
Paola Binda and Maria Concetta Morrone. Vision during saccadic eye movements. Annual review of vision science, 4(1):193-213, 2018.
Sander M Bohte, Joost N Kok, and Johannes A La Poutré. Spikeprop: backpropagation for networks of spiking neurons. In ESANN, volume 48, pp. 419-424. Bruges, 2000.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv: 1908. 09791, 2019.
Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113:54-60 66, 2015.
Stefano Caviglia, Maurizio Valle, and Chiara Bartolozzi. Asynchronous, event-driven readout of posfet devices for tactile sensing. In 2014 IEEE International EneeraationalSympssuum on Circuits and Systems (ISCAS), pp.
IEEE, 2014. doi: 10. (0.11099isaas221146666777
Chengeheng Chen, Weiming Zeng, Xiliang Zhang, and Yuhao Zhou. Cs net: a remote sensing de- tection network breaking the second-order limitation of transformers with recursive convolutions. IEEE Transactions on Geoscience and Remote Sensing, 2023.
Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. Proceedings of the IEEE/CVF international con- ference on computer vision, pp. 357-366, 2021.
Gong Cheng, Peicheng Zhou, and Junwei Han. Learning rotation-invariant convolutional neural net- works for object detection in vhr optical remote sensing images. IEEE transactions on geoscience and remote sensing, 54(12):7405-7411 2016.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, 18655-)1865-11 1883, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-254-2 leee, 2009.
Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re- re-weighting. arXiv preprint arXiv:2202. 1996, 2022.
Alexey Dosovitskiy, An image is worth 16x 16 words; Transformers for image recognition at scale. arXiv preprint arXiv:2010. 19929, 2020.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34:21056-2100 2021a.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. In- corporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2661-2671, 2021b.
11

---

Published as a conference paper at ICLR 2025
Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. Advances in Neural Information Processing Systems, 34:26183-26197, 2021c
Jiamei Fu, Xian Sun, Zhirui Wang, and Kun Fu. An anchor-free method based feature balanc- ing and refinement network for multiscale ship detection sar images. IEEE Transactions on Geoscience and Remote Sensing, 59(2):1331-144 2020.
Wulfram Gerstner and Werner M Kistler. Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press, 2002.
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer in convnet's clothing for faster inference. In Proceedings of the TEEE/CVF international conference on computer vision, pp. 12259-12269, 2021.
Leslie Guadron, John van Opstal, and Jeroen Goossens. Speed-accuracy tradeoff's influence the main sequence of saccadic eye movements. Scientific reports, 12(1):5262, 2022.
Meng-Hue Meng-HHo Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pct: Point cloud transformer. Computational Visual Media, 7:187-199, 2021.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13558-13567, 2020a.
Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vi- sion transformer using focused linear attention. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 5961-5971, 2023.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF conference on computer vision and pattemn recognition, pp. 1580-1589, 2020b.
Nina M Hanning, Antonio Fernández, and Marisa Carrasco. Dissociable roles of human frontal eye fields and early visual cortex in presaccadic attention. Nature communications, 14(1):5381, 2023.
Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, 34(8):5200-5205, 2021.
Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks toward deep residual learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Saad Idrees, Matthias P Baumann, Felix Franke, Thomas A Münch, and Ziad M Hafed. Perceptual saccadic suppression starts in the retina. Nature communications, 11(1):1977, 2020.
Eugene M Izhikevich. Simple model of spiking neurons. IEEE Transactions on neural networks, 14(6):1599-1522 2003.
Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation vision transformer. the AAAI conference on artificial intelligence, volume 37, pp. 1060-1068, 2023. In Proceedings the
Jacob Devlin Ming-Wor Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of nLL.L-HLTT volume 1, pp. 2. Minneapolis, Minnesota, 2019.
Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant self-attention. In International Conference on Machine Learning, pp. 5562-5571, PMLR, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifarl0-dvs: an event-stream dataset for object classification. Frontiers in neuroscience, 11:309, 2017.
12

---

Published as a conference paper at ICLR 2025
Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Rethinking vision transformers for mobilenet size and speed. Proceedings of the IEEE/CVF International Conference Computer Viston, pp.
2023.
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng. Yongqing Hai, and Shi Gu. Differ- entiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 14:23426-234333 2021.
Jing Liu, Zizheng Pan, Haoyu He, Jianfei Cai, and Bohan Zhuang. Ecoformer: Energy-saving attention with linear complexity. Advances in Neural Information Processing Systems, 35:10295- 10308, 2022.
Yanfeng Liu, Qiang Li, Yuan Yuan, Qian Du, and Qi Wang. Abnet: Adaptive balanced network for multiscale object detection in remote sensing imagery. IEEE transactions on geoscience and remote sensing, 60:1-14, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021b.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9):1659-1771, 1997.
Timothée Masquelier, Rudy Guyonneau, and Simon Thorpe. Spike timing dependent plasticity finds the start of repeating patterns in continuous spike trains. PloS one, 3(1):e1377, 2008
Lane McIntosh, Niru Maheswaranathan, Aran Nayebi, Surya Ganguli, and Stephen Baccus. Deep learning models of the retinal response to natural scenes, Advances in neural information pro- cessing systems, 29, 2016.
Sachin Mehta and Mohammad Rastegari. Mobilevit: light- general friendly vision transformer. arXiv preprint 10.02,2, 2021.
general-purpose, purpose, and mobile-
Yiqun Mei, Yuchen Fan, and Yuqian Zhou. Image super-resolution with non-local sparse attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 3517-3526, 2021.
David Melcher and M Concetta Morrone. Spatiotopic temporal integration of visual motion across saccadic eye movements. Nature neuroscience, 6(8):877-88 2003.
Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. X-linear attention networks for image captioning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp- 10971-10980, 2020.
Yimeng Shan, Malu Zhang, Rui-jie Zhu, Xuerui Qiu, Jason K Eshraghian, and Haicheng Qu. Ad- vancing spiking neural networks towards multiscale spatiotemporal interaction learning. arXiv preprint arXiv:2405.1 13672, 2024.
Xinyu Shi, Zecheng Hao, and Zhaofei Yu. Spikingresformer: Bridging resnet and vision transformer in spiking neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5610-5619, 2024.
Qiaoyi Su, Yuhong Chou, Yifan Hu, Jianing Li, Shijie Mei, Ziyang Zhang, and Guoqi Li. Deep directly-trained spiking neural networks for object detection. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 6555-6565. 2023.
Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, and Surya Ganguli. From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction. Advances in neural information processing systems, 32, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data- image transformers & distillation through attention. In International conference machine learning, pp. 10347-10357. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2025
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Qingyu Wang, Tielin Zhang, Minglun Han, Yi Wang. Duzhen Zhang, and Bo Xu. Complex dynamic neurons improved spiking transformer network for efficient automatic speech recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 102-109, 2023a.
Shuai Wang. Dehao Zhang, Ammar Belatreche, Yichen Xiao, Hongyu Qing. Wenjie We, Malu Zhang, and Yang Yang. Ternary spike- based neuromorphic signal processing system. arXiv preprint arXiv:2407.0533 2024.
Yuanyuan Wang, Chao Wang, Hong Zhang, Yingbo Dong, and Sisi Wei. A sar dataset ship detection for deep learning under complex backgrounds. remote sensing, 11(7):765, 2019.
Yuchen Wang, Kexin Shi, Chengzhuo Lu, Yuguo Liu, Malu Zhang, and Hong Qu Apatial-tempone self-attention for asynchronous spiking neural networks. In IJCAI, pp. 3085-3093, 2023b.
Wenjie Wei, Malu Zhang, Hong Qu, Ammar Belatreche, Jian Zhang, and Hong Chen. Temporal- coded spiking neural networks with dynamic firing threshold: Learning with event-driven back- propagation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp- 10552-10562, 2023.
Jibin Wu, Chenglin Xu, Xiao Han, Daquan Zhou, Malu Zhang, Haizhou Li, and Kay Chen Tan. Progressive tandem learning for pattern recognition with deep spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1 44(11):7824-7840 2021.
Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European conference on computer vision, pp. 68-85. Springer, 2022.
Yujie Wu, Lei Deng, Guoqi Li. Jun Zhu, and Luping Shi. Spatio-temporey backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Fu Xing, Ye Yuan, Hong Huo, and Tao Fang. Homeostasis based enn-to-smn conversion of inception and residual architectures. In Neural Information Processing: 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, December 12-15, 2019, Proceedings, Part III 26, pp- 173-184. Springer, 2019.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. Advances neural information processing systems, 32, 2019.
Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. IEEE transactions on pattern analysis and machine intel- ligence, 45(8):9393-9410 2023.
Man Yao, Jiakui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. arXiv preprint arXiv:2404.03360 2024a.
Man Yao, Jiakui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer. Advances in neural information processing systems, 36, 2024b.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10819-1088 10829, 2022.
Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xin- chao Wang. Metaformer baselines for vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Dehao Zhang, Shuai Wang, Ammar Belatreche, Wenjie Wei, Yichen Xiao, Haorui Zheng, Zijian Zhou, Malu Zhang, and Yang Yang. Spike-based neuromorphic model for sound source localiza- tion. In The Thirth-eighth Annual Conference on Neural Information Processing Systems, 2024.
14

---

Published as a conference paper at ICLR 2025
Jilin Zhang, Dexuan Huo, Jian Zhang, Chunqi Qian, Qi Liu, Liyang Pan, Zhihua Wang. Ning Qiao, Kea-Tiong Tang, and Hong Chen. 22.6 anp-i: A 28nm 1.5 pj/sop asynchronous spiking neural network processor enabling sub-o. pj/sample on-chip learning for edge-ai applications. In 2023 JEEE International Solid-State Circuits Conference (ISSCC), pp. 21-23. IEEE, 2023.
Malu Zhang, Jiadong Wang, Jibin Wu, Ammar Belatreche, Burin Amornpaisannon. Zhixuan Zhang, Venkata Pavan Kumar Miriyala, Hong Qu, Yansong Chua, Trevor Carlson, et al. Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks. IEEE transactions on neural networks and learning systems, 33(5): 2021.
Jing Zhao, Ruiqin Xiong, Jiyu Xie, Boxin Shi, Zhaofei Yu, Wen Gao, and Tiejun Huang. Recon- structing clear image for high-speed motion scene with retina-inspired spike camera. IEEE Transactions on Computational Imaging, 8:12-27, 2021.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 1062-11070, 2021.
Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Zhengyu Ma, Han Zhang, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual learning for transformer-based spiking neural net- work. arXiv preprint ar3iv:2304. /955, 2023a.
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, YAN Shuicheng, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. In The Eleventh Interna- tional Conference on Learning Representations, 2023b.
Rui-Jie Zhu, Malu Zhang, Qihang Zhao, Haoyu Deng, Yule Duan, and Liang- Deng. Teja- snn: Temporal-channee joint attention for spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2024.
15

---

Published as a conference paper at ICLR 2025
A LIMITATIONS OF DOT-PRODUCT FOR SPIKE TRAINS
The Dot-Product is the operation measure relevance between two vectors and v in an n- dimensional space, which is defined as:
u.v=
0:4;; [v|] cos
(17)
where on and b2 are the components of vectors u and v respectively, ||u]] and ||v|] denote the mag- nitudes (norms) of the vectors, and is the angle between them. This expression clearly illustrates that the Dot-Product influenced by both the magnitudes of the vectors and the cosine the angle between them. Variations in either magnitude or angle will affect the result of the Dot-Product, thus affecting the measure of relevance between the vectors.
Problem: If the Q and K controlled to be similar distributions in SNNs, would the effec- tiveness of the Dot-Product still be influenced by magnitude differences?
Analysis: To deepen our investigation, we present the following mathematical assumptions: assum- ing query Q and the key vector K in SNNs are independent and share the same firing rate. Then we examine the qll Let = (I1, ..,...,, .D) E {0,1}" represent q or k, where each element
takes the value I with probability p. The square of the magnitude follows binomial distribution:
D Σ
D
||0|| =
x
=
Σ
B(D,P)
(18)
1=1
1=1
Its probability given by:
- p(] (D) P))))) k= 0,,,,,
(19)
We randomly select q and le from this distribution and calculate their magnitude ratio R Without considering the case when ||||| = 0, the calculation proceeds as follows:
|g]
[6
P(R=)) - P(R = r²)
ΣΣ1 7 =
((X=x)P(C =1)
(20)
k=0 1=1
= و²)
11 pht - -p)
k
O= t=1
where 1 k =r² is the indicator function, which equals when k r2 and 0 otherwise.
Given the complexity of this distribution, we employ experimental simulation for approximation. Referencing the data shown in Fig. [ set p 0.15 and D 128. As the Dot-Product opera- tion is symmetric, we adjust our calculation to ensure that the numerator is always greater than or equal to the denominator, enhancing the clarity of our visualization. Specifically, we compute [g]] [[]]]
when ||사|| >> ||A=]] and otherwise. The simulation results shown in Fig 6(b) Clearly, the distribution of ||al] is notably disordered. For comparison, we conduct the same assumptions and simulations in the self-attention module of ANN. Let (æ), 22, #p) E R represent g k.
|
[q]l
[A]l
Then its magnitude is given by | = VE x/. Similarly, we randomly select a q and a k and simulate the distribution of their magnitude ratio Rt' Based on the data in Fig. I approximate each element W; as independently normally distributed with æ; ~ N (35,, 10). The results are shown in Fig 6()) By calculating the variance of 19ll it is found to be approximately 0.2322 in SNNs [k]|
and only around 0.00844 in ANNs. This indicates significant magnitude fluctuations in SNNs, vealing high degree of instability. As result, the efficiency and effectiveness the Dot-Product computation negatively impacted.
16

---

Published as a conference paper at ICLR 2025
Dot: Product
Magnitude Rate in SNNs
Magnitude Rate ANNS
1400
12001
L3
Fraquency
1000
0,2322
Proquency
0.0084
L4
600
600
400
400
200
2.0
2.0
(a)
(b)
(c)
Figure 6: (a) The impact of varying magnitudes on the results of Dot-Products. (b) and (c) The
distribution of magnitude entropy for || in ANNs and SNNs.
||ke
B CROSS-ENTROPY FOR BETTER RELEVANCE COMPUTATION
We calculate the relevance in spatial dimensions separately for the Q and K vectors at different moments. The vectors q E Q and k E K, and Pg and рк represents the spike firing rate for them. Here, introduce cross-entropy method to more effectively compute the relevance between and l vectors:
H (g. k) lpg log Pk (1 P4) log (1 -x)]--
(21)
The former term quantifies the degree of relevance when predictions are positive, while the latter reflects the relevance of negative predictions. When using cross-entropy as a measure of relevance, spike trains are first normalized and then transformed into probability distributions. Spike trains typically comprise only two states: spike and silence. Therefore, the probability distribution primar- ily reflects the spike firing rate. This measurement approach focuses on comparing the differences between two probability distributions, disregarding their magnitudes. In summary, cross-entropy an effective distribution-baseed tool for assessing the relevance of spike trains, allowing for more precise comparisons and evaluations of similarities and differences between Q and K.
Approximation Methodology: Although cross-entropy is an effective method for measuring rele- vance, a comprehensive analysis of spike states and silent periods may reduce the system's is sensitiv- ity. This is primarily because excessive focus inactive silent periods can obscure critical informa- tion present during active spike periods when dealing with sparse spike trains. Given our focus on the spike states rather than silent periods in subsequent analyses, we can neglect the (1-P4) Ppy) log(1 Pt) component. Therefore, H (u. k)) can be simplified as follows:
H (a, b) ~ P4 log
(22)
where Pa and Po respectively represent the spike firing rates. However, the log(.) function introduces non-linear operations that compromise the energy efficiency of SNNs. To address this, we propose a further approximation and simplification.
As described in the previous section, within the Transformer blocks of the SNNs, the spike firing rate the Q vector and K vector primarily range from 10% to 20%. Consequently, we perform a Taylor expansion of log(x) at 0.15. This can be expressed as:
log(x) = log (0) (0.15) +... log(") (0.15) n! (x 0.15)"
(23)
Here, (o) function denotes the result of the n-th order derivative of the log(.) function. Given that is essentially between 0.1 and 0.2, The terms (P2- 0.15)² and higher-order terms very small, which can be neglected. Consequently, H(A, B) can consider only the first term of the expansion:
log(z) x log (0.15)) + log (1) (0.15) (x 0.15) -
(24)
1!
In the training process of SNNs, since io and b can be learned weights and biases, we to replace log(x) to simplify computations. Although this may introduce slight errors, it avoids nonlinear operations and significantly enhances the network's 's energy efficiency.
17

---

Published as a conference paper at ICLR 2025
C SACCADIC TEMPORAL INTERACTION
Saccadic mechanism biologic Vision: Numerous neuroscience findings(Melcher & Morrone 2003 Binda & Morrone 2018 Guadron al. 2022) confirm that the eyes do not acquire all details of scene simultaneously. Instead, attention is focused on specific regions of interest (ROIs) through series of rapid saccadic called saccades. Each saccade lasts for Very brief period-typically only tens of milliseconds allowing the retina's 's high-resee nigh-resolution area to sequentially align with different visual targets. This dynamic saccadic mechanism enables the visual system process information efficiently by avoiding redundant processing of the entire visual scene.
Other similar works inspired by visual mechanisms: Zhao al. 2021 introduces model uti- lizing cetina-inspired spiking camera to enhance image clarity in high-speed motion scenarios. McIntosh et al. (2016) explores how deep convolutional neural networks can model the retina's response to natural scenes. [Tanaka al. (2019) discusses the use of deep learning models to un- derstand the computational mechanisms the retina. These advanced features of biological vision effectively inform the rational design of deep neural networks, promoting the efficient integration of biological and machine intelligence.
Leaky Integrate-and-Fing (LIF) neuron model: In the LIF models, resetting and decay mech- anisms significantly impair the neuron's ability to retain long-term historical information. The model's dynamics described by the differential equation:
dV
Ti dt = (V((t) Vrea) + RI((),
(25)
where (t) is the membrane potential, Vist is the resting potential, Tm is the membrane time con- stant which influences decay rate, R is the membrane resistance, and [tt) is the input current. This equation illustrates how the membrane potential responds to input currents and decays towards Vrest- When the membrane potential V (tt) reaches the threshold VzA, the neuron fires and resets the poten- tial to Vieset- This resetting process can be mathematically described as:
v(t+)
Vieset
if V(t) Vih:
(26)
where ++ is the time immediately following the spikes. This resetting not only disrupts the continuity of Vtt) but also eliminates all accumulated potential exceeding the threshold. Moreover, in the absence of input (I(t) 0), the decay mechanism mercilessly forces the membrane potential to exponentially converge to the resting potential Vress, following the equation:
V(I) = Vrest + (Vo- - Vrest) Tm
(27)
where Vo the initial potential. This decay process gradually diminishes the stored information in the neuron, causing the accumulated potential to disappear quickly over time. It severely limits the neuron's 's ability to maintain historical information. To address this issue, we specifically designed saccadic spiking neurons without decay and reset mechanisms. The training and inference processes are described as follows.
D SACCADIC NEURONS
the training process of SNN-VIT, the information from all timesteps is inputted in parallel. During this phase, the dynamics of the saccadic spiking neuron can be described as follows:
Training Phase:
H M.SS
Wi1
M
(28)
S
Wnl
Wnn
These neurons utilize a learnable lower triangular matrix M to integrate information across all timesteps without any loss of historical data, enhancing long-term memory and processing capa- bilities. As M is a lower triangular matrix. it naturally associates earlier inputs with current states, facilitating a comprehensive understanding of the whole visual scene over time. This method ensures that contributions from each timestep precisely modulated and accumulated through matrix M.
18

---

Published as a conference paper at ICLR 2025
This approach not only prevents information loss due to decay and resets but also allows neurons to utilize all historical data more effectively for decision decision-making. Such capability markedly improves their capacity to understand the context of entire image scenes.
Inference Phase: During the training phase, all information is inputted into the network simul- taneously, enabling efficient interaction through direct matrix multiplication. However, this paral- lel processing approach incurs significant resource expenditure, which is unfriendly to resource- constrained edge devices. Therefore, maintain the energy efficiency and asynchronous processing advantages of SNNs, we propose an asynchronous decouple method that only computes the input at the current timestep. The dynamics of the inference process can be described as follows:
[1 S[t] S| = θ [H[ M V (n
(29)
During the inference process, the threshold Vth varies at each moment, thus Vaa E IR.. To ensure the existence of the inverse M for the matrix Mw, certain constraints must be imposed M It can be described as follows:
det(Mw) mi1 m22 ... X Mnn +
(30)
By incorporating the inverse of M into the threshold of the saccadic spiking neurons, we ensure temporal decoupling between H and Specifically, H[]] only requires input from S[t), facilitating asynchronous inference. Additionally, dynamic thresholds at each moment enrich the dynamical properties of the spiking neurons. This approach effectively highlights the spatio-temporal attributes of SNNs and ensures efficient performance of vision tasks resource-constrained edge devices.
E ABLATION STUDIES ON SSSA
We add ablation studies on the two key components of the SSSA module: (1) Replacing Distribution- Based Spatial Similarity Computation with traditional Dot Product (DP) similarity; (2) Replacing saccadic neurons with LIF neurons. Finally, we also compare the performance of versions V1 and V2. Experiments are performed on the CIFAR100 dataset, and the results are presented in the following Table
Table 5: Ablation Studies on SSSA.
Model
Param (M)
Complexity
Acc (%)
SSA
5.76M
O(N
76.95
SSSA+DP
5,52M
O(N2D)
77.12
SSSA+LIF
5.52M
O(D)
78.84
SSSA-VI
5.52M
O(N2)
79.71
SSSA-V2
5.52M
O(D)
79.60
The SSSA+DP shows almost no performance improvement compared to the baseline (Zhou et al.) 2023b). This outcome underscores that effective spatial similarity computation is the foundation for subsequent saccadic interactions. Then, substituting saccadic neurons with LIF neurons led to an approximate 0.8% decrease in performance relative SSSA. This demonstrates that saccadic inter- actions can indeed enhance performance. Finally, while there is virtually no performance disparity between V1 and V2, the computational complexity of V2 only O (D). In summary, our SSSA-V2 module achieves an optimal trade-off between computational complexity and performance.
F EXPERIMENT SETTING FOR IMAGE CLASSIFICATION
On the ImageNet- 1K classification benchmark, we propose an architecture according to Spike- driven V2 (Yao al. 2024a). As illustrated in Table 6 model introduces the GL-SPS part and SSSA block, which respectively replace the Patch Embedding block and self-attention compu- tation blocks. The architecture of SNN-ViTs primarily consists of four stages. Each stage includes
19

---

Published as a conference paper at ICLR 2025
GL-SPS encoding operations and SSSA module, facilitating efficient and precise visual information processing. Specifically, in the initial stage, the imput sequence E x3x1Xx is processed through a GL-SPS layer, which encodes it into X E xxx " Then, the encoded images are input into spiking saccadic self-attention block to enhance feature extraction. This block comprises a SSSA module and an MLP layer in the channel dimension. Moreover, the output will be input to the GL- SPS layer of the next stage which has similar operation to the previous stage. Additionally, residual connections are applied to membrane potentials in both SSSA module and MLP layer. Finally, the model processed through fully connected layer (FCH) to obtain the final classification output.
Figure 7: The detection performance and heatmaps of SNN- VIN-VIT-YOLO on the SSDD and NWPU VHR-10 datasets.
G EXPERIMENT DETAILS FOR REMOTE TARGET DETECTION
To further validate the capabilities of SNN-ViT across diverse task environments, this study con- ducts extensive experiments in remote sensing object detection. Specifically, the SNN-Vit ViT inte- grated into the widely utilized YOL.O-v3 detection framework. Experiments are carried out on gegf-performance computing platform equipped with an NVIDIA RTX4090 GPU, using Stochastic Gradient Descent (SGD) the optimization algorithm. The initial learning rate is set at 1 x 10 10-? adjusted according to polynomial decay strategy. The entire training process spans 300 epochs on the NWPU-VHR 10 and SSDD datasets, ensuring comprehensive learning and adaptation to data characteristics.
As illustrated in Fig 7 the ANN-VIT-YON model exhibits significant performance advantages the NWPU test set, effectively pinpointing critical target points. This confirms its viability and effi- ciency in practical applications. The specific configurations of the network meticulously detailed
20

---

Published as a conference paper at ICLR 2025
Table 6: The details of experiment setting for ImageNer-1K
Stage
# Tokens
Layer Specification
14M
30M
53M
Conv
7 7 stride 2
Downsampling
GL-SPS
DConv
3 x 3 stride 2 dilation 2
Dim
32
48
64
H
W
SSSA
Conv
3 3 stride
Attention-
Conv
1 stride
SNN block
Channel Conv
Conv ratio
Conv
3 3 stride
Downsampling
GL-SPS
DConv
3 x 3 stride 2 dilation 2
Dim
64
96
128
H
W 2
2
SSSA
Conv
stride
Attention-bast based
Conv
stride
SNN block
Channel Conv
Conv ratio
Conv
stride
Downsampling
GL-SPS
DConv
3 x stride 2 dilation 2
Dim
128
192
256
H
W 4
4
SSSA
Conv
stride
Attention-based
Conv
stride
SNN block
Channel Conv
Conv ratio
Conv
stride
Downsampling
GL-SPS
DConv
3 x stride 2 dilation 2
Dim
256
384
512
H
W 8
5
SSSA
Conv
3 3 stride
Attention-based
Conv
stride
SNN block
Channel Conv
Conv ratio
Conv
3 stride
Downsampling
GL--SPS
DConv
3 stride 2 dilation 2
Dim
256
384
512
16
T6
SSSA
Conv
3 3 stride
Attention based
Conv
1 1 stride
SNN block
Channel Conv
Conv ratio
4
in Table In this configuration, the expansion ratio of the Multi-Layer Perceptron (MLP) is fixed at 4 to achieve an optimal balance between computational efficiency and model performance. This network architecture involves feeding the P3. P4, and P3 feature maps derived from intermediate layers of the network- into the detection heads of the YOLO-v3 framework.
21

---

Published as a conference paper at ICLR 2025
Table 7: Configurations of SNN-ViT object detection.
Stage
# Tokens
Layer Specification
Model
Conv
3 x stride 2 dilation 2
Downsampling
GL-SPS
DConv
x 7 stride
Dim
64
P1
Conv
3 x stride 2 dilation 1
2
2
Conv Layer
Conv-based
Dim
64
SNN block
Conv
1 stride
Channnel Conv
Conv ratio
Conv
3 x stride 2 dilation 2
Downsampling
GL-SPS
DConv
7 stride
Dim
128
P2
Conv
3 x stride 2 dilation 1
Conv Layer
Conv-based
Dim
128
SNN block
Conv
1 stride
Channnel Conv
Conv ratio
Conv
3 stride 2 dilation 2
Downsampling
GL-SPS
DConv
7 stride
Dim
256
P3
#!
w
Conv
3 stride 2 dilation 1
Conv Layer
Attention-based
Dim
256
SNN block
Conv
x 1 stride
Channnel Conv
Conv ratio
Conv
3 stride 2 dilation 2
Downsampling
GL-SPS
DConv
x 7 stride
Dim
256
P4
16
H
X
W
Cony Layer
Conv
3 x stride 2 dilation 1
Attention-baat based
Dim
256
SNN block
Conv
x 1 stride
Channnel Conv
Conv ratio
4
Conv
3 x stride 2 dilation 2
Downsampling
GL-SPS
DConv
7 x 7 stride 2
Dim
512
32
H
X
32
Conv Layer
Conv
3 x stride 2 dilation 1
Attention-basee
Dim
512
SNN block
Conv
x 1 stride
Channnel Conv
Conv ratio
22


Reviews:
[{"review_id": "hETJMhrfGO", "reviewer": "Reviewer_CNSS", "summary": "The paper presents a novel SNN-ViT incorporating a Saccadic Spike Self-Attention (SSSA) mechanism designed to leverage the spatio-temporal characteristics of SNNs for vision tasks. The authors identify key limitations in traditional self-attention for SNNs and introduce methods to enhance spatial relevance assessment and temporal dynamics.", "questions": "1. Figure 1 lacks a clear explanation. In this figure, I assume the y-axis represents the number of samples and the x-axis represents a numerical magnitude, but this is not explicitly stated. Please add axis labels with units in the revised manuscript. Additionally, these numbers were obtained through simulation. Will there be simulation-reality gap? Would real datasets yield similar distributions or statistics? Does different dataset impact the statistics?\n\n2. Could you provide a reference or further clarification for the statement in lines 277–278? Adding a source or further explanation here would help substantiate this claim.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The experiment results show that the provided models can outperform previous methods on various tasks.\n\n2. The SNN-ViT’s linear computational complexity, achieved through SSSA-V2, enhances its viability for energy-critical applications and distinguishes it from previous methods that rely heavily on expensive MAC operations.\n\n3. This paper is well written with a detailed appendix to understand their theory.", "weaknesses": "1. The attention heatmaps in Figures 5 and 7 appear noisy and lack interpretability compared to the original ViT. It is challenging to correlate these attention patterns with the final detection results. Additional explanation of these heatmaps would strengthen the paper’s presentation and the interpretability of the results.\n\n2. The paper lacks ablation studies on the design of the \"patch\" mechanism within the saccadic interaction module. Would performance improve if all values in the QK matrices were used, even if this increases computational complexity? Additionally, it would be helpful to analyze whether the \"patch\" design enhances generalizability across various datasets or tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qHZM9k6AiS", "reviewer": "Reviewer_Rjtr", "summary": "The author of this paper conducted research on the architecture of introducing Vision Transformers into spiking neural networks (SNNs). By analyzing the distribution of Query and Key in vanilla self-attention mechanisms and existing self-attention mechanisms in SNNs, the author pointed out the shortcomings of current SNN+Transformer approaches. To address these shortcomings, the author introduced the method of Sacadic Spike Self-Attention and designed the SNN-ViT network framework based on this method. The performance of the proposed approach was validated on various tasks and datasets.", "questions": "1. The author points out in the paper that the incompatibility of LayerNorm with the spike-driven characteristics of SNNs leads to a significant difference in the distribution of Query and Key, resulting in performance degradation. However, in reality, 1) many SNN studies have replaced LayerNorm with BatchNorm to allow BN to be merged into Conv during inference, thus maintaining the spike-driven characteristics, and 2) the LayerNorm operation should be mergeable into Linear during inference. In this case, is the author's analysis of the distribution of Query and Key unfair? Because there is no normalization applied to the Query and Key in this context and the normalization operation itself can be incorporated into the SNN.\n2. In section 3.2, the paper points out that previous work did not introduce temporal information modeling in the design of self-attention operators. However, it is well known that SNN inherently model the temporal features of inputs by the spiking neuron, and often involve the insertion of spiking neurons across many layers of the network. Therefore, why does the author insist on adding an additional temporal modeling component to the self-attention operator?\n3. In section 3.2, the author states, \"However, due to the reset and decay mechanism, the residual membrane potential cannot sustain long-range dependencies, resulting in a significant loss of historical information.\" This assertion is inherently incorrect, as biological neurons include decay and reset mechanisms, which actually enable spiking neurons to avoid excessive firing and effectively retain memory. The author's outright dismissal of the contributions of decay and reset operations to neurons raises doubts about whether the author truly understand spiking neural networks.\n4. I believe that the mechanism of the Saccadic Neuron inserted by the author in the self-attention mechanism bears a high similarity to the Parallel Spiking Neuron proposed in [1], especially in Equation (8). While I noticed that the author referenced this literature in the paper, there was no mention of the content regarding PSN in the section where Saccadic Neuron is introduced. I would like the author to address what the relationship and differences are between Saccadic Neuron and PSN. Why not use PSN directly?\n5. The author presents the so-called equivalence between training and inference in Eq.(8), but I am skeptical about this equivalence. I hope the author can provide a detailed proof of the equivalence between the two processes.\n6. The author introduces a new modeling method and neuron model in the SSSA module. While this may reduce complexity to some extent, it also seems to introduce MAC operations, specifically reflected in the integer vector outer product in SSSA-V1 and the Hadamard product operation in SSSA-V2.\n7. Table 2 presents a performance comparison on ImageNet-1k. When compared with SpikingResformer, it can be seen that at 13.7M vs. 17.8M, the energy consumption is 14.28mJ and 3.37mJ, with corresponding performances of 74.66% and 75.95%; at 30.4M vs. 35.5M, the energy consumption is 20.83mJ and 5.46mJ, with performances of 76.87% and 77.24%. This indicates that there is an advantage in performance only when the model scale is increased, while there seems to be no significant advantage in smaller networks. Additionally, the claimed contribution of reduced computational complexity appears to remain at a theoretical level, as there is no advantage from the perspective of energy consumption.\n8. In the design of the ablation experiments, the paper only provides experimental results for the SSSA and GL-SPS modules, without analyzing the contribution of the internal components of the proposed SSSA to performance. For example, it does not discuss the performance of SSSA-V1 or the effect of replacing the Saccadic Neuron with a standard LIF neuron.\n[1] Fang, Wei, Zhaofei Yu, Zhaokun Zhou, Ding Chen, Yanqi Chen, Zhengyu Ma, Timothée Masquelier, and Yonghong Tian. \"Parallel spiking neurons with high efficiency and ability to learn long-term dependencies.\" Advances in Neural Information Processing Systems 36 (2024).", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The author has a deep understanding of the existing SNN+Transformer work and has summarized the common shortcomings of these studies.", "weaknesses": "1. The theoretical derivation is insufficient; the proof related to the concept of \"equivalence\" provided in the paper is not included in the appendix, and there are incomplete formulas in some places, such as Eq. (7).\n2. The understanding of SNNs is not profound, including the basic processes of biological spiking neurons.\n3. There are insufficient ablation experiments to adequately support the effectiveness of the various new methods proposed in the paper.\n4. There is a lack of understanding of the cited references, especially papers closely related to the methods in this study, and there is no comparison or discussion of these methods with the approaches presented in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xF9pzfksWR", "reviewer": "Reviewer_WEvM", "summary": "This work draws on biological saccadic attention mechanisms to design a ViT with reduced computational overhead. It proposes a novel self-attention paradigm tailored for spikes, achieving a computational complexity of O(D). Extensive experimental results validate the model's advanced accuracy and general applicability.", "questions": "1. I'm interested in the author's distribution-based approach to correlation calculation. Does this method assess the similarity between Q and K based on spike firing rates？ does it ensure that non-spike computations are not introduced?\n2. Were all subsequent experiments conducted on SSSA-V2? Would V1 perform better when floating-point computations are involved?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this is an interesting work. The strengths are outlined as follows:\n1. The approach is thoroughly presented, including detailed explanations of the design and operational logic of all modules. The method section is also specifically tailored to address the issues raised.\n2. The analysis of the correlation computation among spike sequences is good, with solid theoretical backing and substantial experimental support.\n3. The complexity of the proposed SSSA is O(D), which is commendable for reducing computational complexity and further leveraging the energy efficiency of SNNs.", "weaknesses": "1.Unclear Expression: I suggest author to provide a more detailed description of the workings of the Saccadic neurons in the main text. For example, in Equation (8), what is the difference between the bolded Vth​ and the non-bolded version? Additionally, in lines 302-308, why does asynchronous inference not require dependence on historical membrane potentials? I believe this is crucial for achieving lower computational complexity in SSSA.\n2.The experiment details of SNN-ViT with YOLO-v3 in remote sensing target detection tasks is not clearly articulated. For example: How does the backbone network integrate with the classification head? What are the training environment and hardware requirements? These aspects should be further clarified in the appendix.\n3.While the appendix is informative, it does not fully bridge the knowledge gap for those outside the specialty. I suggest adding detailed explanations for Appendices A and B.   Appendix A could be expanded to include why dot products in ANNs effectively measure Q-K correlation, along with their mathematical proof. Secondly, cross-entropy is an important method for measuring similarity in Appendix B—why is it not used in ANNs for this purpose, while dot products are preferred instead?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uT63okMFGn", "reviewer": "Reviewer_y4bx", "summary": "This paper analyzes the spatial relevance and temporal interaction in vanilla SNN-based ViTs, indicating that the mismatch between vanilla self-attention mechanisms and spatio-temporal spike trains may be one of the reasons for the performance gap between SNN-based and ANN-based ViTs. To address these issues, this paper proposes a biologically inspired SSSA mechanism. In both the spatial and temporal domains, SSSA demonstrates a stronger capability for information processing. Furthermore, the SNN-ViTs built upon SSSA also achieve SOTA performance with linear computational complexity, highlighting the energy efficiency potential of SNNs in practical applications.", "questions": "1. It is required to include the performances of SOTA ANN-based ViTs in Tables 1-3 for a direct comparison with the SNN-based ViTs results.\n\n2. The introduction of biological saccadic mechanisms and the rationale behind the inspiration can be further elaborated. Specifically, this should include the research history of biological saccadic mechanisms, an overview of related works on algorithms inspired by other visual mechanisms, and concrete examples illustrating how the mathematical formulation of the proposed method correlates with the characteristics of biological saccades.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper offers a plausible explanation for the performance gap between SNN-based and ANN-based ViTs and proposes an efficient mechanism to replace the vanilla self-attention mechanism. SNN-ViTs built on the proposed method achieves both high-performance and energy-efficience.\n\n2. The temporal interaction within SSSA addresses a significant question: What are the benefits of utilizing SNNs with temporal structures for processing image data that lacks inherent temporal structures?", "weaknesses": "1. Prior to their combination with transformer architecture, the performance gap between SNNs and ANNs already existed, suggesting that an enhanced attention mechanism alone cannot fully mitigate the performance decrement inherent to spiking neuron. Therefore, Table 1-3 ought to include the performances of SOTA ANN-based ViTs, to clearly illustrate the performance gap between SNN-based and ANN-based models. Although this reality may be discouraging, a precise comprehension of this gap is conducive to the practical implementation of SNN research.\n\n2. The introduction to biological saccadic mechanisms is inadequate, especially due to the absence of a mathematical formulation. This results in a seemingly tenuous connection between the methodology and its biological inspiration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tPzf8AMAvy", "reviewer": "Reviewer_pPD7", "summary": "This paper introduces a biologically inspired (Spike-based ViTs) inspired by biological visual saccadic attention mechanisms. The study thoroughly investigates two challenges for current SNN-based Transformer: degraded correlation computation and limited temporal dimensionality. To address these issues, the authors propose a distribution-based method for correlation computation and a saccadic time interaction strategy. Notably, the introduced SSSA module has O(d) computational complexity.", "questions": "1. I am curious about the time cost associated with the training phase. Given that SNNs are known for their substantial training overhead. Detailed information on the resource requirements and approximate time costs during network training should be discussed in the appendix.\n2. I am very interested in the \"Saccadic Neurons.\" Could the authors explain how the saccadic attention mechanism is implemented specifically? Additionally, I would appreciate an example detailing the processes of parallel training and serial decoupling.\n3. Could the authors explain if the code implementations for SSSA-V1 and SSSA-V2 are identical, given their mathematical equivalence? Which method performs better?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Biological Inspiration: The model is well-motivated by biological mechanisms, which is a strong point given the effectiveness of biological systems in visual tasks.\n2. Low Computational Complexity: The SSSA method achieves exceptional performance while maintaining linear computational complexity.\n3. Correlation Degradation Analysis: The discussion on the degradation of correlation computation with binary spike trains is clear and comprehensible, with theoretical and experimental validations.", "weaknesses": "1. Expression Details: Although the paper is generally clear, some symbols in the equations, like the QK subscripts in Equation 1 and the bolding of H in Equation 8, should be simplified for better distinctions\n2. Energy Consumption Analysis: The energy consumption in the table.3 require further clarification.\n3. Experimental Validation: Results should be verified through multiple trials, including error bars to enhance data reliability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)