Neural Dynamics Self-Attention for Spiking Transformers
Authors: Dehao Zhang
Fukai Guo
Shuai Wang
Jingya Wang
Jieyuan Zhang
Yimeng Shan
Malu Zhang
Yang Yang
Haizhou Li
Venue: ICLR 2026 Poster (2026)
Keywords: Spiking Neural Networks
Spiking Self-Attention
Spiking Transformer
Primary area: applications to neuroscience & cognitive science

Abstract: Integrating Spiking Neural Networks (SNNs) with Transformer architectures offers a promising pathway to balance energy efficiency and performance, particularly for edge vision applications. However, existing Spiking Transformers face two critical challenges: 
(i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference. Through theoretical analysis, we attribute both limitations to the Spiking Self-Attention (SSA) mechanism: the lack of locality bias and the need to store large attention matrices. Inspired by the localized receptive fields (LRF) and membrane-potential dynamics of biological visual neurons, we propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements. Specifically, we introduce a LRF method into SSA to assign higher weights to neighboring regions, strengthening local modeling and improving performance. Building on this, we approximate the resulting attention computation via charge–fire–reset dynamics, eliminating explicit attention-matrix storage and reducing inference-time memory. Extensive experiments on visual tasks confirm that our method reduces memory overhead while delivering significant performance improvements. These results establish it as a key unit for achieving energy-efficient Spiking Transformers.

Published as a conference paper at ICLR 2026
NEURAL DYNAMICS SELF-ATTENTION FOR SPIKING TRANSFORMERS
Dehao Zhang', Fukai Guo', Shuai Wang', Jingya Wang', Jieyuan Zhang', Yimeng Shan', Malu Zhang!.2; Yang Yang', Haizhou Li233
Uniivessity of Electronic Science and Technology of China,
Shenzhen Loop Area Institute, 'The Chinese University of Hong Kong (Shenzhen)
ABSTRACT
Integrating Spiking Neural Networks (SNNs) with Transformer architectures of- fers a promising pathway to balance energy efficiency and performance, particu- larly for edge vision applications. However, existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artifi- cial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference. Through theoretical analysis, we attribute both limitations to the Spik- ing Self-Attention (SSA) mechanism: the lack locality bias and the need to store large attention matrices. Inspired by the localized receptive fields (LRF) and membrane-pee pottentaa dynamics of biological visual neurons, we propose LRF- Dyn, which uses spiking neurons with localized receptive fields to compute at- tention while reducing memory requirements. Specifically, we introduce a LRF method into SSA to assign higher weights neighboring regions, strengthening local modeling and improving performance. Building on this, we approximate the resulting attention computation via charge-fire-resct dynamics, eliminating explicit attention-matrix storage and reducing inference-time memory. Extensive experiments on visual tasks confirm that our method reduces memory overhead while delivering sigmificant performance improvements. These results establish it as a key unit for achieving energy-efficiee Spiking Transformers.
INTRODUCTION
Vision Transformers (ViTs) (Vaswani al. 2017 Liu al. 2021 Yu al. 2022 achieve markable breakthroughs in computer vision tasks, including image classification (Chen et al. [202]] Deng al. 2009). object detection (Carion al.; 2020 Shan et al. 2025). and semantic segmen- lation (Zhou et al. 2017: Xie et al., 2021 Yu et al. 2018). As the core of ViTs, the vanilla self- attention (VSA) computes query-key similarity through dot-product operation followed by softmax operation, incurring quadratic computational and memory costs with respect to the sequence length N (Yang al. 2023). To address this issue, recent methods such as Linear attention (Katharopou- los et al. [020 Zhang et al.) 2024b) approximate or eliminate the softmax operation to explicitly compute the entire N attention matrix, thus reducing memory usage and achieving linear-time complexity. However, these methods still rely on full-precision matrix multiplications, which incur substantial energy overhead and ultimately hinder their deployment on resource-constrrinee devices.
Spiking Neural Networks (SNNs) (Maass 1997 Gerstner & Kistler 2002 Masquelier et al.) 2008
attract increasing attention due to their biological plausibility and potential for low-power comput- ing. As the fundamental computational units, spiking neurons fire spikes only upon activation and remain silent otherwise, thereby enabling event-driven computation (Deng et al.) [2020] Zhhng et al. 2025d). This mechanism ensures sparse information transmission and effectively avoids redundant multiply-accumulaa (MAC) operations, leading to reduced energy consumption and lower computa- tional overhead (Caviglia et al. 20144 Roy al. 2019). The combination of SNNs and Transformer architectures offers a potential pathway to balance energy efficiency and high performance (Zhou
al. 2023b Huang al. 2024 Yao al. 2025 Wang al. 2025a Wang al.).
*Corresponding author; maluzhang@uestt.edu...

---

Published as a conference paper at ICLR 2026
In recent years, several CNN-based Transformer architectures proposed (aao al. 2024a Shi
jet al. 2024, Cao et al. 2025; Wang et al. 2025b), including Spikformer (Zhou et al., 2023b). QK- Former (Zhou al., 2024). and Spike-Driven-yy (Yao et al., 2025]. Nevertheless, baN-based Transformers still exhibit performance gap and incur substantial memory overhead. As shown in Fig. Ta), the attention distributions of VSA and SSA differ markedly: SSA exhibits discrete and global patterns, whereas VSA shows localized and sparse ones. This mismatch hinders SNNs from focusing on specific regions of information. Moreover, as illustrated in Fig, [b), SNNs can leverage matrix aggregation and flexibly adapt their computational paradigms to different applica- tion scenarios. However, they still require explicit storage of large attention matrices, such as QK matrices of size N2 or KV attention matrices of size d,, which severely restricts deployment on resource-limited devices. Therefore, balancing the low-energy benefits of SNNs with the need for low memory overhead and high performance remains a critical open challenge.
(a) Limited Local Modeling Capability
(b) High Memory Requirements
Raw Image
Spike Trains
SSA
SSA
Spike Trains X
Conv&BN
ConveBN
Conv&BB
Conv&BN
Conv&HN
CN
VSA
Transpooe
Tranispose
Ned
Nxd
Nxd
dxN
Nxd
dxd
QK
SSA
Matrices
Attention
N=N
Nkd
onNaa OONE
O
OLd
Nxd
Attention Matrices
Neuron
Dat- Dat-Product
Compute Complexity
Memory Requirements
Figure I: (a) Limited Local Modeling Capability: For given n-th query (blue), VSA captures only limited and local relation.
(b) High Memory Requirements: SSA requires explicit storage of their associated attention scores (QK or KV), leading to substantial computational overhead.
Inspired by local receptive fields (Gaynes et al. 2022) and the temporal dynamics of neuronal mem- brane potentials, we propose LRF-Dyn, which enables attention computation via spiking neurons endowed with localized receptive fields. First, theoretically and empirically compare VSA and SSA, showing that the lack of local modeling capacity in SSA accounts for its performance gap relative to VSA. To address this, propose Local Receptive Field (LRF) methods into SSA, assigning higher weights to spatial neighbors to strengthen locality. Building on this, we introduce LRF-Dyn, which establishes an approximate correspondence between self-attention aggregation and the charge-fir--reset dynamics of spiking neurons, thereby eliminating the need to explicitly store attention matrices. Extensive experiments across diverse vision tasks and architectures show that our method improves performance while further reducing inference- time memory requirements. The main contributions are as follows:
We identify two major challenges in existing Spiking Transformers: (i) the performance gap caused by limited local modeling capability resulting from the removal of the softmax operation, and (ii) the high memory overhead due to the need to store attention scores.
We propose LRF-Dyn to address these limitations. First, we incorporate a LRF method into SSA to assign higher weights neighboring regions, thereby strengthening local modeling and improving performance. Building on this, we further approximate the computation via neuronal cembbrane-potnntii dynamics, eliminating the need store attention matrices and reducing memory overhead during inference.
Extensive experiments across diverse SNN architectures and visual tasks demonstrate that our method improves performance while further reducing inference-time memory require- ments. It offers practical solution for deployment in resource-constrained environments.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
Vision Transformer: VIT (Dosovitskiy et al. 2020; Liu ct al., 2022b) converts images into patch- based tokens and utilizes self-attention establish global contextual relationships. facilitating the selective aggregation of informative features (Naseer et al. 2021 Yang et al. (021]. However, these models suffer from O(N2) computational complexity and substantial memory overhead, which hin- der scalability to large-scale visual tasks (Yang et al. 2023: Zhang et al. 2024a). To address these
limitations, several studies (Choromanski et al., 2020 Guo et al. (024)) explore linear attention, which replaces the softmax operation with kernel based approximations to reduce the complexity from (N2) (N). These methods significantly reduce both computational and memory require- ments, making them more suitable for nigh-resolution images (Shen et al. [021 Guo et al.) 2022).
However, existing models still rely on full-pred full-precision matrix multiplications, potentially increasing
the model's energy consumption (Wang et al. 2020 Liu et al. 2022a).
Spiking Transformer: recent years, researchers (Wang al. 2023 Yao et al., 2023 Zhou al.)
2023a Wei et al.) have explored combining SNNs with Transformers to achieve trade-off between energy efficiency and performance (Yao et al. 2024b). Spikformer (Zhou et al. 2023b) introduces the SSA mechanism, which preserves spike friendly properties while significantly improving the performance of SNNs. SpikingResformer (Shi et al. (2024) integrates ResNet with Transformer architectures to further reduce the model parameters. Furthermore, Spike-Driven-V3 (Yao et al., 2025) incorporates the Spike Frequency Approximation (SFA) mechanism into Spiking Transform- ers, enhancing their performance advantages. Although these models significantly reduce energy consumption, the inference process of SNN-based Transformers exhibits higher memory demands, limiting their deployment on resource-constrained devices (Aguirre et al. 2024 Zhang et. 2025a)
PRELIMINARY
3.1 SPIKING NEURON MODEL
As the fundamental units of SNNs, spiking neurons (Izhikevich) 2003 Maass 1997)
receive presy-
naptic inputs and integrate them into the membrane potential, which is compared with the threshold to determine whether a spike is generated. Among them, the Leaky Integrate-amd seterate-an-Fiin and-Fire (LIF) neuron the widely used model, whose dynamics are defined follows:
Uic + 1] [(4] + W + 1).
(1)
S[t + 1] = @U] + Vn),
(2)
H[t + 1] Vte + 1] + - - + 1]).
Here, H(t and U[t] denote the pre-synaptic and post-synaptic membrane potentials, respectively, while S(t] indicates the input spike at timestep t. W denotes the synaptic weight matrix. Spike
generation is defined by the Heaviside function (--): if spike occurs (Sit = 1). H[] is reset to Vreset; otherwise, Ult + 1] decays with time constant 7 and updates H + 1]. For clarity, we denote the above process as SN..), which represents the dynamics of spiking neurons.
3.2 SPIKING SELF-ATTENTION MECHANISMS
As the core component of the Spiking Transformer, the SSA mechanism captures spatio-temporas dependencies among tokens from spike trains and adaptively allocates importance across different regions. Given an input sequence X E 7 Nx it is projected through convolutional layers with distinct parameter matrices to obtain the Query Q Key K, and Value V representations:
Q SN
K = SN/BN(Conv((X)))
V SN(BB(Convv(())
(4)
where Conv(-) denotes a 1x 1 convolution, and BN(-) represents batch normalization. Inspired by the VSA mechanism, SSA computes the similarity via the dot product of Q and K, and employs the resulting weights aggregate V. Specifically, the process of SSA is defined as follows:
Score 8 Q KT, Attn' Score V, Attn SN SN(tttn'),
(5)
s denotes the scaling factor. The attention output Attn' is processed by spiking neurons to ensure event-driven characteristics. Unlike VSA, SSA omits the softmax operation, thereby preserving the event-driven and spike-friendly characteristies of the attention mechanism.
3

---

Published as a conference paper at ICLR 2026
4 PROBLEM ANALYSIS IN SPIKING TRANSFORMER
This section examines two main limitations of applying self-attention in SNNs: i) the omission of the softmax operation leads SSA to produce attention score distributions that deviate from those in VSA and ii) compared to other softmax-free attention variants, SSA introduces higher memory usage and inference overhead. Further details are provided in the following sections.
4.1 LIMITED LOCAL MODELING CAPABILITY
As the core mechanism of ViT, the VSA mechanism computes attention scores through the dot- product operation followed by the softmax operation. Specifically, given Query Q E RN and K E NXC, the attention score Attnvit can be defined as follows:
( x K" (x<K)= .Vaa
exp(qik;}
Attivit = softmax
abtnvit [i] = E exp(q;kz]
(6)
Vd
d denotes the features dimension, and attnvulj E RIxN represents the attention score correspond- ing between the i-th query qi and remaining tokens. When q and k are more similar, the attention score is higher. Since neighboring tokens usually exhibit stronger similarity, ViT demonstrates su- perior local modeling ability (Yang et al.) 202T). As shown in Fig. 2 76.8% of the attention scores in ViT are concentrated at short Manhattan distances. In contrast, SSA produces an almost uniform distribution of attention scores. This mismatch constrains the local modeling capacity of SSA and hinders its ability to capture spatial similarities among neighboring regions.
(a) Similarity in VSA
(b) Similarity in SSA
Attennin Score in VSA (d) Attention Score in SSA
0.1
0.1
Jugi
[0. 5): 76.85%
0:35
[0, 5): 20.31%
H-0 1277
5637
(5.25) 23.32%
(5.26) 79.599%
0.5
0:5
031
0.55
0:3
0.3
031
0.30
0.345
20
Distance
Distance
Attention Score
Attention Score
Figure 2: Mismatch between VSA and SSA attention scores: (a) and (b) show the average attention scores at different Manhattan distances, with VSA demonstrating stronger local modeling capabili- ties. (c) and (d) illustrate the distribution of attention scores, with VSA exhibiting lower entropy.
Furthermore, we compare the attention score distributions of SSA and VSA. As shown in Fig. SSA exhibits more uniform distribution, which hinders the model's ability to emphasize the relative importance of different regions. In contrast, VSA yields lower-entropy distribution that focuses attention on a few critical regions, thereby enhancing feature extraction effectiveness.
4.2 HIGH MEMORY REQUIREMENTS DURING INFERENCE
Similar to other softmax-free models Waan et al. 2020), SSA leverages the associative property of matrix operations to reduce computational complexity to O(Nd?). Nevertheless, this computational benefit is accompanied by a pronounced increase in memory overhead during inference. Specifically, for the query [ at timestep t, the attention score attn, [[t] can be defined as follows:
attn [4]
qo 14 e5) v([ qn [4
X
kg o,(4),
N denotes the total number of tokens, and e; is a column vector of length N with the j-th entry set to and all others set to 0. As shown in Eq. for the n-th token, it is necessary to store not only the Q K, and V matrices at each timestep, but also the intermediate results of the KV multiplication, leading to additional O(d?) memory overhead. Notably, when d 512, SSA incurs substantial memory demands. These limitations substantially hinder its deployment on resource-constrainee devices, particularly on neuromorphic chips (Davies et al. 2018 Pei al. 2019
4

---

Published as a conference paper at ICLR 2026
5 METHOD
In this section, we first incorporate the LRF mechanism into SSA to enhance local modeling and improve performance. Furthermore, we propose the LRF-Dyn that restructures this computations from the perspective of neuron modeling, reducing memory overhead while preserving performance.
5.1 SPIKING SELF-ATTENTIN WITH LOCAL RECEPTIVE FIELDS
To address the limited local receptive field of spiking self-attention, we propose LRF-SSA, which integrates local convolution module into the SSA mechanism to increase sensitivity to neighboring positions. Specifically, for the 7-th token, its self-attention output sattn,, [t] can be expressed as:
N
sattn [t]] = qn qn(t] x
k ky
ΣΣ p dijERaa
(8)
Global Receptive Fields
Local Receptive Fields
Qa = ((i.))ii. {-d,0,d}} represents the positional information of the neighboring region. To further reduce model complexity, we introduce multi-scale dilated convolutions to model local receptive fields. Specifically, two 3 x 3 depth-wise convolution kernels with dilation factors and d = 5 are employed, where rij denotes the convolutional parameter at position (i.j).
Theorem Leti N {1.- . denotes the token position and defined the Manhattan distance between two elements as = d(i, = - jl. The normalized attention weight of VSA is di
via &
cxp(-BA). For SSA, the weight satisfies of x (a- BB)++ The LRF-SSA is defined as a
Infasa
(1 A)ag + Araj- Specifically, the expected receptive fields of LRF-SSA are defined:
(9) Since 14, represents the receptive field around each token, it naturally satisfies that H4, S Pssa- The- orem demonstrates that LRF-SSA preserves the local attention characteristics similar to those of VSA, whereas SSA exhibits uniformly distributed attention weights. This difference arises because SSA removes the softmax operation, resulting in a linear decay of attention scores and diminishing its ability to distinguish between neighboring positions. In contrast, LRF-SSA incorporates addi- tional learnable weights within the spatial domain, allowing the model to concentrate more effec- tively on information captured by the local receptive field. We need to further evaluate whether this operation can effectively preserve the low-entropy distribution property of the softmax operation.
E[Antssal =(1 (1- A)Husa + Aptr where
Uesa =
Bjmpps [Asa)
Hr - Ey (Ar),
Theorem 2
For given attention weights x (21...., IN), the information entropy is defined as H (x)). In particular, the entropy of VSA is expressed as H (exp(-3Ai), while SSA satisfies H H(aa 10--A +)) LRF-SSA satisfies H H(1 (1- A)pt + 2r.s). These entropies satisfy the ordering:
H(pl < h((a)) + HpH + +1 a:) (r:) I H(p where 0 aq S
(10)
Theorem demonstrates that LRF-SSA exhibits lower-entropy distribution more closely aligned with VSA. This effect arises primarily from the presence of local receptive field modules, which amplify the differences among attention scores. A detailed proof is provided in the Appendix C and Appendix [ Compared with VSA, our method eliminates the need for the softmax operation, thereby achieving performance comparable SSA while reducing computational cost.
In summary, our method effectively preserves the local receptive field and low-entropy distribution characteristics of VSA. Compared with SSA, it demonstrates stronger local receptive field capabil- ity while introducing only minimal additional overhead (two 3x3 convolution kernels and an extra (d?) computational cost). The effectiveness will further validated in the experimental section.
5.2 IMPLEMENTING SELF- THROUGH NEURONAL DYNAMICS
To further reduce the inference-time memory footprint and computational latency of LRF-SSA, we propose LRF-Dyn. As previously noted, the LRF-SSA module must store the Q, K, and matrices at each timestep, along with their corresponding attention scores. Specifically, when the model applics the SSA Version illustrated in Fig. (b). LRF-SSA reduces the computational complexity to (NdP), while requiring the additional storage of an attention matrix of size d².
5

---

Published as a conference paper at ICLR 2026
(a) Biological Visual Cognition
(b) Local Receptive Field Method
(c) Our Method
Local Receptive Field
Local Receptive Field Method
LRT-SSA
LRF
Hird
U-Coev
Linesr Linear Linear
BN
BN
BN
BNN
vibual field
Dillteed
BN
BetchNorm
Multi-Dendriess and Soma Structure
Dot Product
Dendrite Spiking Neuron
Branch
Attnss
Branchy
Dot Product
Branch
Dar by
Telo
Soma
Th
B.U.
Paug
Neuron
Dendritic
Dendrite
Dendrite
Sama
Usera
Memocy. O(d')
Memory: (NG)
Figure 3: (a) Cognitive processes in biological vision, which exhibit local receptive field properties realized through multi-dendritic neurons. (b) The proposed LRF method together with the dynamic processes of dendritic neurons. (c) The implementation of LRF-SSA and LRF-Dyn.
Inspired by other softmax-free attention (Ynng et al. 2023 Zhang et al.
2024b Shen et al. 202
LRF-SSA can be reformulated through causal inference significantly reduce memory consump- tion. Specifically, Eq. 8 can be rewritten as follows:
sattn, [t]' 9n (t)
kyld th 1
+
ΣE i.jEn
1 varl)
(11)
Memory Posential
Presymaptic Input
In this manner, LRF-SSA method only needs to store Σ}= k vJ_ thereby reducing the computa- tional complexity ONd?) to (d?). The attention output is then multiplied by the current query vector 9n and transformed into a spike sequence through the SN layer. It closely parallels the charge-fire-reset dynamics spiking neurons, where the first term represents membrane potential information and the second term represents presynaptic input.
Therefore, we propose LRF-Dyn which leverages neuronal dynamics to formulate a novel paradigm for self-attention computation, whose dynamical process can be defined as follows:
X, it] = A@ [t] PToken, [t), sattn', [t] = Xx, [t] ΣΣ
X it)
(12)
diJeRa
Here, Token, denotes the token input at position E IRd defined as the membrane capacitance constant. Inspired by the multi-timesscae behavior of photoreceptor neurons (Zheng al. 2024 Chen et al. (024 /Zhang al.] (0255), define A
A E Rd denotes the decay factor, and
and r in a dendritic form with local receptive fields to better allocate attention scores across tokens:
T
1
32,1
0
Cti
C2
$1,2
Ba,
0
Y2,
A
?
(13)
Cn-1: 1:
Bun--
Mn-11
Tn
On
Bn-1.
Yn
Here, d, denotes the number of dendrites, and C represents the weights assigned to different dendrites. As shown in Fig. b)) for the n-th token, different dendritic branches produce distinct responses, which are further integrated by the soma through a specific mechanism to enhance spatial interactions and transform them into spike trains. Owing to the time-invariant property of the A matrix, the neuron can be trained efficiently following the (Chen et al., 2024). Compared with LRF. SSA, our model climinates the SSA computation and only requires storing the membrane potential at each position, thereby substantially reducing memory usage during inference. In this study, is set as 8. We will further validate the effectiveness of this method in the experimental section.

---

Published as a conference paper at ICLR 2026
5.3 OVERALL ARCHITECTURE
In this section, we present the overall architectures of LRF-SSA and LRF-Dyn. respectively. Specif- ically, for an input spike train & E RT'xXx,, the computation of LRF-SSA is defined as follows:
Q.K. V - SN(BN(Conv (X))), Score SN(Ss - (Q K" +Σ ijens ( x V).
(14)
Compared with SSA, LRF-SSA introduces almost no additional parameters, yet it significantly en- hances the local modeling. Building on this, LRF-Dyn further reduces the memory requirements during model inference. The dynamics of LRF-Dyn are defined as follows:
H H= (F(K) (F(K)
F(X)), Score SN M iijen
s.jen, ry
og agH pc()))
(15)
where FF1 denotes the forward and inverse Fourier transforms, respectively, and represents the convolution operation. (C()) is the convolution kernel, defined as TC re A. As shown in Fig. Bic), both methods can be integrated into existing Transformer frameworks without any ad- ditional modifications. We further demonstrate the advantages of our approach in terms of both performance and memory efficiency in the experimental section.
EXPERIMENT
In this section, we compare with advanced SNN models on image classification (Deng et al. 2009) and semantic segmentation (Zhou al. 2017) tasks. Additionally, conduct ablation studies to validate the effectiveness of the proposed method, demonstrating its improved performance and reduced memory consumption. Detailed experimental results are provided below.
6.1 IMAGE CLASSIFICATION & SEMANTIC SEGMENTATION
Image Classification Tasks: we evaluate both LRF-SSA and LRF-Dyn on the ImageNet- 1k dataset. Specifically, we substitute SSA mechanism existing Spiking Transformers, including Spik- former (Zhou et al. (2023)), QKFormer (Zhou et al., 2024), and Spike-Driven-V. V3 Yao et al. 2025) with the proposed LRF-SSA and LRF-Dyn. Furthermore, we compare our approach with recently
advanced SNN models (Fang et al. 2021
Hu ct al. 2024 Yao et al. 2023 2024b).
( Image Classification Tasks
(b) Semantic Segmentation Tasks
Attention Heattmpp
Raw Image
SSA
TRR--SSA
Raw Tmage
Ground Truth
IRF-SSA
I.RF-Dn
Figure 4: Visual results for image recognition and semantic segmentation. Both LRF-SSA and LRF- Dyn produce sparser attention scores and achieve finer-grad grained segmentation results.
As shown in Table the proposed LRF-SSA method consistently delivers performance improve- ments across multiple architectures, with its two instantiations emphasizing different aspects. LRF- SSA primarily focuses on accuracy enhancement. and 0.85% at different parameter scales, while introducing fewer than 0.2M additional parameters. For QKFormer, LRF-SSA achieves an accuracy gain 0.44% and 0.48% under the dim=384 and dim=512 configurations, respectively. Within the SDT-V3 architecture, LRF-SSA further demon- strates a favorable balance between efficiency and accuracy, attaining 76.22% recognition accuracy with only 5.24M parameters, substantially outperforming models of comparable size.
Spikformer, it improves accuracy by 1.24%
In contrast, LRF-Dyn preserves recognition performance while reducing storage complexity to Okd)), where k denotes the number of dendrites. For instance, SDT-V3, LRF-Dyn achieves

---

Published as a conference paper at ICLR 2026
gain of 0.82% and 0.44% while requiring significantly less inference memory. Moreover, pared with CNN-based SNN models (Fang et al. 2021 Hu et al. 2024). LRF-Dyn further delivers substantial performance gains without relying attention mechanisms. As illustrated in Fig. Ia), both LRF-SSA and LRF-Dyn exhibit ViT-like attention patterns but with sparser distributions, en- abling the models to focus more effectively salient regions. These results further demonstrate the superior performance and reduced memory requirements of our LRF-SSA framework.
Table 1: Comparison with Similar Methods on ImageNet- IK.
Method
Architecture
SR.
Param.(M)
Acc.
SeWResNet
Fang al. 2021
SEW-ResNet-34
21.79
67.04
MeResNet
Hu et al. 2024
MS-ResNet-34
21.80
69.42
SDT-V1
Yao et al.
2023
Spike-driven-8-512
(
29.68
74.57
Meta-SpikeeFormer-3844
Odd2)
15.08
74.10
SDT-V2
Yao et al.
2024b
Meta-SpikeFFormer-512
O(d2)
55.35
79.70
Spikformer-8-512
O(d²)
29.68
73.38
Spikformer
Zhou et al.
2023b
Spikformer-8-768
O(d))
66.34
74.81
Spikformer-8-512
O(d²)
29.71
74.62
(11.24)
Spikformer + LRF-SSA
Spikformer-8-
O(d))
66.53
75.66 (10.85)
Spikformer-8-55
(kdd
29.71
74.51 (11.13)
Spikformer + LRF-DDn
Spikformer-8-768
((kd)
66.53
75.58 (10.77)
HST-10-384
Odd))
16.47
78.80
OKFormer Zhou et al. 2024
HST-10-512
(dd)
29.08
82.04
HST-10-384
Odd))
16.55
79.24 (10.44)
QKFormer LRF-SSA
HST-10-512
(dd)
29.18
82.52 (10.48)
HST-10-384
O(kd)
16.44
79.21 (10.41)
QKFormer + LRF-Dyn
HST-10-512
(kd))
29.18
82.48 (10.44)
Dfficient-Transformeer
(dd))
5.11
75,30
SDT-V3 Yao et al.
2025
Efficient-TransfommerLL.
(dd)
18.99
79.80
Sfficient-Transformee-S
(dd))
5.24
76.22 (10.92)
SDT-V3 + LRF-SSA
Efficient-Transformeell (d?)
19.25
80.31 (10.51)
Efficient--Transfrrmeet
((k))
5.24
76.12 12 (10.82)
SDT-V3 + LRF-Dyn
Efficient- Transiem-TTr O(kd)
19.25
80.24 (10.44)
SR represents the storage complexity requirements during inference.
Semantic Segmentation: To evaluate the effectiveness of our methods, we further con- ducted experiments on more challenging segmentation tasks. In particular, we evaluate the ADE20K dataset (Zhou et al.) (017). which consists 20K training images and 2K validation images across 150 semantic categories.
Table 2: Performance of segmentation.
Following the experimental protocol
SDT-V3 (Yao et al., 2025), we evaluate our method on models with 5M and 19M parameters. As shown in Table 2 approach achieves notable performance gains, improving by 2.6% and 2.2%, re-
Model
Para. (M)
Attn
MloU(%
ResNet
15.5
X
32.9
Yu al.
2022
28.5
X
36.7
PVT
28.2
/
39.8
Wang al.
2021
48.0
/
41.6
spectively. In addition, compared with
ResNet (Yu al., 2022) without atten- tion modules, LRF-Dyn achieves higher performance 136.3% and 13)1%) while re- quiring fewer model parameters. As il- lustrated Fig. we further provide qualitative comparisons selected exam- ples. The proposed method produces more fine-grained segmentation results, whereas SSA tends to yield only localized segmen- tations. This observation provides additional evidence of the effectiveness of LRF-SSA.
SDT-V3
5.1 + 1.4
4
33.6
Yao et al.
2025
18.99 1.4f
4
41.3
SDT-V3
5.1 1.4
36.2 (12.6)
+ LRF-SSA
1.4
/
43.5 (12.2)
SDT-V3
5.24
X
36.3 (12.7)
IRF-DDn
19.25 1.4
X
4
43.1 (11.8)
Results reproduced by ourselves.
8

---

Published as a conference paper at ICLR 2026
6.2 ENHANCED LOCAL MODELING ABILITY WITH LOWER MEMORY REQUIREMENTS
To further validate the effectiveness of our method, we follow Zhang et al. 2025c Ding et al. 2022) visualize the receptive fields. As shown in Fig- a).. LRF-SSA significantly enhances the local receptive field SSA and exhibits ViT-like local modeling capability. Similarly, LRF-Dyn also demonstrates a strong local modeling capacity. These results show the effectiveness of our method.
(a) Higher Local Modeling Capability
(b) Lower Memory Requirements
QKForre 3250
Buse
Atm
LRF-SSA LRF-Dyu
472
4.55
SDEVI
1.0
Accuracy
SDDT-
ERP
TH
Spitdformrr
1590
Raw Image
12.75
MS-Resset
Param (M)
VSA
SSA
LRF-SSA
LRF-Dyn
Figure 5: (a) Visualization of effective receptive field for different methods, where both LRF-SSA and LRF-Dyn demonstrate strong locality. (b) Comparative analysis of memory usage, accuracy, and parameter efficiency. The results show that LRF-Dyn maintains performance comparable to LRF-SSA while substantially reducing memory requirements during inference.
Furthermore, to compare the memory requirements of LRF-Dyn during inference, we visualize model accuracy and storage consumption at different scales. As illustrated in Fig. []b), Under architecture, our method achieves a 1.13% increase in accuracy while simul- the Spikformer-8-57
taneously reducing memory usage by 49.4%. These results further confirm the effectiveness of our approach based on local receptive field enhancement and neuro-dynamics-ing inspired modeling.
6.3 ABLATION EXPERIMENT
To further substantiate the effectiveness of the proposed approach, we conduct comprehensive periments on the CIFAR-100 dataset. Specifically, we investigate the respective contributions of local capability modeling and neurodynamics-insired self-attention both model performance and memory consumption, using the Spikformer architecture as the evaluation framework.
For the LRF module, we systematically exam- ine the impact of varying the number of convo- lution kernels on the results. Notably, without the LRF module, LRF-SSA is equivalent to the SSA method. As shown in Table 3 increasing the kernel count consistently improves recogni- tion accuracy, demonstrating that introduc- tion of local receptive fields enhances the per-
Table 3: Ablation Experiment.
Method
w/o LRF
Ω S
Q
LRF-SSA
77.86
78.26
78.52
78.64
LRF-Dyn
77.78
78.16
78.50
78.57
Caused SSA
74.30
75.30
76.20
76.50
Results reproduced by ourselves.
formance of both LRF-SSA and LRF-Dyn. Furthermore, by comparing the LRF-Dyn approach with causal SSA model, observe that LRF-Dyn consistently demonstrates improved performance under the same conditions. This further supports the effectiveness of our method.
7 CONCLUSION
In this paper, we analyze the challenges of integrating SSA into SNNs, focusing on two key limi- tations: the distinct attention distribution form compared to VSA, which limits further performance improvements, and the substantial memory overhead caused by the self-attention mechanism, which requires storing large attention weight matrices. To address these challenges, propose LRF-Dyn, achieving an effective balance between performance and memory efficiency. First, we provide the- oretical and empirical evidence for the importance of local modeling in self-attention, leading to propose LRF-SSA, which incorporates LRF module into SSA to enhance its local modeling capac- ity. Building this, approximate LRF-SSA with neuronal dynamics to remove the dependency on storing explicit attention matrices. This approach provides new theoretical insights and practical potential for deploying piff-merfrrm SNN models in resource-conssrainne edge environments.

---

Published as a conference paper at ICLR 2026
8 ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (Grants 62576080 and 62220106008), the Sichuan Science and Technology Program (Grant 2024NSFTD0034), the Guangdong Introducing Innovative and Entrepreneurial Teams (Grant 20232T10x044), and the Shenzhen Science and Technology Research Fund (Grant JCY/200208181000000000
REFERENCES
Fernando Aguirre, Abu Sebastian, Manuel Le Gallo, Wenhao Song, Tong Wang, Joshua Yang. Wei Lu, Meng-Fan Chang, Daniele Ielmini, Yuchao Yang. et al. Hardware implementation of memristor-basee artificial neural networks. Nature communications, 15(1):1974, 2024.
Honglin Cao, Zijian Zhou, Wenjie Wei, Ammar Belatreche, Yu Liang, Dehao Zhang, Malu Zhang, Yang Yang, and Haizhou Li. Binary event-driven spiking transformer. arXiv preprint arXiv:2501.0528 055904, 2025.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pp. 213-229. Springer, 2020.
Stefano Caviglia, Maurizio Valle, and Chiara Bartolozzi. Asynchronous, event-driven readout of posfet devices for tactile sensing. In 2014 IEEE International Sympossumm on Circuits and Systems (ISCAS), pp. 2648-2651. IEEE, 2014. doi: 10.1 1..110//cca.2221.66665533
Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF international ference on computer vision, pp. 357-366, 2021.
Xinyi Chen, Jibin Wu, Chenxiang Ma, Yinsong Yan, Yujie Wu, and Kay Chen Tan. Pmsn: A parallel multi-compartment spiking neuron for multi-scale temporal processing. arXiv preprint arXiv:2408.149111 2024.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
Mike Davies, Narayan Srinivasa, Tsung- Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. leee Micro, 38(1):82-99, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Lei Deng, Yujie Wu, Xing Hu, Ling Liang, Yufei Ding, Guoqi Li, Guangshe Zhao, Peng Li, and Yuan Xie. Rethinking the performance comparison between snns and anns. Neural networks,
121:294-307, 2020.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernlls to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVF conference computer vision and pallern recognition, pp. 11963-11975, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 /999, 2020.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34:21056-21069 2021.
10

---

Published as a conference paper at ICLR 2026
John A Gaynes, Samuel A Budoff, Michael J Grybko, Joshua B Hunt, and Alon Poleg-Polsky. Classical center-surround receptive fields facilitate novel object detection in retinal bipolar cells. Nature Communications, 13(1):5575, 2022.
2022.
Wulfram Gerstner and Werner M Kistler. Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press, 2002.
Jialong Guo, Xinghao Chen, Yehui Tang, and Yunhe Wang. Slab: Efficient transformers with simplified linear attention and progressive re-parameterized batch normalization. arXiv preprini arXiv:2405. 1582, 2024.
Meng-Hao Guo, Zheng-Ning Liu, Tai-Jiang Mu, and Shi-Min Hu. Beyond self-attention: External attention using two linear layers for visual tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5436-5447, 2022.
Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks toward deep residual learning. IEEE transactions on neural networks and learning systems, 36(2):2353- 2367, 2024.
Zihan Huang, Xinyu Shi. Zecheng Hao, Tong Bu, Jianhao Ding, Zhaofei Yu, and Tiejun Huang. Towards high-performance spiking transformers from ann to snn conversion. In Proceedings of the 32nd ACM international conference on multimedia, pp. 10688-10697, 2024.
Eugene M Izhikevich. Simple model of spiking neurons. IEEE Transactions on neural networks, 1466):159-11572 2003.
Angelos Katharopoulos, Apoory Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 3156-5165. PMLR, 2020.
Jing Liu, Zizheng Pan, Haoyu He, Jianfei Cai, and Bohan Zhuang. Ecoformer: Energy-saving attention with linear complexity. Advances in Neural Information Processing Systems, 35:10295- 10308, 2022a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12009-12019, 2022b.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9)1155-1667,, 1997.
Timothée Masquelier, Rudy Guyonneau, and Simon J Thorpe. Spike timing dependent plasticity finds the start of repeating patterns in continuous spike trains. PloS one, 3(1):e1377, 2008
Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34:23296-233 23308, 2021.
Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767)100-111) 2019.
Kaushik Roy. Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike- spike-based machine intelligence with neuromorphic computing. Nature, 575(7784)607--617 2019.
Yimeng Shan, Zhenbang Ren, Haodi Wu, Wenjie Wei, Rui-Jie Zhu, Shuai Wang, Dehao Zhang, Yichen Xiao, Jicyuan Zhang, Kexin Shi, et al. Sdtrack: baseline for event- based tracking via spiking neural networks. arXiv preprint arXiv: 2503.000330. 2025.

---

Published as a conference paper at ICLR 2026
Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on appli- cations of computer vision, pp. 3531-3539, 2021.
Xinyu Shi, Zecheng Hao, and Zhaofei Yu. Spikingresformer: Bridging resnet and vision transformer in spiking neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5610-5619, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Jingya Wang, Xin Deng, Wenjie Wei, Dehao Zhang, Shuai Wang, Qian Sun, Jieyuan Zhang, Han- wen Liu, Ning Xie, and Malu Zhang. Training-free ann-to-snn conversion for high-performance spiking transformer. arXiv preprint wrXiv:2508.077100 2025a.
Shuai Wang, Malu Zhang, Jingya Wang, Dehao Zhang, Yimeng Shan, Jieyuan Zhang, Yichen Xiao, Honglin Cao, Haonan Zhang, Zeyu Ma, et al. Bipolar self-attention for spiking transformers. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
Shuai Wang, Malu Zhang, Dehao Zhang, Ammar Belatreche, Yichen Xiao, Yu Liang, Yimeng Shan, Qian Sun, Enqi Zhang, and Yang Yang. Spiking vision transformer with saccadic attention. arXiv preprint arXiv:2502. 12677, 2025b.
Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.047666 2020.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 568-578, 2021.
Ziqing Wang, Yuetong Fang, Jiahang Cao, Qiang Zhang, Zhongrui Wang, and Renjing Xu. Masked spiking transformer. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pp. 1761-1771, 2023.
Wenjie Wei, Xiaolong Zhou, Malu Zhang, Ammar Belatreche, Qian Sun, Yimeng Shan, Dehao Zhang, Zijian Zhou, Zeyu Ma, Yang Yang. et al. Tp-spikformer: Token pruned spiking trans- former. In The Fourteenth International Conference on Learning Representations.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077-12099, 2021.
Boxun Xu, Hejia Geng, Yuxuan Yin, and Peng Li. Dista: Denoising spiking transformer with intrinsic plasticity and spatiotemporal attention. arXiv preprint vrXiv:2311.09377 2023.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao.
Focal attention for long-range interactions in vision transformers. Advances in Neural Information Processing Systems, 34:30008 2021.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware--fffiiiet training. arXiv preprint arXivv2312.06633, 2023.
Man Yao, Jiakui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer. Advances in neural information processing systems, 36:64043-640000 2023.
Man Yao, Jiakui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips, arXiv preprint arXiv:2404.0366 2024a.
Man Yao, JiaKui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, Bo XU, and Guoqi
Li. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. In The Tweifth International Conference on Learning Representations, 2024b. Rtll.httss://ppenee
12

---

Published as a conference paper at ICLR 2026
Man Yao, Xuerui Qiu, Tianxiang Hu, Jiakui Hu, Yuhong Chou, Keyu Tian, Jianxing Liao, Luziwei Leng, Bo Xu, and Guoqi Li. Scaling spike-driven transformer with efficient spike firing approxi- mation training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilat- eral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 325-341. 2018.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10819-10829, 2022.
Dehao Zhang, Shuai Wang, Yichen Xiao, Wenjie Wei, Yimeng Shan, Malu Zhang, and Yang Yang. Memory-free and parallel computation for quantized spiking neural networks. In ICASSP 2025- 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025a.
Dehao Zhang, Malu Zhang, Shuai Wang, Jingya Wang, Wenjie Wei, Zeyu Ma, Guoqing Wang, Yang Yang, and Haizhou Li. Dendritic resonate-and and-fir neuron for effective and efficient long sequence modeling. arXiv preprint arXiv:2509. 17186, 2025b.
Jieyuan Zhang, Xiaolong Zhou, Shuai Wang, Wenjie Wei, Hanwen Liu, Qian Sun, Malu Zhang, Yang Yang, and Haizhou Li. Unveiling the spatial-temporal effective receptive fields of spiking neural networks. arXiv preprint arXiv:2510.2 1402, 2025c.
Malu Zhang, Shuai Wang, Jibin Wu, Wenjie Wei, Dehao Zhang, Zijian Zhou, Siying Wang, Fan Zhang, and Yang Yang. Toward energy-efficiee spike-based deep reinforcement learning with temporal coding, IEEE Computational Intelligence Magazine, 20(2):45-55 57, 2025d.
Shuoxi Zhang, Hanpeng Liu, Stephen Lin, and Kun He. You only need less attention at each stage in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6057-6066, 2024a.
Yu Zhang, Songlin Yang, Rui-Jie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, et al. Gated slot attention for efficient linear-time sequence modeling. Advances in Neural Information Processing Systems, 177710870-11689,, 2024b.
Hanle Zheng. Zhong Zheng, Rui Hu, Bo Xiao, Yujie Wu, Fangwen Yu, Xue Liu, Guoqi Li, and Lei Deng. Temporal dendritic heterogeneity incorporated with spiking neural networks for learning multi-timescale dynamics. Nature Communications, 15(1))277, 2024.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 633-641,
Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Zhengyu Ma, Han Zhang, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual learning for transformer-bast spiking neural net- work. arXiv preprint 1rX14 2023a.
Chenlin Zhou, Han Zhang, Zhaokun Zhou, Liutao Yu, Liwei Huang, Xiaopeng Fan, Li Yuan, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. Qkformer: Hierarchical spiking transformer using qk attention. arXiv preprint arXiv:2403. 1144003..55... 16552, 2024.
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, YAN Shuicheng, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. In The Eleventh Interna- tional Conference on Learning Representations, 2023b.
Maria Zontak and Michal Irani. Internal statistics of a single natural image. In CVPR 2011, pp- 977-984, 201 1. doi: 10. 1.15pP.9000111999400
13

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODEL
In the preparation this manuscript, we employ a Large Language Model (LLM) solely to aid and polish the writing. The LLM is used exclusively for language polishing, grammar correction, and improving the clarity and readability of the text. All technical ideas, methods, experiments, and conclusions presented in this paper are original contributions of the authors.
B TRAINING STABILITY OF LRF-DYN
To demonstrate the training stability of our LRF-Dyn, plot the convergence curves of Spikformer and QKFormer on CIFAR- 100. The results are as shown in Fig. 6
Spiksoemer
(2) Spikformer LRF- LRF-Dyn
Train
Teit Loos
Loos
Test Loss
Tea Lews
Figure 6: Comparison convergence performance across different models.
can be observed that our model exhibits a stable convergence pattern (Xu et al. 2023 which further demonstrates the effectiveness of our approach.
C PROOF THEOREM
In this section, analyze the problem from the perspective of expected distance. Under an identical distance-bias assumption, SSA exhibits an expected distance that is almost global when compared with VSA. The LRF-SSA method partially alleviates the impact of this global receptive field.
C.I DEFINITION OF EXPECTED RECEPTIVE RADIUS
Given sequence r = {II,I2-- T2.... ........ Zzy of length N, we compute the expected receptive radius to measure the receptive field size under different attention paradigms, which defined as follows:
N
== = Σ A
A =
= -
where âif denotes the normalized attention weight from position to position j, and A represents the distance between and j. Considering the spatial continuity of image sequences, we adopt the Manhattan distance as the metric. In this way, the expected receptive radius μ provides an effective measure the receptive field across different attention paradigms. larger value of y indicates larger effective receptive radius, corresponding to more global receptive field.
C.2 COMPARISON OF VSA, SSA AND LRF-SSA
Empirical studies natural image statistics have demonstrated that patch similarity decreases as spatial distance increases (Zontak & Irani 2011). For simplicity, model this property linear decay with respect to distance. Specifically, for given pair (g;, k2), their similarity can be defined:
q.T k2
BA,
Accordingly, for SSA, the expected receptive radius can be represented as follows:
q. kj
exp( -BA]
x exp(-3li j1}, E[A Σ Dai A ΣN exp -3t
(3)
= o. kg
14

---

Published as a conference paper at ICLR 2026
As oo, the expected receptive radius of VSA can be defined as follows:
exp(-3) = (
(4)
1 exp(-8)
Therefore, VSA exhibits localized receptive field. In contrast, for SSA, since the softmax operation is omitted, its attention weight Okij and the corresponding expected receptive radius can be defined:
x - 3A)+ B[A] = Σ Dai = (a - 34)+
s5a i
(5)
Ei 34)+
As - oc, the expected receptive radius of VSA be defined as follows:
(N (3a - 3(2N - 1)) ((N).
(6)
3(2a B(N 1))
Therefore, SSA exhibits a broader receptive radius than VSA.
As SSA variant with local receptive fields, the attention weight of SSA is not only determined by agjy but also incorporates an additional contribution from rij. Hence, the LRF-SSA attention weight can be expressed convex combination:
aij
Ira-ssa (1 - A) ofj Argy
E Im-ssl = (1 A) Mssa Pry
where rij 0 denotes the local weights within the neighborhood f2 {j j1 S d), and Hy S Pssa- Therefore, LRF-SSA exhibits an effective receptive field no larger than that of SSA, and this receptive field becomes increasingly local À increases. In the extreme case where À > 1, LRF-SSA tends toward a highly localized receptive field. Therefore, the receptive field radii of the three methods follow the relation:
DVSA S MIRP-SSA JASSA
Therefore, the proposed method exhibits local modeling capabilities more similar to SSA, while avoiding the use of the softmax operation.
D PROOF OF THEOREM
In this appendix, provide detailed theoretical analysis of the entropy properties of different self-attention mechanisms. Benefiting from the softmax operation, VSA exhibits a sharp and low- entropy distribution, concentrating most of the attention mass few neighboring positions. In contrast, SSA produces a more dispersed distribution, leading to higher entropy.
D.1 INFORMATION ENTROPY
From an cntermatiionthhorettee perspective, the entropy of the attention distribution provides nat- ural measure of its uncertainty and sharpness. Specifically, for given attention weight vector (I....., I, the entropy is defined as:
H(x) =
Σ xx log
where we use natural logarithms. A larger entropy indicates a more uniform and smoother distribu- tion of attention scores, whereas smaller entropy corresponds to more concentrated and sharper allocation of attention.
D.2 ENTROPY OF VSA, SSA AND LRF-SSA
As shown in Eq. 17, the similarity decays linearly with distance A. Therefore, for VSA, the induced truncated geometric distribution over distances (ignoring multiplicity) is:
exp -
(1-r) r)r
Pesa A)
exp(-3) E (0,1),
(10)
ΣN-1 exp( -Bt}
1 N.
15

---

Published as a conference paper at ICLR 2026
discretized over A O....,N N 1. The corresponding entropy admits the closed form:
1 1-r
1 N + (N- 1)r
Hesa (N, log and in the infinite-length limit, we obtain:
N logr,
(11)
1 --r
(oo, log(1 - 1-r logr O(1).
(12)
Thus VSA yields a bounded, low-entropy distribution independent of N.
In contrast to the truncated geometric distribution of VSA, SSA employs linearly decaying weights without softmax normalization. Specifically, the unnormalized score modeled as:
w( ((A)
3A,
0 H
(13)
where the constraint ensures non-negativity across all positions. After normalization, the distance distribution becomes:
a - a
P A
A = N
(14)
aN B N.N-NII
which leads to the entropy:
BA
3) Σ A=
[log(a 3A) - log Sol
So = = B N(N-1)
(15)
So
For the special case 3 0, Pasa reduces to the uniform distribution and the entropy attains its
maximum Hşa = log N. At the other extreme, when = a/(N - 1), the distribution becomes triangular and the entropy satisfies Hya log N + O(1). Therefore, the entropy of SSA scales as:
H
(log N),
(16)
indicating a high-entropy distribution that spreads broadly across the sequence and corresponds to a nearly global receptive field.
Finally, we analyze the entropy of LRF-SSA. LRF-SSA can be formulated as a convex combination of SSA and concentrated local distribution R₁
pira
(1 - A.) (A) A,))
A E [0.
(17)
The entropy of LRF-SSA is thus:
N-1
H (Plraass) Σ
(1)))) A; log [1 P: + AA)
(18)
Although a closed-form solution is difficult to obtain, we can derive an information iiformation-thhoretiie upper bound by introducing an auxiliary Bernoulli variable Bernoulli(A,) and applying the chain rule of entropy:
H (Plta-ss) < h(A,) + (1- (P + HHH))
(19)
where h(.)) denotes the binary entropy. Since R2 is designed to be more localized, typically have (R.) H (Psa), which implies:
H (p < ((PS)).
(20)
Therefore, LRF-SSA consistently produces lower entropy than SSA, and the degree of reduction is controlled by the mixing coefficient A;; in other words, LRF-SSA interpolates between the global high-entropy behavior of SSA and the localized low-entropy property of VSA, providing a flexible trade-off between global coverage and local sharpness.
16


Reviews:
[{"review_id": "Mkz4JcsrMq", "reviewer": "Reviewer_LacD", "summary": "This paper tackles two critical limitations in Spiking Transformers: the performance gap compared to their Artificial Neural Network (ANN) counterparts and excessive memory overhead. The authors argue these issues stem from the unfocused global attention of Spiking Self Attention (SSA) and the high cost of storing attention matrices. Inspired by biological visual neurons, they introduce LRF-Dyn, a novel method that enhances SSA by integrating a Localized Receptive Field (LRF) mechanism. This addition enables the model to focus on neighboring regions, thereby improving its local modeling capabilities. Furthermore, LRF-Dyn reformulates the attention computation to approximate the charge–fire–reset dynamics of spiking neurons, which significantly reduces memory requirements during inference. Extensive experiments confirm that LRF-Dyn successfully lowers memory overhead while delivering substantial performance gains on visual tasks, establishing it as a key advancement for creating more practical and energy-efficient Spiking Transformers.", "questions": "1. Under identical configurations, how does LRF-Dyn perform when employing different spiking neuron such as LIF, ALIF or DH-LIF?\n2. Can the proposed module maintain its effectiveness and generality when evaluated on additional neuromorphic datasets, such as CIFAR10DVS?\n3. How does the number of dendrites affect the model's performance and results?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper demonstrates significant strengths across all key dimensions. Its originality is high, stemming from a novel problem analysis that pinpoints the lack of local modeling and high memory cost in Spiking Self Attention (SSA) as key barriers . The proposed LRF-Dyn method is a creative and well-motivated solution, uniquely integrating a Localized Receptive Field (LRF) mechanism to enhance local feature capture and, more innovatively, reformulating the attention mechanism to mimic neuronal dynamics, thereby reducing memory overhead. The quality of the work is excellent, substantiated by a solid theoretical foundation, including theorems that formally prove LRF-SSA preserves the desirable local attention and low-entropy properties of standard attention mechanisms . These theoretical claims are rigorously supported by extensive experiments that confirm both performance improvements and reduced memory consumption. The paper is presented with outstanding clarity, logically progressing from a well-defined problem to a tailored solution, with concepts effectively illustrated through figures and formal proofs. Finally, the significance of this work is substantial; by directly tackling the critical performance and efficiency bottlenecks of Spiking Transformers, it provides a practical and effective component that could accelerate the deployment of energy-efficient vision models on resource-constrained edge and neuromorphic hardware.", "weaknesses": "1. To substantiate the claimed benefits of the proposed method, such as the reduction in memory overhead, the authors should provide concrete quantitative data, for instance, measurements of energy consumption and inference latency.\n2. To further justify the necessity of multi-dendritic neurons, the authors are encouraged to include comparisons under identical conditions with other spiking neuron (e.g., LIF, ALIF or DH-LIF) that serve as the core neurons in LRF-Dyn.\n3. Given that LRF-Dyn exhibits more complex dynamical behavior, the authors should consider providing an algorithmic description or pseudocode to facilitate a clearer understanding of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xWB7A1WssV", "reviewer": "Reviewer_Cfzh", "summary": "This paper addresses two critical challenges in existing SNN-Transformer architectures: the degradation of local receptive fields and high inference memory cost. The authors present empirical evidence through visualization and statistical analysis, and further propose LRF-Attn and LRF-Dyn to effectively address these issues. Experimental results on both image classification and segmentation tasks demonstrate consistent performance gains, indicating the method’s effectiveness and generality.", "questions": "1. While the authors have validated the proposed module on mainstream image classification and segmentation tasks, it would be valuable to examine whether comparable effectiveness can be achieved on more challenging NLP tasks [1].\n2. Does the proposed LRF-Dyn introduce additional training overhead compared to mainstream CNN-based SNNs [2] and Transformer-based SNNs [3]?\n3. Is the local receptive field essential to the effectiveness of the LRF-Dyn method? The authors are encouraged to include experiments to verify this.\n[1] Spikelm: Towards general spike-driven language modeling via elastic bi-spiking mechanisms, ICML 2024.\n[2] Deep residual learning in spiking neural networks, NeurIPS 2021.\n[3] Scaling spike-driven transformer with efficient spike firing approximation training, T-PAMI 2025.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies key limitations in existing SNN-Transformer architectures and substantiates these claims through comprehensive empirical observations and quantitative analyses, making the motivation convincing.\n2. Extensive experiments are conducted on recent state-of-the-art Transformer architectures, showing consistent and notable improvements in both image classification and segmentation tasks.\n3. The authors provide an analysis of the local receptive field degradation in SNNs from both receptive field radius and information entropy perspectives, and convincingly demonstrate the effectiveness of the proposed approach.", "weaknesses": "1. The adoption of multi-dendritic neurons increases training complexity, potentially leading to higher computational costs on large-scale datasets such as ImageNet.\n2. The presentation of inference memory requirements could be improved. Providing a tabular summary for LRF-Attn and LRF-Dyn would make the comparison clearer and more intuitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CFedVFxx0T", "reviewer": "Reviewer_Ttxv", "summary": "This paper addresses the issue of unfocused global attention and high memory overhead in spiking transformers. To resolve these problems, this paper proposes Local Receptive Field Spiking Self-Attention (LRF-SSA), which strengthens local attention by assigning higher weights to spatially adjacent positions. Building on LRF-SSA, the paper also introduces LRF-Dyn, a method that eliminates the need for explicit attention-matrix storage and reduces memory overhead.", "questions": "Please compare existing methods and explain why LRF is more effective than these approaches in addressing the limitations of SSA.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. Theoretical analyses are provided to support the proposed LRF-SSA mechanism.\n2. The proposed LRF-Dyn maintains performance comparable to LRF-SSA while reducing memory requirements during inference.", "weaknesses": "1. The motivation behind LRF-SSA requires further validation. The paper points out a mismatch in attention patterns between SSA and VSA: as shown in Fig. 1(a) and Fig. 2, VSA emphasizes local relations, while SSA exhibits unfocused global attention. However, this lack of local attention in SSA is a result of its inherent limitations rather than the root cause of its shortcomings. Therefore, enhancing local attention may not fundamentally resolve the issue. In fact, this issue stems from information loss due to the binary nature of spikes in SSA. Previous research [1] has identified this problem and proposed solutions that effectively focus on key features. As a result, the motivation for this paper needs reconsideration and further validation.\n2. The proposed method achieves only marginal accuracy improvements on the ImageNet-1k classification task, particularly on the state-of-the-art architecture QKFormer, where the accuracy gain is less than 0.5%.\n\n[1] Xiao, Y., et al. (2025). Rethinking Spiking Self-Attention Mechanism: Implementing a-XNOR Similarity Calculation in Spiking Transformers. Proceedings of the Computer Vision and Pattern Recognition Conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rUp9Yg9HqK", "reviewer": "Reviewer_MjDS", "summary": "### 1. Research Problem\n\n*  Existing Spiking Transformer models underperform compared to their ANN counterparts, primarily due to limitations in the spiking self-attention (SSA) mechanism which lacks effective attention localization.\n\n*  SSA requires explicit storage of large attention matrices (such as QK and KV), resulting in significant memory overhead, especially during inference on resource-constrained hardware.\n\n*  Without the softmax operation, SSA generates nearly uniform attention distributions, losing the ability to focus on spatially relevant regions, which is critical in visual tasks.\n\n\n### 2. Proposed Method\n\n* LRF-SSA (Local Receptive Field Self-Attention):\n\n  * Enhances SSA by incorporating multi-scale dilated convolutions that introduce spatial bias through local receptive fields.\n  * Adjusts attention weights to favor nearby tokens, improving local modeling capability while preserving spike-driven characteristics.\n  * The attention weights are computed as a convex combination of SSA attention and a local convolutional kernel:\n    α_lrf-ssa = (1 - λ) * α_ssa + λ * r_ij\n\n* LRF-Dyn (Local Receptive Field with Neural Dynamics):\n\n  * Reformulates attention using principles of spiking neuron dynamics, simulating membrane potential updates and dendritic processing.\n  * Avoids explicit computation and storage of QK or KV matrices by using accumulated membrane states, reducing memory complexity.\n  * The attention computation becomes a charge–fire–reset-like process that aligns with the biological plausibility of SNNs.\n\n\n### 3. Theoretical Contributions\n\n* Entropy-based attention analysis:\n\n  * Demonstrates that LRF-SSA produces lower-entropy attention distributions than SSA, which better reflect the selective nature of softmax-based VSA.\n\n* Expected receptive field radius: Introduces a formal measure of attention locality and proves that LRF-SSA reduces the average receptive field size compared to SSA, while remaining closer to the behavior of VSA.\n\n* Neural dynamics formulation of attention: Establishes an equivalence between spiking attention and dynamic state updates in multi-dendritic neurons, paving the way for biologically grounded attention mechanisms.\n\n* Memory and computation complexity: Reduces inference-time memory from O(d²) to O(kd), with k representing the number of dendritic branches, offering theoretical scalability for edge deployment.\n\n\n### 4. Experimental Results\n\n* Image classification (ImageNet-1k): LRF-SSA improves top-1 accuracy by up to 1.24% with negligible parameter increase (<0.2M). LRF-Dyn achieves similar or slightly reduced accuracy (within 0.1%) while cutting memory usage to O(kd).\n\n* Semantic segmentation (ADE20K): LRF-SSA improves mIoU by 2.6% and LRF-Dyn by 1.8% over the SSA baseline, showing stronger spatial discrimination with fewer parameters than attention-free baselines like ResNet.\n\n* Ablation studies (CIFAR-100): Performance scales with number of convolution kernels (larger Ω yields higher accuracy), validating the contribution of local spatial modeling.", "questions": "* Can the authors provide a more rigorous theoretical analysis of LRF-Dyn? Specifically, is there a formal approximation bound or information-preserving guarantee that connects LRF-Dyn to standard attention mechanisms?\n\n* How does the proposed method handle long-range temporal dependencies inherent to spiking inputs? Would LRF-Dyn remain effective in tasks requiring fine-grained temporal precision, such as DVS-based continuous input streams?\n\n* Have the authors considered evaluating the method on neuromorphic datasets?\n\n*  Does the LRF-Dyn formulation introduce stability issues during training, such as exploding or vanishing membrane potentials? Can the authors provide convergence plots or empirical analysis to support its robustness?\n\n* Please add some missing citations, eg:  https://arxiv.org/abs/2311.09376", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper introduces a novel perspective on spiking self-attention by recasting attention as a neural dynamic process, drawing from the charge–fire–reset behavior of biological neurons. This is a conceptually original contribution that bridges the gap between algorithmic attention mechanisms and neurophysiological processes.\n\n  * The integration of localized receptive fields into spike-based self-attention (LRF-SSA) is a creative fusion of biologically inspired modeling and modern deep learning practice. It departs from the conventional global attention view in SNNs and enables spatial bias without softmax.\n\n  * The work removes a critical limitation in existing spiking Transformers by eliminating the need for explicit QK/KV matrix storage, allowing for scalable inference on neuromorphic or memory-limited hardware—an underexplored but increasingly relevant design constraint.\n\n  * The experimental content is detailed and clear, and the effectiveness of the method is verified on different tasks.", "weaknesses": "* Limited theoretical depth for LRF-Dyn: While the paper presents a compelling neuro-dynamic approximation of self-attention, the theoretical treatment of LRF-Dyn remains less rigorous than that of LRF-SSA. Specifically, the membrane potential update formulation lacks an explicit connection to the global attention weighting mechanism it replaces.\n\n\n* Experiments limited to static vision benchmarks: Despite the motivation from real-time and neuromorphic computing, the evaluation is restricted to frame-based, static datasets (ImageNet-1k and ADE20K). These benchmarks do not fully capture the event-driven potential or energy-efficiency implications of spiking systems. The paper would benefit from experiments on neuromorphic datasets (e.g., N-Caltech101, DVS Gesture, or N-Cars) to substantiate claims of biological plausibility and resource efficiency.\n\n* Experimental performance improvement is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)