Bipolar Self-attention for Spiking Transformers
Authors: Shuai Wang
Malu Zhang
Jingya Wang
Dehao Zhang
Yimeng Shan
Jieyuan Zhang
Yichen Xiao
Honglin Cao
Haonan Zhang
Zeyu Ma
Yang Yang
Haizhou Li
Venue: NeurIPS 2025 spotlight ()
Keywords: Spiking Neural Networks
Spiking self-attention
Spiking transformer
Primary area: neuroscience_and_cognitive_science

Abstract: Harnessing the event-driven characteristic, Spiking Neural Networks (SNNs) present a promising avenue toward energy-efficient Transformer architectures. However, existing Spiking Transformers still suffer significant performance gaps compared to their Artificial Neural Network counterparts. Through comprehensive analysis, we attribute this gap to these two factors. First, the binary nature of spike trains limits Spiking Self-attention (SSA)’s capacity to capture negative–negative and positive–negative membrane potential interactions on Querys and Keys. Second, SSA typically omits Softmax functions to avoid energy-intensive multiply-accumulate operations, thereby failing to maintain row-stochasticity constraints on attention scores.
To address these issues, we propose a Bipolar Self-attention (BSA) paradigm, effectively modeling multi-polar membrane potential interactions with a fully spike-driven characteristic. Specifically, we demonstrate that ternary matrix multiplication provides a closer approximation to real-valued computation on both distribution and local correlation, enabling clear differentiation between homopolar and heteropolar interactions. Moreover, we propose a shift-based Softmax approximation named Shiftmax, which efficiently achieves low-entropy activation and partly maintains row-stochasticity without non-linear operation, enabling precise attention allocation. 
Extensive experiments show that BSA achieves substantial performance improvements across various tasks, including image classification, semantic segmentation, and event-based tracking. These results establish its potential as a fundamental building block for energy-efficient Spiking Transformers.

Bipolar Self-attention for Spiking Transformers
Shuai Wang's Malu Zhang 1,2 Jingya Wang', Dehao Zhang', Yimeng Shan', Jieyuan Zhang', Yichen Xiao', Honglin Cao', Haoman Zhang', Zeyu Ma', Yang Yang', Haizhou Li
University of Electronic Science and Technology of China Shenzhen Loop Area Institute, The Chinese University of Hong Kong (Shenzhen)
Abstract
Harnessing the event-driven characteristic, Spiking Neural Networks (SNNs) present promising avenue toward energy-efficient Transformer architectures. However, existing Spiking Transformers still suffer significant performance gaps compared to their Artificial Neural Network counterparts. Through comprehensive analysis, we attribute this gap to these two factors. First, the binary nature of spike trains limits Spiking Self-attention (SSA)'s capacity capture negative- and positive-negatiin membrane potential interactions on Querys and Keys. Sec- ond, SSA typically omits Softmax functions to avoid meery--ineensin multiply- accumulate operations, thereby failing to maintain row-stochasticity constraints on attention scores. To address these issues, propose Bipolar Self-attention (BSA) paradigm, effectively modeling multi-polar membrane potential interactions with a demonstrate that ternary matrix
fully spike-driven characteristic. Specifically, multiplication provides a closer approximation to real-valued computation on both distribution and local correlation, enabling clear differentiation between homopolar and heteropolar interactions. Moreover, we propose shift-based Softmax approx- imation named Shiftmax, which efficiently achieves low-entropy activation and partly maintains row-stochasticity without non- linear operation, enabling precise attention allocation. Extensive experiments show that BSA achieves substantial performance improvements across various tasks, including image classification, se- mantic segmentation, and event-based tracking. These results establish potential as fundamental building block for mergy-efffcient Spiking Transformers.
Introduction
As the core computational unit of Transformers 5 15 II
32 91. self-attention mechanism dy-
namically models the global dependencies among sequence elements, overcoming the long-range dependency challenges 33 [O]. However, its' O(N2d) computational complexity [18 48 incurs exponential rise during both training and inference, restricting its application in many resource- constrained environments. Consequently, how to develop energy-efficient and high-performance Transformers remains a critical research focus.
Spiking Neural Networks (SNNs) 23 12 have gained significant attention due to their brain-inspired dynamics 17 24
Spiking neurons fire discrete spikes only when activated, remaining silent other- wise. Compared with Artificial Neural Networks (ANNs) that rely on multiply-accumulan (MAC)
computation, the spike- -riven mechanism [61 43 44 in SNNs supports sparse accumulate (AC) operations [2]. Such sparse spike- spike-based computation 20
[6] delivers significant power efficiency,
particularly neuromorphic platforms such Tianjic [26
22 and Loihi 66 25 Recently,
numerous researchers focus on developing Spiking Transformers, including Spikformer [70]. Spik-
ingformer (67), Spike-driven Transformers [51 49 52. SpikingResforrerr [30], and QKformer [68]
CCoresponding author: maluzhang ues.c.edu.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

These approaches enhance the performance ceiling of SNNs in various tasks 664 demonstrating that Spiking Transformers achieve trade-off between high performance and efficiency.
45
36
62
34
(a) Vanilla Self-attention (VSA)
(b) Spiking Self-attention (SSA)
0.5
Linor & LN
Lincar LN
Lincar & LN
Cony
Core BN
VSA
K
Q
A
0.4
axd
M x
X
X
0.2
Socre'
Q X
Score Softmax K
Miso
E.04
Score
X K
6:1
12.3
2
1L*
0.6
4:58
2
E:5
0.24
0.13
Z
1a
60
10
11X
0.0
Figure I: Comparison of Vanilla Self-attention (VSA) and Spiking Self-attention (SSA) mechanisms. Score' contains rich polarity information and Score maintains strict cow-stochasticity via Softmax in VSA, while Score in SSA lacks negative polarity and row-stochasticity constraints.
As a core component of Spiking Transformer architectures, the Spiking Self-ttention (SSA) 770 51 computational paradigm represents the critical factor limiting their performance ceiling. As shown in Fig I Vanilla Self-attention (VSA) in ANNs effectively captures magnitude and multi-polarity information by computing Score' through query (Q) and key (K) correlation. Meanwhile, calculates the Score using the Softmax function under row-stochasticity constraint. In contrast, SSA in SNNs operates with binary spike trains (0 1), which not only sacrifices quantization precision but also interactions. Furthermore, to maintain energy efficiency, SSA typically omits Softmax function, resulting in severely imbalanced how-stochasticity in Score. These issues limit SSA's ability to effectively compute attention allocation between Q and K, causing SSA more like simplified Token Mixer [54 55). Therefore, how to overcome these limitations is crucial for pushing Spiking Transformers beyond their current performance bottlenecks.
disregards negative-negatiee and
In this paper, we propose a Bipolar Self-attention (BSA) paradigm to effectively address these issues. Unlike the SSA paradigm that exclusively captures positive-positive Q-K interactions, BSA employs ternary spiking neurons [4 35) to comprehensively process both homopolar and heteropolar interaction patterns in Q-K correlation computation. We theoretically demonstrate that ternary matrix multiplication more closely approximates real-valued computation in terms of both distributional similarity and local correlation than binary part. Moreover, we propose the innovative Shiftmax method, which approximates Softmax's low-entropy activation characteristies and cow-stochasticity constraints through energy-efficient bit-shift operations. Finally, we conduct extensive experiments across diverse tasks including image classification 770, 68), semantic segmentation (49 52 and event-based tracking [58) 28). consistently demonstrating that BSA delivers substantial performance improvements. The main contributions of our work are outlined as follows:
We first identify two limitations of Spiking Self-attention: (1) binary matrix products exclusively capture positive- positive-positive correlations while neglecting negative-negative and positive-negative polarity features, leading to a complete loss of polarity information. (2) without Softmax, attention scores across differrnnt rows exist on incomparable scales, render- ing attention allocation ineffective. These deficiencies prevent SSA from fully harnessing the potential of the self-attention mechanism.
We propose Bipolar Self-attention (BSA) to overcome these limitations. BSA employs teraay matrix products to extract Q-K correlations, comprehensively process different polarity interaction patterns. In addition, we propose a Shiftmax method to approximate Softmax that achieves low-entropy activation and maintains partial row-stochasticity without non-linear operations, enabling precise attention allocation.
Extensive experiments demonstrate that BSA achieves significant performance improve- ments across various advanced Spiking Transformers on ImageNet-1 K. Furthermore, our method establishes state-of-the-art performance in both semantic segmentation and event- based tracking tasks compared existing SNNs approaches.

---

2
Related Work
Recently, growing attention has been paid to energy-efficien Spiking Transformers 4 50]. For instance, Spikformer [70] first proposes Spiking Self-attention computation, establishing the first Spiking Transformer. Building on this, Spikingformer [67] introduces hardware- frienndl residual learning architecture that avoids non-spike computations in SNNs. Then, the Spike-driven Trans- formers (SDT-VI) [5I] incorporates the Hadamard product into the self-attention module, achieving a fully spike-driven mechanism. Furthermore, SpikingResformer (30) integrates a Dual Spike self- attention module, enhancing both performance and energy efficiency, Recently, QKformer [83 and Spike-driven Transformer-V: V3 [52] both significantly elevate the performance ceiling of Spiking Trans- formers. Nonetheless, these models primarily utilize self-attention as a token mixer [54], without paying attention to exploring effective similarity calculations suited to spike trains [60 There- fore, designing an innovative spiking self-attention mechanism that fully leverages self-correlation computation under spike-driven characteristics is crucial for further advancement.
3 Preliminary
3.1 Vanilla Self-attention
The self-attention mechanism effectively captures global dependencies within sequences by dynam-
ically allocating attention across tokens (33 29 13]. Given an input matrix X E Raxd, where denotes sequence length and d represents embedding dimensionality, the mechanism initially projects X into query Q, key K, and value V matrices via distinct parameter matrices WQ. W K, and Wy:
X LN(X),
Q Linear(X), X), K == Linear(X) (X), V= =ineea (X).
(1)
Here, LN denotes layer normalization, and Linear refers to a fully connected layer. To determine contextual relationships, the model computes token-wise similarities through dot-product operations between Q and K, generating an attention score matrix:
Score' Vd
Score' Q KT, Score Softmax
Attn Score V.
(2)
The mechanism applies a scaling factor v to mitigate gradient instability issues. Subsequently. the Softmax function normalizes these scaled Score to probability distribution, ensuring attention weights sum to unity while emphasizing relevant tokens and suppressing irrelevant ones.
3.2 Spiking Self-attention
Building on VSA, Spikformer 77 introduced SSA. For an input matrix X E Rxd queries Q, keys K, and values V are first computed via learnable weight matrices and then converted into spike trains by binary spiking neurons for subsequent processing. The dynamics of these neurons are given by:
U(t + 1] [t) + X(t 1],
Sit + 1] @ V(n,,,
(4)
[tt + 1] = S Sit + [-1] +] L](1 SIt 1))-
(5)
[ + 1] denotes the input current, while H[] and U({] represent the pre-synaptic and post-synaptic membrane potentials, respectively. The Heaviside function ( is employed for spike generation. If a spike occurs (Sit + 1] = 1). H [t] resets to Vreseti otherwise, U[t + 1] decays with a time constant r and feeds into [t 1]. Here spiking neuron layer denoted S.N(.). which takes X as input and produces the spike S(t + 1] as output. Then the Q K and V in SSA can be decribed as:
Q SNN (BN(BB(Conv)) (X)), K (N((N((on)2((X))
(B (X)), (6)
where K, V Txxxd BN(-) denotes batch normalization and Conv(.) refers to convolution operation. Unlike VSA, SSA omits the Softmax operation while retaining scaling to control the magnitude of the attention output. The attention output Attn is obtained by performing matrix multiplication of the spiking Q and K, scaled by factor 8, and then convert to spike trains:
Score s- Q 10*'* KT, Attn = S.N x V).
(7)
Therefore, SSA provides an energy efficcent self-attention computation paradigm without any nonlin- ear operations. allows the ordering of the Q, K, and V matrices to be flexibly adjusted as needed, enabling the Spiking Transformer to capture global dependencies efficiently.
3

---

4 Method
4.1 Problem Analysis for Spiking Self-attention
As previously discussed, SSA reduces computational costs by eliminating nonlinear computations and MAC operations. However, there remains a significant performance gap compared to VSA. We thoroughly analyze the reasons and attribute them to the following parts: (1) binary Q-K Matrix multiplication cannot effectively capture polarity correlations, and (2) eliminating Softmax removes the attention score's 's row-stochasticity constraints. The details are elaborated in the following.
Polarity Information Interaction Loss
Erroneous Self-attention Score Matrix
misos (+-)
miss (1:)
miss (+:)
miss
(+)
miss,
bv
6.7 MAR
K AURZ
-11
AXMLDIS Puters
3.86
Query
ANNS
match (++)
match (+++
X0
Spake Fiskng
Martor Malipiicatzn
Spdit
Spkc
Figure 2: Problem analysis. (a) The binary activation characteristic of spike trains can only preserve (+,+) interactions on the membrane potentials of Q and K, while all others are ignored. (b) The correlation matrix between binary and K in SNNs may be completely different from its' VSA counterpart in ANNs.
4.1.1 Incomplete Polarity Information Modeling
In VSA, Q and K are represented by continuous real-valued vectors, enabling matrix multiplication can efficiently calculate the correlation between Q and K. However, SSA typically binarizes the membrane potential information of Q and K, limiting its representation to binary values (0 or I). This spike-driven approach not only narrows the precision of Q-K interactions, but also undermines the self-attention mechanism's 's capacity to detect vector polarity relationships. Consider ΩyyA = representing correlation types in VSA and USSA {(b). b2)
{(a1, 02) | a, {+, +,i = 1,
b, (0, 1), = 1.2] denoting binary states in SSA, with á : Rd - 10,
binarization
mapping. Let P = {Bo, B,} partition DSSA where B, = (00, 0). (0, 1). (1,0), B₁ = (1,
We establish surjection g flysa + P that fundamentally characterizes the polarity information loss during the binarization transformation process, such that:
Case 1: & (((+,,+ +))) Bi, Case
(((+)-))((-+))(((-))))))
(8)
Eq demonstrates that SSA completely discards the ability to compute negative-negativ polarity or mixed polarity correlations. Due to the sparse activation characteristics SNNs, the amount of discarded information far exceeds what is retained, resulting in substantial information loss. As shown in Fig 2 a). the green-boxt boxed region represents the positive polarity information that SSA can
preserve, while the red- boxed regions indicate the entirely neglected (-,-) and (+. -) polarity information. More importantly, SSA may display entirely disparate attention regions from VSA as shown in Fi.. Thus, devising efficient mechanisms to embed polarity information offers promising avenue for improving SSA's performance ceiling.
4.1.2 Missing Softmax Operation
SSA not only suffers from loss of polarity information but also neglects the critical Softmax component. In VSA, Softmax operation (37] serves two pivotal roles: first, amplifying highly relevant through low-entropy activation effect while suppressing low-value regions; second, ensuring comparability of Score across varying scales via sow-stochasticity constraints. These mechanisms ensure that Softmax highlights key features while maintaining the comparability of the Score.
By contrast, SSA's binary matrix multiplication preserves limited low-entropy activation properties (detailed in the Appendix A and entirely lacks row-stochasticity constraints. Consequently, attention

---

scores from different query vectors exist on disparate numerical scales. The disparity in scales makes difficult effectively compare the relative degree of elements across different rows during the allocation of attention to V. Therefore, how to allocate global attention on comparable scales will be another potential area for improvement.
(a) Bipolar Self-attention (BSA)
(c) Distributions Score'for RMP, BMP and
Marte Prodact Ur-dditt
Vin 1.0, 7: V/s
+1.0
8: 0.5, 7: 20.5
CND
Temary Spikigg Neason
27.76
AvgeO.100
27.78
TMP Tereay Maris Predect
Arti
(b) Bit-shift Operation to Apply Score to the Real-value V
7.89
Shiftmax(X)
2 3 1
Score
V
Attn
Score'
Score
Hardware sit-shift Operation
0
-2
-
2-1
2-3
0
3.00
-1
2
2-5
-2 -- 2-1
2-3
Figure 3: (a) Overall structure of Bipolar Self-attention. (b) Detailed process of Shiftmax. It can directly shift the membrane potential of for efficient attention allocation. (c) Distribution of three types of matrix product different thresholds (Vr.).
4.2 Bipolar Self-attention for Spiking Transformers
To address two critical issues existing SSA: incomplete polarity correlation and Softmax loss, propose a novel Bipolar Self-attention (BSA) paradigm. First, model the different polarity interactions between Querys and Keys using ternary matrix multiplication, ensuring comprehensive representation of polarity information. Furthermore, we introduce the Shiftmax method, which approximates Softmax through energy-efficient bitwise operations while preserving the critical low-entropy activation and now-normalization properties. The details are as follows:
4.2.1 Ternary Matrix Product for Comprehensive Polarity Correlation
To effectively capture the comprehensive polarity correlation in SNNs, we introduce ternary spike neurons (TSN) [14] for characterizing the pre-synaptic membrane potential of Q and K. The dynamics of the (.) are defined as follows:
U[l + 1| H X + 1),
(9)
Sit + 1] Sign (U]t + 1)) ([U[t 1]1 Vox)
(10)
H[t + 1] Vreser Sit + +1]++UU[ + 1j(1 |S[t + 1]])
(11)
(-)((-) fires signed spikes Sit + 1] and [/ and reset processes are similar to those in binary spike neurons. Subsequently, BSA computes Score' through matrix multiplication using ternary Q and K. The Termary-valuse Matrix Product (TMP) method effectively preserves the fundamental characteristics of both Binary-valued Matrix Product (BMP) in SSA and Real-valued Matrix Product (RMP) in VSA. First, it preserves the spike-driven characteristics BMP, enabling full AC operations and achieving computation. Additionally, it retains the polarity information in the membrane potential U(4). thereby capturing polarity-related correlations comparable to RMP operations. To validate this proposition, we conduct detailed analysis of the three matrix product, examining both their distributional relationships and local correlation. Detailed proof's are presented in Appendices
reset to 0. The membrane potential accumulation
neuromorphic
Theorem For independent random vectors q Q₁, k Kj E with elements qes kg NO o²), p ( representing the probability P(q > 0), A represent the threshold for SN and T SN(.). Then the result element of RMP satisfies Rt = 0 and Varl R) = do BMP satisfies
E[B] = dp and VBr(B) dp² (1 - p²); and TMP satisfies E E[T] 0 and Var(T") = 4dp2.
-p²)),
5

---

Theorem[] establishes that TMP preserves the zero-mean statistical property of RMP, while BMP exhibits systematic bias. This bias results from binary activation's 's exclusion of negative polarity information, corroborating our hypothesis. Furthermore, as shown in Fig 3(c), at typical operating points (Vin = 1.0.5, corresponding to common SNNs' thresholds), TMP's variance consistently exceeds BMP's, enhancing information capacity and providing a more faithful representation of RMP's statistical characteristics. In summary, TMP approximates RMP's overall distribution more accurately than BMP, consequently enhancing Q-K correlation representation. Nevertheless, global distributional similarity does not guarantee accuracy at individual key-value pairs. Therefore, we further explore whether TMP also demonstrates stronger local correlation with RMP relative to BMP.
$(-2) representing the probability P(q > 8). 0 represent the threshold for spiking neurons. We define B--) and (-) to represent the binary and terary activation functions. respectively. According to the covariance calculation formulas and Theorem T(4) (le) can be expressed as:
Theorem 2.
For independent variables 9 Q3 k K2 E Rd, qi,k; ki ~ o00,
the covariance between qk. B()) and
Covigk, (()) = d)(0)² 0²= = do(0)'0² Cov(qk, ((g)))) = 4do (0)
Where (00) represents the value of the probability density function at the threshold 0.
Based Theorem 2 we derive the approximate Pearson correlation coefficients between RMP, BMP, and TMP for each q-k pair (R, B, T), which can be formulated as follows:
p R, B) = Cov(R, B)
$10)²
Cov(BTT) 20(0)² po
Var po -1 P(R,T)
(12)
VVar((1)Var
By analyzing the ratio of these correlation coefficients, we obtain: p(R,T)
p(R.B) 2 p2, where p
ranges from (0, 1). For typical spike firing rate (0.1 < < 0.4), TMP provides approximately 2x local correlation than BMP. In conclusion, both in terms of distribution and local correlation, TMP is more closer to RMP compared to BMP, thereby better capturing the correlation of Q and K.
4.2.2 Shiftmax for Energy-efficiency Softmax Alternative
As previously mentioned, the distribution of TMP mirrors the bell-shaped distribution as RMP in ANNs. Empirical evidences [T8] 27 indicate that this distribution necessitates the Softmax function to generate effective self-attention scores. However, Softmax needs massive computing resources conflicting with the energy efficiency of SNNs. To address this, we propose an energy-efficient Shiftmax method based on bit-shift operations, which partially replicates the Softmax effect while better aligning with SNNs. Given an input vector x approximate Softmax function as follows:
[x1, F2,..., xnl" E I"", we define the
).
Shiftmax(x): = 2xx->(x), 7(x) logz Σ 2
(13)
2 represents the element-wise power operation. 7xx defined as the minimal power of 2 that is greater than or equal to the summation of 2 Shiftmax(-) retains the key advantages of Softmax while introducing unique computational efficiency features. Firstly, the 2 effect induced by the exponential operation effectively amplifies important weights while suppressing irrelevant ones, ensuring low-entropy activation characteristics. Secondly, it achieves quasi-normalization through a meticulously structured denominator, explicitly constraining the row-stochasticity constraints of Score within a bounded range, formally expressed as:
FL
Shiftmax(x), 2 [ogg 2 1.
(14)
(x) constrains the row-stochasticity constraints of attention scores within the interval (0.5, 1]. Although this constraint does not strictly enforce a row-stochasticity constraint, the limited range ensures consistency across attention distributions, enhancing the overall stability of the attention mechanism. Notably, the Shiftmax(-) converts traditional MAC operations into highly efficient bit-shift operations when multiplied by matrix V, which can be decried as:
Score's > V =aax= Σ (Uyk >> nig)
Sij = 2-1
(15)

---

In summary, the Shiftmax(") preserves the key advantages of the Softmax while transforming intensive exponentiation and normalization into bit-shift operations. It achieves optimal attention score allocation without incurring significant additional computational overhead.
4.2.3 Overall Architecture
Through ternary Q-K matrix product and energy-efficient Shiftmax operations. we propose the BSA module. When the imput E R2''xxxd the computational process of BSA are as follows:
K, V (B)()))(X)))
Q, K. V RTxxxd
(16)
Score' (Q) x TS)(K),
Score' [Txnxa
(17)
Score = Shiftmax(Scorc")
Score E [Txxxx
(18)
Attn TSN(Se V),
Attn E gl'xxxd
(19)
Where II denotes integer values and S represents spike trains. Since Shiftmax(()) directly yields
powers (2"), we can perform bit-shift operations on the membrane potentials of V without employing MAC operations. This approach simultaneously preserves the rich membrane potential
information V while maintaining minimal energy overhead, as bit-shift operations [66 merely 1/20 of the energy required for AC operations.
consume
5
Experiments
5.1 Image Classification
In image classification Task, we evaluate the proposed BSA module on ImageNet- IK 7 using three representative state-of-the-an (SOTA) Spiking Transformer architectures: Spikingformer [67]. QKformer [68], and Spike-drive Transformer- V3 [52). Additionally, we perform comprehensive
comparative analyses against recent Spiking Transformers 770
Raw Picture
SSA
BSA(Ours)
Figure 4: Attention heatmap comparison with Spikingformer 66
architecture ImageNet-
As shown in Table T our BSA module consistently improves performance across all three Spiking Transformer architectures. The most substantial gains appear Spikingformer, where BSA increases accuracy by 1 1.35% (D=512) and 0.97% (D=768), reaching 76. 14% and 76.82% respectively. For Spike-driven V3, BSA yields improvements ranging from 0.36% 0.74% across different parameter settings (5M, 10M, 19M). Similarly, QKformer's accuracy increases by 0.41% and 0.35% in its two configurations. These results can demonstrate BSA' versatility across different architectures. Additionally, shown in Fig. 4 BSA exhibits sparser attention distribution with enhanced focus on critical visual features. Notably, performance improvements in the QKformer and SDT-V3 architecture are less compared to those in the Spikingformer. We attribute this to two factors. First, QKformer and SDT-V3 already demonstrate strong baseline performance on image classification tasks, leaving limited room for further enhancement. Second, both QKformer and SDT-V3 structurally align with the Pyramid Vision Transformer 21 39 54 which substantiates the conclusion that self-attention modules contribute relatively minor performance gains within Metaformer (55) designs.

---

Table 1: Detailed comparison with other similar methods ImageNet- -
Method
Architecture
T
Param.(M)
Acc.
Spiklormer-8
16.8
70.24
Spikformer
Spikformer-8-55
4
29.7
73.38
Spike-driven-8-335
4
16.8
72.28
SDT-VI [51
Spike-driven-8-
29.7
74.57
SpikingResffrmerrSS
4.
17.8
75.95
SpikingResformer 30
SpikingRecsfrmer--M
35.5
77.24
Meta- Meta-Spike Leta-SpikeForrmer-338.
15.1
74.10
SDT-V2 44
Meta- MetasSppiccFommee-5522
55.4
79.70
Spikingformerr-8-5555
29.7
74.79
Spikingformer |67
Spikingformer-8 8-88 8-7
4
66.4
75.85
Spikingformer-3 ----rr-8--222
29.7
76.14 (11.35)
Spikingformer BSA (Ours)
Spikingformer- 768
4
66.3
76.82 (10.97)
HST-10-384
16.47
78.80
QKformer 168
HST-10-512
4
29.08
82.04
HST-10-384
16.47
79.21 (10.41)
QKformer BSA (Ours)
HST-10-512
29.08 4
82.39 (10.35)
Efficiient-rranfformer-S
5.1
75.30
SDT-V3 152
Elficieet--rranffomme---
4
10.0
78.50
Efficient-transormmerl..
19.0
79.80
Efficieen--rrnssfomme---
4
5.1
75.72 (f0.42)
SDT-V3 + BSA (Ours)
Prieiietttransfoome--M
10.0
79.24 (10.74)
deffcient-transformee
19.0
80.16 (10.36)
5.2 Semantic Segmentation and Event-based Tracking Tasks
To further validate the efficacy of the pro- posed BSA, we evaluate performance on more regression tasks, such as Semantic Seg- mentation and Event-based Tracking tasks. For semantic segmentation, we employ the challenging ADE20K dataset [65]. which comprises 20K and 2K images in the train- ing and validation sets, respectively, covering 150 semantic categories. We strictly adhere to the experimental protocol SDT-V3 [52) to assess BSA's performance As shown in T2 BSA achieves signif- icant improvements of 1.8% and 2.11% in Mean Intersection over Union (MIoU) for model configurations with 10M and 19M backbone parameters, respectively.
Table 2: Performance of segmentation.
Model
Param.(M) T
MIoU(%%
ResNet-18 54
15.5
32.9
PVT-Small
39
28.2
1
39.8
InternImage-
59.0
48.1
SDT-V2 49
16.5
4
33.6
SDT-V2 49
59.8
4
35.3
ADE20K.
SDT-V3 [52
5.1+1.4
4
33.6
SDT-V3 [52]
10,0+1.4
4
40.1
SDT-V3 (52
19.0+1.4
4
41.3
SDT-V3 BSA
10.0+1.4
4
41.90 (11880)
SDT-V3 BSA
19.4+1.4
4
43.41 (12.11)
Furthermore, examine BSA's efficacy in event-based tracking, a particularly challeng- ing yet practical SNN application domain.
Results reproduced by ourselves.
We implement the SDTrack Pipeline (88 methodology, employing the Global Trajectory Prompt method to process event streams into event frames. We adhere rigorously to its prescribed training protocol, substituting only the SDTrack backbone with our SIDTrack+BS. BBA backbone. As shown in Table. comprehensive experiments across the FE108 58 FELT 440 and VisEvent [4 datasets consistently demonstrate that SDTrack+BSA significantly outperforms the original SDTrack archi- tecture multiple metrics. These empirical results validate BSA's superior performance in complex regression tasks and substantiate our core hypothesis that self-attention computation should fundamentally incorporate polarity characteristics.

---

Table 3: Performance of BSA three event-based tracking datasets.
FELT
40
FE108
58
VisEvent
41
Param.
Methods
(M)
AUC(%)
PR(%)
AUC(%)
PR(%)
AUC(%%
PR(%)
STARK 47
28.23
39.6
51.7
57.4
89.2
34.1
46.8
ARTrack
42
202.56
39.5
49.4
56,6
88.5
33.0
43.8
OSTrackı5
153
92.52
35.9
45.5
54.6
87.1
32.7
46.4
HIPTrack
31
120.41
38.2
48.9
50.8
81.0
32.1
SNNTrack
[59
31.40
35.4
STNct 57
20.55
35.0
50.3
SDTTrack-Tng
28
19.61
39.3
51.2
59.0
91.3
35.6
49.2
SDTrack--BSA(Our)
19.61
40.9(11.6)
51.8(10.6)
59.2(10.2) 91.4(10.1) 36.8(11.2) 52.3(13.1)
5.3
Ablation Study
To validate the efficacy of BSA components, we conduct ablation studies on the CIFAR100 dataset [19]. examin- ing various matrix products (MP) and row-stochasticity constraints (RSC) methods. Our experiments utilize the Spikingformer architecture. As shown in the Ta- ble combining BMP with either Softmax or Shift- max yields no performance improvement (even showing slight degradation). The performance decline demon- strates that BMP exhibits errors correlation compu- tation. These errors are amplified by Softmax, conse-
Table 4: Ablation study
Method
MP
RSC
Acc.(%)
BMP
Nonc
79.13
SSA
BMP
Softmax
79.07
BMP
Shiftmax
79.14
TMP
None 79.27
BSA
TMP
Softmax
80.76
TMP
Shiftmax
80.48
quently impairing the network's 's decision-making per- formance. Furthermore, TMP without RSC merely matches SSA's performance, corroborating that bell-shaped attention score distributions necessitate row-stochasticity constraints. Finally, while Chiftmax+TMB exhibits marginally lower performance than SoftmaxTTMP, it achieves an optimal balance between energy efficiency and high performance.
(a) Semantic Segmentation
(b) Event-based Tracking
319
22
#178
#395
Ground Truth
BSA(Ours)
Image
TDTraak
SNNTrack BSSTack-
Figure 5: (a) Semantic segmentation comparison for ADE20K. (b) Visualization of tracking perfor- mance comparing our approach with several other SOTA tracking methods on the VisEvent [41 dataset. Red boxes indicate ground truth, and frame numbers displayed in the upper left corner.
Conclusion
In this paper, we identify two fundamental limitations of Spiking Self-attention: the binary matrix product's inability capture anything beyond positive-positive correlations, and the lack of row- stochasticity constraints leading to incomparable attention scores. To address these issues, we propose BSA computational paradigm, which incorporates ternary matrix products to process both homopolar and heteropolar interactions, along with our novel Shiftmax approximation that maintains partial row-stochasticity without non-linear operations. image classification, BSA achieves significant performance improvements across multiple advanced Spiking Transformers. Simultaneously. it establishes new SOTA results semantic segmentation and event-based tracking tasks. These findings demonstrate BSA's potential to become core component in Spiking Transformers.

---

Acknowledgments
This work is supported in part by the National Natural Science Foundation China (No. 62220106008 and 62271432) 1432), in part by Sichuan Province Innovative Talent Funding Project for Post- doctoral Fellows (BX202405). in part by the Shenzhen Science and Technology Program (Shenzhen Key Laboratory, Grant No. ZDSY DDSS$220336260910000000 Introducing Innovative and Entrepreneurial Teams, Grant No. 20232T10X042 and in part by the State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Grant No. SKL.BI-K020000.
in part by the Program for Guangdong
References
[1] Sander M Bohte, Joost N Kok, and Johannes A La Poutré. Spikeprop: backpropagation for networks of spiking neurons. In ESANN, volume 48, pages 419-424. Bruges, 2000.
[2] Maxence Bouvier, Alexandre Valentian, Thomas Mesquida, Francois Rummens, Marina Reyboz, Elisa Vianello, and Edith Beigne. Spiking neural networks hardware implementations and challenges: A survey, ACM Journal on Emerging Technologies Computing Systems (JETC), 15(2):1-35, 2019.
[3] Wenrui Cai, Qingjie Liu, and Yunhong Wang. Hiptrack: Visual tracking with historical prompts. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pallern Recognition, pages 19268-19267, 2024.
[4] Honglin Cao, Zijian Zhou, Wenjie Wei, Ammar Belatreche, Yu Liang, Dehao Zhang, Malu Zhang, Yang Yang, and Haizhou Li. Binary event-driven spiking transformer. arXiv preprint arXiv:2501.05900 2025.
[5] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi- scale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 357-366, 2021.
[6] Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro, 38(1):82 021882999990 2018.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
[8] Lei Deng, Guanrui Wang, Guoqi Li, Shuangchen Li, Ling Liang, Maohua Zhu, Yujic Wu, Zheyu Yang, Zhe Zou, Jing Pei, al. Tiajjie: A unified and scalable chip bridging spike-hased and continuous neural computation. IEÉE Journal of Solid-State Circuits, 55(8):2228-22 2222222626,6 2020.
[9] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18/000000 2018
[10] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arxiv:2010.1199000 2020.
[11] Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. Advances in Neural Information Processing Systems, 34:26183-26197 2021.
[12] Wulfram Gerstner and Werner M Kistler. Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press, 2002.
[13] Meng-Hao Guo, Zheng-Ning Liu, Tai-Jiang Mu, and Shi-Min Hu. Beyond self-attention: External attention using two linear layers for visual tasks. IEEE transactions on pattern analysis and machine intelligence, 4546))5436-5447, 2022.
10

---

[14] Yufei Guo, Yuanpei Chen, Xiaode Liu, Weihang Peng, Yuhan Zhang, Xuhui Huang, and Zhe Ma. Temary spike: Learning temnry spikes for spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12244-12252, 2024.
[15] Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. Proceedings of the IEEE/CVF international conference on computer vision, pages 5961-5971, 2023.
[16] Mark Horowitz. 1.1 computing's 's energy problem (and what can do about it). In 2014 IEEE international solid-state circuits conference digest of technical papers (ISSCC), pages 10-14. IEEE, 2014.
[17] Eugene M Izhikevich. Simple model of spiking neurons. IEEE Transactions neural networks, 1466)1169-1572, 2003.
[18] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns:: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156-5165, PMLR, 2020.
[19] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[20] Yu Liang, Wenjie Wei, Ammar Belatreche, Honglin Cao, Zijian Zhou, Shuai Wang. Malu Zhang, and Yang Yang. Towards accurate binary spiking neural networks: Learning with adaptive gradient modulation mechanism. arXiv preprint urXiv:2502. 14344, 2025.
[21] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[22] Songchen Ma, Jing Pei, Weihao Zhang, Guanrui Wang, Dahu Feng, Fangwen Yu, Chenhang Song, Huanyu Qu, Cheng Ma, Mingsheng Lu, al. Neuromorphic computing chip with spatiotemporal elasticity for multi-imtelligen--aaskee tasking robots. Science Robotics, x 2022.
[23] Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9):1659-1671, 1997.
[24] Timothée Masquelier, Rudy Guyonneau, and Simon J Thorpe. Spike timing dependent plasticity finds the start of repeating patterns in continuous spike trains. PloS one, 3(1777 2008
[25] Garrick Orchard, Paxon Frady, Daniel Ben Dayan Rubin, Sophia Sanborn, Sumit Bam Shrestha, Friedrich T Sommer, and Mike Davies. Efficient neuromorphic signal processing with loihi In 202/ IEEE Workshop on Signal Processing Systems (SiPS), pages 254-259. IEEE, 2021.
[26] Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip 2019.
architecture. Nature, 572(7767):
[27] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing mnns for the transformer era. arXiv preprini ar3iv 13048, 2023.
[28] Yimeng Shan, Zhenbang Ren, Haodi Wu, Wenjie Wei, Rui-Jie Zhu, Shuai Wang, Dehao Zhang, Yichen Xiao, Jieyuan Zhang, Kexin Shi, et al. Sdtrack: A baseline for event-based tracking via spiking neural networks. arXiv preprint arXiv:2503.08703. 2025.
[29] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. arXiv preprint arXiv: 1803002555 2018
[30] Xinyu Shi, Zecheng Hao, and Zhaofei Yu. Spikingresformer: Bridging resnet and vision transformer in spiking neural networks. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 5610-5619, 2024.

---

[31] Qiaoyi Su, Yuhong Chou, Yifan Hu, Jianing Li, Shijie Mei, Ziyang Zhang, and Guoqi Li. Deep directly-trained spiking neural networks for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6555-6565, 2023.
[32] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[33] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems,
2017.
[34] Jingya Wang. Xin Deng, Wenjie Wei, Dehao Zhang. Shuai Wang, Qian Sun, Jieyuan Zhang, Hanwen Liu, Ning Xie, and Malu Zhang. Training-free ann-to-snn conversion for high- performance spiking transformer. arXiv preprint +xXiv:2508.07710. 2025.
[35] Shuai Wang, Dehao Zhang, Ammar Belatreche, Yichen Xiao, Hongyu Qing, Wenjie Wei, Malu Zhang, and Yang Yang. Ternary spike-based neuromorphic signal processing system. Neural Networks, 187:107333, 2025.
[36] Shuai Wang, Malu Zhang, Dehao Zhang, Ammar Belatreche, Yichen Xiao, Yu Liang, Yimeng Shan, Qian Sun, Enqi Zhang, and Yang Yang. Spiking vision transformer with saccadic attention. arXiv preprint 250. 2025.
[37] Sinong Wang. Belinda Z Li. Madian Khabsa, Han Fang. and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.00 04768, 2020.
[38] Wenhai Wang, Jifeng Dai, 7he Chen, Zhenhang Huang. Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages I 14408-14419, 2023.
[39] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 568 578, 2021.
[40] Xiao Wang, Ju Huang, Shiao Wang, Chuanming Tang, Bo Jiang, Yonghong Tian, Jin Tang, and Bin Luo. Long-tem term frame-event visual tracking: Benchmark dataset and baseline. arXiv preprint arXiv:2403. 058339 2024.
[41] Xiao Wang, Jianing Li, Lin Zhu, Zhipeng Zhang, Zhe Chen, Xin Li, Yaowei Wang. Yonghong Tian, and Feng Wu. Visevent: Reliable object tracking via collaboration of frame and event flows. IEEE Transactions on Cybernetics, (993):1997-2011 2023.
[42] Xing Wei, Yifan Bai. Yongchao Zheng, Dahu Shi, and Yihong Gong, Autoregressive vi- sual tracking. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9697-9706, 2023.
[43] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
[44] Yujie Wu and Wolfgang Maass. A simple model for behavioral time scale synaptic plasticity (btsp) provides content addressable memory with binary synapses and one-shot learning. Nature communications, 16(1) (6322 2025.
[45] Yichen Xiao, Shuai Wang, Dehao Zhang. Wenjie Wei, Yimeng Shan, Xiaoli Liu, Yulin Jiang, and Malu Zhang. Rethinking spiking self-attention mechanism: Implementing a-xnor similarity calculation in spiking transformers. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5444-5454, 2025.
[46] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077-12090 2021.
12

---

[47] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio- tempoaal transformer for visual tracking. Proceedings of the IEEE/CVF international conference on computer vision, pages 10448-10457, 2021.
[48] Songlin Yang. Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficien training. arXiv preprint urXiv:2312.06633 2023.
[49] Man Yao, Jiakui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. arXiv preprint arXiv:2404.088 000663, 2024.
[50] Man Yao, Jiakui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. arXiv preprint urXiv:2404.03663. 2024.
[51] Man Yao, Jiakui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike- driven transformer. Advances in neural information processing systems, 36:64043-64053 2023.
[52] Man Yao, Xuerui Qiu, Tianxiang Hu, Jiakui Hu, Yuhong Chou, Keyu Tian, Jianxing Liao, Luziwei Leng, Bo Xu, and Guoqi Li. Scaling spike- transformer with efficient spike firing approximation training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[53] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: A one-stream framework. In European conference on computer vision, pages 341-357 Springer, 2022.
[54] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference computer vision and pattemn recognition, pages 10819-10829, 2022.
[55] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10819-10829, 2022.
[56] Dehao Zhang, Shuai Wang, Ammar Belatreche, Wenjie Wei, Yichen Xiao, Haorui Zheng, Zijian Zhou, Malu Zhang, and Yang Yang. Spike-based neuromorphic model for sound source localization. Advances in Neural Information Processing Systems, 37: 17:113911 1-113936, 2024.
[57] Jiqing Zhang, Bo Dong, Haiwei Zhang, Jianchuan Ding, Felix Heide, Baocai Yin, and Xin Yang. Spiking transformers for event-based single object tracking. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 8801-8810, 2022.
[58] Jiqing Zhang, Xin Yang, Yingkai Fu, Xiaopeng Wei, Baocai Yin, and Bo Dong. Object tracking by jointly exploiting frame and event domain. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13043-13052, 2021.
[59] Jiqing Zhang, Malu Zhang, Yuanchen Wang, Qianhui Liu, Baocai Yin, Haizhou Li, and Xin Yang. Spiking neural networks with adaptive membrane time constant for weent-based tracking. IEEE Transactions on Image Processing. 2025.
[60] Malu Zhang. Jiadong Wang, Jibin Wu, Ammar Belatreche, Burin Amorppaisannon, Zhixuan Zhang, Venkata Pavan Kumar Miriyala, Hong Qu, Yansong Chua, Trevor E Carlson, et al. Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks. JEEE transactions on neural networks and learning systems, 33(5):1947-1958 2021.
[61] Malu Zhang, Shuai Wang, Jibin Wu, Wenjie Wei, Dehao Zhang, Zijian Zhou, Siying Wang, Fan Zhang, and Yang Yang. Toward energy- spike- based deep reinforcement learning with temporal coding. IEEE Computational Intelligence Magazine, 20(2):45-57, 2025.
[62] Jing Zhao, Ruiqin Xiong, Jiyu Xie, Boxin Shi, Zhaofei Yu, Wen Gao, and Tiejun Huang. Reconstructing clear image for high-speed motion scene with retina-inspired spike camera. IEEE Transactions on Computational Imaging, 8:12-27, 2021.
13

---

[63] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from sequence-t--sqqueee perspective with transformers. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6881-6890, 2021.
[64] Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High- speed image reconstruction through short-term plasticity for spiking cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2021.
[65] Bolei Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
[66] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
[67] Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Zhengyu Ma, Han Zhang, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual learning for transformer-basee spiking neural network. arXiv preprint arXiv:2304.11954 2023.
[68] Chenlin Zhou, Han Zhang, Zhaokun Zhou, Liutao Yu, Liwei Huang, Xiaopeng Fan, Li Yuan, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. Okformer: Hierarchical spiking transformer using ςk attention. arXiv preprint arXiv:2403.16552, 2024.
[69] Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmentation: A prototype view. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2322-259,, 2022.
[70] Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, YAN Shuicheng, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. In The Eleventh International Conference Learning Representations, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:[Yes:
Justification: have clearly outlined our contributions and scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in the appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We provide explicit conditions for the proposed theory and offer detailed
proofs in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the experimental conditions and hyperparameter settings in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, have submitted the code for the experiments and will upload it to GitHub once the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
Soblic//guddes/ooeeuubbiss
casssiionlllyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
wowpeeee/oooSSemmsssssdsee. www.ssssiicggoooooo for more details. / wwtps/cccppppgguuuaaaeoeaec
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, have outlined some details of the experimental hyperparameters in the appendix, though it may not cover everything. However, we have submitted the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our experimental results represent the average of multiple runs, but we have not reported the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we have reported the model specifications of the computational resources used for the experiments and provided some experimental details in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, we comply with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is foundational research and not tied to particular applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, the paper properly credits the creators or original owners of assets (e.g.,code, data, models) and explicitly mentions and respects the relevant licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: new assets are introduced in this article.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The LLMs are only used for writing, editing, and formatting purposes. They did not contribute to the methodology, scientific rigor, originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy lhttps://neuria) hatt:e//eaarrps.cc/oddeeeeessss /ctps/////cooeeecc///2///////// for what should or should not be described.
21

---

A Binary Matrix Product Exhibits Limited Low-entropy Activation
Assumption:
Let Q,K K [R with elements qiy kiy NN o²) ii.d. Under typical parameter settings, we denote the entropy of three attention mechanisms as H1 = Hreal, H2 = Sofimax and Hg Homury. Then H2 < 1i3 111.
Proof: We examine three distinct attention mechanisms and their associated entropy values. In
standard dot product attention, the scoring function &tj Ei= qakıt follows standard normal distribution (0, 1) (aceording the Central Limit Theorem), after which Softmax normalization is applied to obtain the weight matrix and the entropy of this distribution can be approximated as:
exp(sij)
1
Aj
H2 ~ logn
(20)
Ek= (s.)
2
where the term results from the concentration effect the Softmax function. Next, considering real-valued matrix multiplication, the dot product 8ty = 9ak3t follows distribution (0,, do yielding the weight matrix based on identically distributed folded normal distributions of sij- By the Law of Large Numbers, resulting in an entropy approximation:
oc, these normalized weights approach uniform distribution,
8ij
H1 Rz log n.
(21)
1== Ssk
Finally, for binary attention mechanisms, where inputs undergo binarization, the dot product öej Σi Takyz follows binomial distribution Binomial(d,p) p²). For sufficiently large d, this binary dot product can be approximated by normal distribution:
B ~ (ddpp dp(1-p) -p), p),
(22)
k= 8
Consider n 100, d 64, o 1, Vn = 0.5, and = 1/vd ~ 0.125. Direct normalization yields approximately uniform distribution with H1 ze log(100) 4.6 bits. For Softmax with 7 = 0.125, simulation shows H2 0.8 bits as probability mass concentrates few large values. For binary attention with Vín = 0.5, the firing probability p z 0.31, resulting in H3 ~ 2.7 bits. This confirms
ordering: H1 H3 > H2.
B Expectation and Variance Properties of TMP, BMP and RMP
Theorm 1 For independent random vectors q = Qj. k = Kj E Rdd with elements q, h; ~ N N(0 ii.d., we define three matrix products:
Real-valued Dot Product: R qk- Di=1 gihig
Binary Dot Product: B = SN(())N((K) = ELe EE=ESEESC SN()))N(())
Temary Dot Product: T = =TN(()TTS(())) = Ei_ TSN(G))TS(())
where S-((-) is the binary sign function and TSN((- is the ternary sign function with threshold 0. If we denote p Φ )) representing the probability P(g: 0), then:
The element of RMP satisfies: B[R] 0 and Var( It) = 4
The element of BMP satisfies: B[B] np² and Var(B) = dp )²))
The element of TMP satisfies: E[T] = 0 and Var(T) = 4dp2 (1- -p).
Proof. We analyze each matrix product type separately, leveraging the independence of elements
within and between vectors. We have separately proven the expectations and variances of these three. (1) Real-valued Matrix Product (RMP): For each element R qk, since y and k; are independent
with B(g.)
[[:;; 0:
[ = Elggk:]]
Elg [k] O,
(23)
22

---

For the variance:
Var [t) = [8[ (E[R))²=Elq?4)
=
[ = o² o² = =a
(24)
Since R = D; Rg and the R, are i.i.d.:
E Σ E 0, Var R)) = Σ Var(R.) = do
(25)
(2) Binary-valued Product: For each term B SN(1)SN(H), both SN())) and SN(k;)
Bernuuli(p) random variables:
B[ B[SN(()) - PP
(26)
The product B; is also Bernoulli with parameter p²: Var(B,) p1 p²)
(27)
Hence for B Ei Bi where the are i.idd.
d
B
Σ
[ = dp",
Var(B) = Σ Var(B,) = (
(28)
1=1
According to the definition of the binomial distribution, B follows a binomial distribution.
(3) Ternary-valued Product: T = E TSN(a)TTN(()) For TSN(g:)) by symmetry of the
Gaussian distribution:
P(TSN(())) = =)= P(TSN(()) -11)= P(TSN())) 0) 1-2p 22p
(29)
Now can derive the expected value of TSN((q) by calculating the weighted sum of all possible outcomes:
E[TSN(4))) = 1 (-1)
2p)
(30)
BITSA (a)) - 12. 02. (1 21) 2p.
(31)
For the product T, TSN(())TTNNNTT Since the terms are independent, the expected value is:
B[T:] (( 0.
(32)
For the E[T,] moment, we analyze all possible outcomes:
B[T P(T: 1) 1)+ + P(T; 11)
= - = P(TSN(()) =1,55 = 1) - TSN((() = --1
(33)
P(TSN({: = (K)) -11) P(TSN():) -1,,NN)) 1)
p² 2² -p² 4p²
Therefore, using the relationship between variance, expected value, and second moment, we calculate the variance of each product term:
Var(T,) = B[T - (E(T,))² = 4p
(34)
However, we need to correct for the covariance structure to account for dependencies. The more precise calculation of variance is:
Var (T)) [T(? P(T: + 0) = 4p p))
(35)
Finally, for the complete sum T = Σd 1 T; we can determine its statistical properties by applying the linearity of expectation and the variance of sum of random variables:
E T
Var(T) 4dp² p).
(36)
23

---

C Local Correlation Comparison for TMP, BMP, and RMP
Based on the conclusions Theorem and Theorem and by applying the Pearson local correlation calculation method, detailed comparison leads to the conclusion that TMP exhibits greater local correlation compared to BMP and RMP. The formal proof is provided below: For the original dot product R = Σ liki
derive the correlation coefficients for each element in R. B, and T. A
B Σ B(k)), and T -=Σ= (k:).. According Theorem I, we can obtain::
Var(T) 4dp². Subsequently, according to Theorem 2, we can obtain Cov(R, B) and Cov(R. T), which can be expressed as:
Var R) do
Var(B) (B) dp (1-p²)
(37)
Cov(R, = Σ
Covlg:k, B(a)B(k)) - -do
(38)
Cov(R, E (()Tgk),(((((k))))) = 4do²233(0/0)
(39)
Based the above data and the Pearson correlation coefficient calculation method, we can obtain the local correlations between R and T, as well as between R and B, which can be described as follows:
p(R, B) = Cov(R, B)
do d0/o)
000/0)
/Var(It) Var(B) Vudao dp -1)) p2)
OPV1 D2
(40)
(R,T) Cov(R, T) 4do (0/o) 20(0/0)
(Var(R)
do 4dp² op
Computing the ratio of correlation coefficients:
p(R,T) T)
2010/o)/op
2
(41)
p B)
6(0/0)/oPV1
v1 p2
Since for any positive threshold 0, we have 0 < p < 0.5 (due to the symmetry of the normal
distribution), it follows that /1 p2 1, which implies:
p(R, p( (R, B)
2
(42)
VI-p²
TMP schemes preserve approximately twice the correlation with original dot products compared to binary quantization. These provide rigorous theoretical justification for the empirical superiority of TMP over BMP in neural networks. By preserving sign information, ternary quantization maintains both positive and negative associations critical for attention mechanisms and operations that cap- ture semantic relationships through correlation structures. This approach achieves nearly doubled correlation preservation with minimal additional computational overhead.
D Image Classification
All experiments are conducted on ImageNet- IK dataset using PyTorch framework. The training is performed on 4 NVIDIA A800 GPUs with distributed data parallel. The specific hyperparameters for each architecture are detailed in Table 5
To ensure fair comparison across different architectures, we strictly follow the open- source network architectures Spikingformer, QKformer, and Spike-driven V3. Importantly, we only replace self-attention computation modules within each architecture while keeping all other components (patch embedding, ML. layers, normalization layers, etc.) unchanged. This controlled experimental design allows to isolate the impact of different attention mechanisms model performance.

---

Table Comparison of Hyperparameters for Different Model Architectures
Hyper-parameter
Spikingformer
QKformer
Spike-driven V3
Timestep
4
Epochs
100
200
200
Resolution
224
224
224
Batch size
64
100
600
Optimizer
Adam W
AdamW
LAMB
Base Learning rate
7e-6
6e-4
6e-4
Learning rate decay
Cosine
Laser-wise 1.0
Layer-wise 1.0
Warmup epochs
5
5
10
Weight decay
5e-2
5e-2
0.05
Rand Augment
rand-mi--mstd0.-iine
rand-mo-mmtd0.--ine
rand-m9--mst00.--inc
Mixup
0
Cutmix
1.0
0
0
Label smoothing
0.1
0.1
0.1
Manti Ja
Low Mes
Figure 6: Attention heatmap for BSA(Ours) with Spikingformer
[67 architecture ImageNet- K.
E
Semantic Segmentation
This study employs the ADE20K semantic segmentation dataset 665
comprising over 20,000 training and 2,000 validation scene-centric images with meticulous level annotations of objects and their constituent parts. The dataset exhibits rich semantic diversity, encompassing 150 semantic categories spanning environmental elements (sky, road, grass) and discrete entities (persons, vehicles, furniture).
66 69
For our experimental architecture, we utilize a SDT-V3 52 pretrained on ImageNet-1K as the backbone network, integrated with PVT (Pyramid Vision Transformer) [46 39 for semantic seg- mentation tasks. Newly introduced parameters are initialized using Xavier method. During model training, configure batch size 20 with 160,000 total iterations. Our optimization strategy implements AdamW optimizer with an initial learning rate of 1 x 10-4 and polynomial decay with power 0.9. Notably, we apply linear decay warmup during the initial 150.000 iterations enhance model stability. Finally, our work demonstrates significant performance improvements over the vanilla V3 architecture both 19m and 10m metrics. Additionally, we visualized the qualitative results of our model on the ADE20K dataset. As shown in Fig.] our segmentation results exhibit remarkable effectiveness, with precise boundary delineation and enhanced semantic coherence across diverse scene contexts.
25

---

SDI-V3+BSArOums)
Emage
Ground BDS-VS)BSA((Or>)
Raw Image
Figure Visualization of semantic segmentation results on the ADE20K dataset.
F
Event-based Tracking
We propose an efficient event-based visual tracker built upon the SDTrack pipeline, utilizing Image Pair Matching localize targets by comparing visual features between reference and current framee. Specifically, we introduce Hanning Window Penalty mechanism tailored for the FE108 dataset to mitigate bounding-boo drift and enhance tracking stability. Dataset details include: FE108 [88] 108 event sequences (3,000-5,000 frames each), annotated bounding boxes, diverse scenes, optimized for high-speed tracking: VisEvent [4]:: 60 RGB-event synchronized sequences (~2500 000 frames total),
varied indoor/outdoor conditions, cross-modal robustness benchmark; FELT 41: 200 sequences, event accumulation in ultra-short windows (1-5 ms), challenging real-time scenarios (motion blur, occlusion, rapid motion). The training procedure includes 100 epochs for FE108 and VisEvent, and 300 epochs for FELT. Random sampling per epoch selects 60k image pairs (maximum interval of 200 frames) for FE108 and FELT, and 30k pairs for VisEvent, ensuring sample diversity. Optimization weight decay=1 10 4)
employs A (initial LR=4 10 decayed at 80% epochs 4 10
G Limitations
The limitations of this study include performance testing of BSA larger model sizes such as LLaMA (7B, 16B and 70B) and the deployment challenges related to the shift hardware. These issues will be addressed in future research. The experimental results presented in this paper reproducible. Detailed explanations of model training and configuration are provided in the main text and supplemented in the appendix. Our code and models will be made available after the paper is accepted. In the future, we will deploy BSA onto hardware platforms such as Field Programmable Gate Arrays (FPGAs) evaluate its practical performance. During this process, will optimize the appropriate wrrt data streams and memory access schemes to enhance the inference speed of the model.
GitHub
26


Reviews:
[{"review_id": "VdeRYmTYgS", "reviewer": "Reviewer_zZBP", "summary": "This paper proposes Bipolar Self-attention (BSA) to improve Spiking Transformers by addressing two key limitations of existing Spiking Self-attention (SSA): the inability to model negative or mixed Q-K interactions and the lack of normalized attention due to the omission of Softmax. BSA uses ternary spiking neurons for richer polarity modeling and introduces Shiftmax, an energy-efficient Softmax approximation via bit-shift operations. Experiments on image classification, semantic segmentation, and event-based tracking show that BSA consistently enhances performance across multiple spiking architectures.", "questions": "See Weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n 1. The paper presents a clear motivation: (1) binary matrix products lose polarity information by ignoring negative correlations, and (2) the absence of Softmax leads to incomparable attention scores across rows, limiting the effectiveness of attention allocation.\n 2. The paper's analysis of existing Spiking Self-Attention methods effectively supports its stated motivation.\n 3. The authors conducted extensive experiments to demonstrate the effectiveness of the proposed method, and the ablation studies further validate the contribution of each component.\n\n**Weaknesses:**\n\n 1. I am curious whether this module can serve as a viable replacement for the corresponding components in existing LLMs, and what level of performance improvement it can bring.", "was_revised": true, "final_justification": "I confirm that the revisions have resolved the key issues, and the manuscript is ready for acceptance.", "rebuttal": "We sincerely appreciate your recognition of our work, which serves as significant encouragement for our research endeavors. Regarding the suggestions you have proposed, we provide a comprehensive overview below.\n\n## Weakness 1: I am curious whether this module can serve as a viable replacement for the corresponding components in existing LLMs, and what level of performance improvement it can bring.\n**A**:  We appreciate your valuable suggestion regarding BSA as a potential alternative to SA. While we agree that evaluating BSA on large language models such as LLaMA[1] and DeepSeek[2] would be meaningful, time and computational constraints during the rebuttal period precluded retraining or fine-tuning these architectures. Instead, we conducted comprehensive experiments using BERT[3] on GLUE benchmark tasks. \n\n|  | Spike-Driven | cola | qqp | qnli | sst-2 | sts-b | mrpc | rte |\n|:----------------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n| Bert | × | 59.7 | 91.4 | 92.1 | 93.2 | 90.1 | 86.3 | 72.1 |\n| Bert+Binary SA[4] | × | Fail | 63.18 | 50.54 | 52.75 | 83.2 | 81.2 | 53.8 |\n| Bert+BSA（Ours） | √ | 55.76 | 87.57 | 90.17 | 91.52 | 88.6 | 84.8 | 64.98 |\n\nAs shown in the above Table, we replace the SA components in BERT with binary SA[4] in BiBERT and our BSA (Ours). The experiments demonstrate that BERT with BSA exhibits only partial performance degradation compared to vanilla BERT and superior performance to Binary SA[4]. These results support your expectation that BSA represents a viable alternative for large language models. We plan to validate BSA's effectiveness on larger model architectures in future work.\n\n\n\n\n## Reference:\n\n[1] Touvron H, Lavril T, Izacard G, et al. Llama: Open and efficient foundation language models[J]. arXiv preprint arXiv:2302.13971, 2023.\n\n[2] Dai D, Deng C, Zhao C, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models[J]. arXiv preprint arXiv:2401.06066, 2024.\n\n[3] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 2019: 4171-4186.\n\n[4] Qin H, Ding Y, Zhang M, et al. Bibert: Accurate fully binarized bert[J]. arXiv preprint arXiv:2203.06390, 2022."}, {"review_id": "p5quvFO0LN", "reviewer": "Reviewer_Tb3D", "summary": "By analyzing the data characteristics of spiking neural networks, the authors identify two shortcomings in the Spiking self-attention—polarity information loss and Softmax failure. They propose targeted remedies for each. The paper is rigorously structured with clear visualizations. experiments on multiple vision benchmarks demonstrate significant performance gains.", "questions": "1. Appendix A shows that binary matrix multiplication partially achieves low-entropy activation：why can’t binary multiplication directly replace ternary multiplication + Softmax? \n2. Why is it necessary to establish local correlation? Does it complement distributional similarity?\n3. In my view, the strengths of SNNs are most evident on dynamic datasets (e.g., DVS), yet the authors evaluate their method predominantly on static image tasks. I recommend validating the proposed approach on a dynamic benchmark to demonstrate its broader applicability.\n4. The ablation study confirms that TMP relies on the softmax operation more heavily than BMP, but omits RMP’s performance results. Given the theoretical claim of RMP’s superior performance bound, please include RMP in the ablation study to substantiate this assertion.\n5. In the presented BSA, V does not activated by Spiking neurons. Does it still retain true spike‐driven behavior?  It would also strengthen the paper to add theoretical energy‐consumption analyses to quantify the claimed efficiency gains.", "limitations": "The proposed method may encounter limitations when applied to dynamic vision tasks.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength:  \nThe paper features clear visualizations and a well‐organized presentation of the problem analysis and methodology. Additionally, the experiments validate the proposed approach across multiple vision tasks, enhancing the credibility of the results.   \nWeaknesses    \n1. This work lacks Ablation experiments comparing RMP and TMP performance, despite both Theory 1 and Theory 2 resting on the premise that RMP outperforms TMP.\n2. The manuscript lacks description of the spiking neuron model within the Transformer Block. \n3. The proposed BSA is validated solely on visual datasets, with no evaluation on sequential or speech processing tasks.\n4. In the semantic segmentation experiments, the authors re-implement only the 19 M-parameter model; to ensure a fair comparison, the 5 M and 10 M variants should also be re-implemented.", "was_revised": true, "final_justification": "This paper makes a valuable contribution to the SNN field, and its writing meets the standards for a scientific paper. Therefore, I recommend acceptance.", "rebuttal": "Thank you very much for your recognition of our work. In response to the weaknesses and suggestions you have raised, we will provide further **detailed explanations**:\n\n## Weakness 1 and Questions 4:This work lacks Ablation experiments comparing RMP and TMP performance, despite both Theory 1 and Theory 2 resting on the premise that RMP outperforms TMP.\n\n**A**: Thank you for your valuable suggestions. We have added performance evaluations of RMP and TMP under the Shiftmax framework. The experiments were conducted using the QKformer architecture on CIFAR-100. As shown in the table below, RMP and TMP achieve comparable performance, while RMP introduces substantial floating-point multiplication operations. Thereofre, TMP serves as a highly efficient alternative to RMP.\n| Model | Spike-Driven | Accuracy |\n|-------|---------|----------|\n| QKFormr+RMP | No | 81.69|\n| QKFormr+TMP | Yes | 80.48 |\n\n## Weakness 2: The manuscript lacks description of the spiking neuron model within the Transformer Block.\n\n**A**: Our BSA implementation comprises two primary neuron architectures. First, the ternary LIF neuron, commonly employed in QKformer and Spikeformer, governed by the following. The dynamics of the ternary spike neuron (TSN) can be concisely expressed as follows. The membrane potential is first updated by\n$$U[t+1] = H[t] + X[t+1],$$\nwhere $H[t]$ is the previous hidden state and $X[t+1]$ is the input at the current time step. The neuron then generates a ternary spike output using\n$$S[t+1] = \\mathrm{Sign}(U[t+1]) \\cdot \\Theta\\left( |U[t+1]| - V_{\\mathrm{th}} \\right),$$ \nwhere $\\Theta(\\cdot)$ is a step function and $V_{\\mathrm{th}}$ is the firing threshold. Finally, the internal state is updated by:\n$$H[t+1] = V_{\\mathrm{reset}} \\cdot S[t+1] + \\tau \\cdot U[t+1] \\cdot (1 - |S[t+1]|),$$\nwhere $V_{\\mathrm{reset}}$ is the reset value and $\\tau$ is a decay factor that determines the leaky integration when no spike occurs. Second, the ternary ILIF model within the STD-V3 framework, which operates in distinct modes during training and inference phases. In the training stage, the output of an I-LIF neuron is expressed as:\n$$\n\\mathbf{s}^\\ell = \\frac{1}{T} \\cdot \\left\\lfloor \\mathrm{clip}(\\mathbf{x}^\\ell, -T, T) \\right\\rceil,\n$$\nwhere $\\mathbf{x}^\\ell$ is the input current at layer $\\ell$, $T$ is the simulation time steps, $\\mathrm{clip}(\\mathbf{x}^\\ell, -T, T)$ constrains $\\mathbf{x}^\\ell$ to $[-T, T]$, and the symbol $\\left\\lfloor \\cdot \\right\\rceil$ denotes rounding to the nearest integer.\n\n\n \n## Weakness 3: The proposed BSA is validated solely on visual datasets, with no evaluation on sequential or speech processing tasks.\n**A**: Thank you for your valuable suggestions. Following the architecture proposed by Wang et al.[], we replaced the T-SSA component with our BSA and validated the approach on the GSC-V2 dataset. Our BSA achieves a 0.85% improvement over ST-Transformer.\n| Model | Network | Accuracy |\n|-------|---------|----------|\n| E2E SNN [1] | SNN | 92.9 |\n| KWS-SNN [1] | SNN |  94.4|\n| ST-Transformer[3] | SNN | 95.18 |\n| **Ours** | SNN | **96.03** |\n\n## Weakness 4: In the semantic segmentation experiments, the authors re-implement only the 19 M-parameter model; to ensure a fair comparison, the 5 M and 10 M variants should also be re-implemented.\nThank you for your valuable suggestions. To further validate the effectiveness and scalability of our approach, we conducted multiple independent implementations and repeated experiments on 5M and 10M parameter models. The corresponding baseline experimental results are presented as follows:\n| Model |  Parameter | MIoU(%) |\n|-------|-----------|-------|\n| SDT-V3† [4] | 5.1±1.4 | 32.8 |\n| SDT-V3† [4] | 10.0±1.4 | 39.4 |\n| SDT-V3† [4] | 19.0±1.4 | 41.3 |\n\n## Questions 1 : Appendix A shows that binary matrix multiplication partially achieves low-entropy activation：why can’t binary multiplication directly replace ternary multiplication + Softmax?\n**A**: Low-entropy activation is an inherent property of Softmax that enables sharp contrast in attention scores—amplifying weights for relevant positions while suppressing irrelevant ones, thereby achieving clear differentiation. Although BMP generates attention matrices with low-entropy-like distributions (exhibiting similar high-low contrast patterns), the resulting high and low attention regions may not accurately identify the truly important and negligible content, potentially failing to achieve optimal attention allocation.\n\n## Questions 2 :Why is it necessary to establish local correlation? Does it complement distributional similarity?\n**A**:We analyze local correlation because while global distributional similarity indicates overall statistical alignment, it does not guarantee accuracy at individual query-key interaction levels. Attention mechanisms fundamentally rely on precise pairwise correlations between specific tokens, making local accuracy crucial for effective attention allocation. Our analysis demonstrates that for typical spike firing rates, TMP exhibits approximately 2× stronger local correlation with RMP than BMP.\n\n## Questions 3 :In my view, the strengths of SNNs are most evident on dynamic datasets (e.g., DVS), yet the authors evaluate their method predominantly on static image tasks. I recommend validating the proposed approach on a dynamic benchmark to demonstrate its broader applicability.\n**A**: Thank you for your valuable suggestions. To further validate the generalizability and robustness of BSA, we conducted supplementary performance evaluations on neuromorphic datasets DVS-CIFAR10 and DVS-Gesture. We employed the QKformer architecture with two Transformer blocks to ensure sufficient representational capacity for processing complex temporal dynamics. BSA and SSA were compared under identical experimental conditions to ensure fair evaluation. As shown in the table below, BSA demonstrates significant performance improvements over SSA on both challenging dynamic datasets, further supporting the viability of BSA as an efficient attention mechanism alternative.\n\n| Dataset | Method | Architecture | Param. | Performance (10, 16 Timesteps) |\n|---------|--------|-------------|------------|---------------------|\n| DVS128 | Spikformer[5] | Spikformer-2-256 | 2.57 | 96.9, 98.3 |\n|        | Spikingformer[6] | Spikingformer-2-256 | 2.57 | 96.2, 98.3 |\n|        | QKFormer[7] | HST-2-256 | 1.50 | 98.3, 98.6 |\n|        | QKFormer+BSA | HST-2-256 | 1.50 | 98.5, 98.7 |\n| CIFAR10-DVS | Spikformer[5] | Spikformer-2-256 | 2.57 | 78.9, 80.9 |\n|             | Spikingformer[6] | Spikingformer-2-256 | 2.57 | 79.9, 81.3 |\n|             | QKFormer[7] | HST-2-256 | 1.50 | 83.8, 84.0 |\n|             | QKFormer+BSA | HST-2-256 | 1.50 | 84.1, 84.6 |\n## Questions 5 :In the presented BSA, V does not activated by Spiking neurons. Does it still retain true spike‐driven behavior? It would also strengthen the paper to add theoretical energy‐consumption analyses to quantify the claimed efficiency gains.\n**A**：Thank you for the valuable feedback. Although the operation between membrane potentials (V) and attention maps resembles matrix multiplication, the computation is fundamentally different. Our attention maps consist of powers of two, representing bit-shift positions rather than conventional weights. The following table presents energy consumption data for different operations on ASIC (45nm)[8], providing a quantitative foundation for assessing the energy advantages of this shift-based approach.\n\n| Format |  | ASIC (45nm) |  |\n|--------|--------|--------|--------|\n| **Operation** | **Format** | **Energy (pJ)** | **Improv.** |\n| **Mult.** | FP32 | 3.7 | - |\n| **Mult.** | FIX8 | 0.2 | - |\n| **Add** | FP32 | 0.9 | 4.1x |\n| **Add** | FIX8 | 0.03 | 6.7x |\n| **Shift** | FIX32 | 0.13 | 24x |\n| **Shift** | FIX8 | 0.024 | 8.3x |\n\n\n| Attention Mechanism | Forward Pass Energy (mJ) | Energy Reduction | Improvement Factor |\n|---------------------|--------------------------|------------------|-------------------|\n| Self-Attention | 1.23 | - | 1.0× |\n| Spiking Self-Attention | 0.062 | 95% | **20.0×** |\n| BSA（Ours） | 0.084 | 93% | **14.6×** |\n\nWe conduct a fair comparison using standardized configurations with Q, K, V matrices of dimension B×C×N = 256×512×512. Under identical computational loads, BSA achieves energy efficiency comparable to Spiking SSA, with both methods significantly outperforming Standard SA.\n## Reference\n[1] Yang, Qu, Qi Liu, and Haizhou Li. \"Deep residual spiking neural network for keyword spotting in low-resource settings.\" Interspeech. 2022.\n\n[2] Wang S, Zhang D, Shi K, et al. Global-Local Convolution with Spiking Neural Networks for Energy-efficient Keyword Spotting[C]//Proc. Interspeech 2024. 2024: 4523-4527.\n\n[3] Wang, Y., Shi, K., Lu, C., Liu, Y., Zhang, M., & Qu, H. (2023, August). Spatial-Temporal Self-Attention for Asynchronous Spiking Neural Networks. In IJCAI (pp. 3085-3093).\n\n[4] Yao M, Qiu X, Hu T, et al. Scaling spike-driven transformer with efficient spike firing approximation training[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.\n\n[5] Zhou Z, Zhu Y, He C, et al. Spikformer: When spiking neural network meets transformer[J]. arXiv preprint arXiv:2209.15425, 2022.\n\n[6] Zhou C, Yu L, Zhou Z, et al. Spikingformer: Spike-driven residual learning for transformer-based spiking neural network[J]. arXiv preprint arXiv:2304.11954, 2023.\n\n[7] Zhou C, Zhang H, Zhou Z, et al. Qkformer: Hierarchical spiking transformer using qk attention[J]. Advances in Neural Information Processing Systems, 2024, 37: 13074-13098.\n\n[8] You H, Chen X, Zhang Y, et al. Shiftaddnet: A hardware-inspired deep network[J]. Advances in Neural Information Processing Systems, 2020, 33: 2771-2783."}, {"review_id": "M1mAYD5k8u", "reviewer": "Reviewer_B2kZ", "summary": "This paper proposes a bipolar spike self-attention module to enhance the performance of Spike Transformer across multiple vision tasks. It primarily consists of ternary matrix multiplication and ShiftMax. On ImageNet, BSA demonstrates improvements over the baseline across various architectures.", "questions": "1. Based on the three vision tasks, BSA shows limited performance improvement in classification tasks, but a significant enhancement in regression tasks. Could the authors provide an analysis of the underlying reasons for this discrepancy?\n2. The authors demonstrate the method’s effectiveness on several network architectures; however, the neurons in Spike‐Driven V3 were specially designed. How does the proposed approach conform to the ILIF neuron’s dynamical properties, and can it support true spike‐driven inference on par with V3?\n3. Please evaluate other spike‐based self‐attention schemes (e.g., SDSA [1], DSSA [2]) by directly replacing them with BSA to quantify any performance gaps. Do these attention mechanisms plus softmax suffer the same softmax‐incompatibility issues observed in SSA? I recommend adding ablation experiments to investigate.\n4. Have the authors compared their method to BNNs (with activations limited to +1 and –1)?  Please support this with both theoretical analysis and experimental validation.\n\nReference：\n[1]. Yao, Man, et al. \"Spike-driven transformer.\" Advances in neural information processing systems 36 (2023): 64043-64058.\n[2] Shi, Xinyu, Zecheng Hao, and Zhaofei Yu. \"SpikingResformer: bridging ResNet and vision transformer in spiking neural networks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The visualizations provided by the authors are clear, and the theoretical proofs are thorough. However, there are a few issues to address:\n1. The term “SOTA Spiking Transformer architectures” seems imprecise; it is recommended to use “advanced” instead.\n2. The paper fails to quantify the additional theoretical energy cost incurred by BSA’s full shift operation relative to SSA.\n3. The authors consider only SSA as a spiking self-attention mechanism, omitting variants such as SDSA[1] and DSSA[2].\n4. In Figure 3, the distribution statistics do not clearly specify the dimensions of N and D. It is recommended to include a more detailed description in the figure caption to enhance rigor.\n\n\nAdditional Comments:\n1. In Figure 2, the caption for the ANN & SNN diagrams is too small; it should match the font size of the surrounding text.\n2. In the experimental section, the dimensions of SpikingFormer appear to be incorrect, as the table lists them as 512 and 768.\n3. The consensus regarding the Pyramid Vision Transformer is not addressed. It would be beneficial for the authors to provide clarification on this.", "was_revised": true, "final_justification": "My questions have been resolved. Since my initial rating was positive, I will maintain my rating and recommend acceptance.", "rebuttal": "Thank you very much for your recognition of our work. \n## Weakness 1 and Additional Comments: imprecise expression. \n**A**: We thank you for this valuable feedback and apologize for the unclear presentation. In the revised manuscript, we will replace \"SOTA Spiking Transformer\" with \"advanced Spiking Transformer\" to ensure academic rigor. Additionally, we will clarify that in Figure 3, the dimensions N and D are 256 and 512, respectively, and update the corresponding dimensions in Table 1 accordingly. We greatly appreciate your constructive suggestions, which significantly contribute to improving the manuscript quality.\n\n## Weakness 2: The paper fails to quantify the additional theoretical energy cost incurred by BSA’s full shift operation relative to SSA.\n\n**A**: Thank you for the valuable feedback. Although the operation between membrane potentials (V) and attention maps resembles matrix multiplication, the computation is fundamentally different. Our attention maps consist of powers of two, representing bit-shift positions rather than conventional weights. The following table presents energy consumption data for different operations on ASIC (45nm)[1], providing a quantitative foundation for assessing the energy advantages of this shift-based approach.\n\n| Format |  | ASIC (45nm) |  |\n|--------|--------|--------|--------|\n| **Operation** | **Format** | **Energy (pJ)** | **Improv.** |\n| **Mult.** | FP32 | 3.7 | - |\n| **Mult.** | FIX8 | 0.2 | - |\n| **Add** | FP32 | 0.9 | 4.1x |\n| **Add** | FIX8 | 0.03 | 6.7x |\n| **Shift** | FIX32 | 0.13 | 24x |\n| **Shift** | FIX8 | 0.024 | 8.3x |\n\n\n| Attention Mechanism | Forward Pass Energy (mJ) | Energy Reduction | Improvement Factor |\n|---------------------|--------------------------|------------------|-------------------|\n| Self-Attention | 1.23 | - | 1.0× |\n| Spiking Self-Attention | 0.062 | 95% | **20.0×** |\n| BSA（Ours） | 0.084 | 93% | **14.6×** |\n\nWe conduct a fair comparison using standardized configurations with Q, K, V matrices of dimension B×C×N = 256×512×512. Under identical computational loads, BSA achieves energy efficiency comparable to Spiking SSA, with both methods significantly outperforming Standard SA.\n\n\n\n## Weakness 3 and Question 3  : The authors consider only SSA as a spiking self-attention mechanism, omitting variants such as SDSA[1] and DSSA[2].\n\n**A**:As you pointed out, both SDSA and DSSA are improved variants of SSA. However, SDSA and DSSA primarily treat the self-attention mechanism as a token mixer, without explicitly computing the correlation between Q and K—which is a key factor behind the success of self-attention in ANNs. Therefore, we aim for spiking self-attention to retain the ability to capture such correlations, and for this reason, we only compare our method against SSA. In the revised manuscript, we will include this comparative analysis to clarify the distinction.\n\n\n## Question 1: Based on the three vision tasks, BSA shows limited performance improvement in classification tasks, but a significant enhancement in regression tasks. Could the authors provide an analysis of the underlying reasons for this discrepancy?\n\n**A**: We appreciate the reviewer’s insightful observation. The performance gap between classification and regression tasks stems from the fact that classification generally requires coarse-grained representations and involves simpler gradient feedback, making it less sensitive to attention refinement. \n\nIn contrast, regression tasks rely on fine-grained modeling and continuous value prediction, which benefit significantly from our BSA's ability to preserve polarity interactions and apply stable attention allocation. Specifically, our BSA offers two main advantages over SSA: (1) a ternary Q-K correlation mechanism that captures richer polarity semantics; and (2) a lightweight Shiftmax normalization that approximates Softmax without introducing non-linear operations, leading to more precise and effective attention allocation.\n\n## Question 2: The authors demonstrate the method’s effectiveness on several network architectures; however, the neurons in Spike‐Driven V3 were specially designed. How does the proposed approach conform to the ILIF neuron’s dynamical properties, and can it support true spike‐driven inference on par with V3?\n**A**: Thank you for your valuable suggestions. STD-V3 employs I-LIF[2] neurons as an innovative approach for decoupling training and inference processes. We would like to provide a detailed introduction to how ternary neurons are trained and perform inference in the V3 architecture.In the training stage, the output of an I-LIF neuron is expressed as:\n\n$$\n\\mathbf{s}^\\ell = \\frac{1}{T} \\cdot \\text{round}\\left( \\mathrm{clip}(\\mathbf{x}^\\ell, -T, T) \\right),\n$$\n\nwhere $\\mathbf{x}^\\ell$ denotes the input current at layer $\\ell$, $T$ represents the simulation time steps of SNNs, $\\mathrm{clip}(\\mathbf{x}^\\ell, -T, T)$ constrains $\\mathbf{x}^\\ell$ to the interval $[-T, T]$, and $\\text{round}(\\cdot)$ denotes rounding to the nearest integer. The computed output $\\mathbf{s}^\\ell$ represents the signed spike firing rate.\n\nDuring inference, the firing rate $\\mathbf{s}^\\ell$ is decomposed into $T$ ternary spike signals. The sign of $\\mathbf{s}^\\ell$ determines the spike polarity: positive spikes are emitted when $\\mathbf{s}^\\ell \\geq 0$, while negative spikes are emitted when $\\mathbf{s}^\\ell < 0$. It can be mathematically described as:\n\n$$\n\\mathbf{s}^\\ell[t] = \\mathrm{sign}(\\mathbf{s}^\\ell) \\cdot \\mathbf{1}(t \\leq |\\mathbf{s}^\\ell| \\cdot T)\n$$\n\n$$\n\\mathbf{s}^\\ell = \\frac{1}{T} \\cdot \\sum_{t = 1}^{T} \\mathbf{s}^\\ell[t]\n$$\nwhere $\\mathbf{s}^\\ell[t] \\in \\{-1, 0, 1\\}$. The front-loaded firing pattern concentrates all $|\\mathbf{s}^\\ell| \\cdot T$ spikes within the first $|\\mathbf{s}^\\ell| \\cdot T$ time steps, with spike polarity determined by the sign of $\\mathbf{s}^\\ell$.\n\n\n## Question 3: Have the authors compared their method to BNNs (with activations limited to +1 and –1)? Please support this with both theoretical analysis and experimental validation.\n**A**: BSA introduces a \"silent\" state (0) that better mimics the sparse activation characteristics of biological neurons. This inherent sparsity provides significant computational advantages in QK correlation calculations—zero-value positions can be directly skipped, whereas BNN's {-1,1} system enforces computation across all positions. Comparative experiments between binary SA and BSA within BERT architecture would effectively validate this theoretical advantage.\n\n|  | Spike-Driven | cola | qqp | qnli | sst-2 | sts-b | mrpc | rte |\n|:----------------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n| Bert | × | 59.7 | 91.4 | 92.1 | 93.2 | 90.1 | 86.3 | 72.1 |\n| Bert+Binary SA[3] | × | Fail | 63.18 | 50.54 | 52.75 | 83.2 | 81.2 | 53.8 |\n| Bert+BSA（Ours） | √ | 55.76 | 87.57 | 90.17 | 91.52 | 88.6 | 84.8 | 64.98 |\n\n\n\n## Reference:\n[1] Yao M, Qiu X, Hu T, et al. Scaling spike-driven transformer with efficient spike firing approximation training[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.\n\n[2] You H, Chen X, Zhang Y, et al. Shiftaddnet: A hardware-inspired deep network[J]. Advances in Neural Information Processing Systems, 2020, 33: 2771-2783.\n\n[3] Qin H, Ding Y, Zhang M, et al. Bibert: Accurate fully binarized bert[J]. arXiv preprint arXiv:2203.06390, 2022."}, {"review_id": "dXuVkdb5jy", "reviewer": "Reviewer_afBx", "summary": "This research identifies that existing Spiking Self-Attention (SSA) mechanisms are limited by the binary nature of spike trains, which restricts their ability to capture the polarity of membrane potential interactions, and they often omit the Softmax function to avoid energy-intensive multiply-accumulate operations, thereby failing to maintain row-stochasticity constraints on attention scores. To address these issues, the paper proposes a Bipolar Self-attention (BSA) paradigm, which effectively models multi-polar membrane potential interactions using ternary matrix products and introduces a shift-based Softmax approximation called Shiftmax to achieve low-entropy activation and partially maintain row-stochasticity without non-linear operations. Experimental results demonstrate that BSA achieves substantial performance improvements across various tasks, including image classification, semantic segmentation, and event-based tracking.", "questions": "1. Give more experiments on CIFAR10, CIFAR100, CIFAR10DVS, DVSGesture.\n2. The implementation of Ternary Spiking Neuron on the SDT-V3 at the inference time should give more details. According to your code and SDT-V3, the result of the Neuron should be the summation of spikes at all the time step, how do you allocate the spike signals at each time step?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\nThe primary strengths of this paper lie in its clear identification of two core limitations in SSA and its proposal of the innovative BSA solution, featuring Ternary Matrix Product (TMP) for comprehensive polarity information capture and the efficient Shiftmax for Softmax approximation; these contributions are supported by theory (such as Theorems 1 and 2 demonstrating TMP's statistical properties and correlation with real-valued products ) and extensive experimental validation across multiple tasks and architectures.\n\nWeaknesses:\n\n1、The implementation of Ternary Spiking Neuron on the SDT-V3 at the inference time is confused. \n2、This work  lacks experimens on neuromorphic datasets, such as CIFAR10DVS.\n3、The authors should provide more details about tracking and segmentation tasks", "was_revised": true, "final_justification": "First, I already believed this work met the acceptance criteria. During the rebuttal phase, the authors addressed every concern I had raised, so I raised my score accordingly.", "rebuttal": "Your suggestions have greatly contributed to improving our quality. We will address your concerns and questions one by one.\n## W1 and Q2: Details for the implementation of Ternary Spiking Neuron on the SDT-V3? \n**A**: \nThank you for your valuable suggestions. STD-V3 employs I-LIF[4] neurons as an innovative approach for decoupling training and inference processes. We would like to provide a detailed introduction to how ternary neurons are trained and perform inference in the V3 architecture.   In the training stage, the output of an I-LIF neuron is expressed as:\n$$\n\\mathbf{s}^\\ell = \\frac{1}{T} \\cdot \\left\\lfloor \\mathrm{clip}(\\mathbf{x}^\\ell, -T, T) \\right\\rceil,\n$$\nwhere $\\mathbf{x}^\\ell$ is the input current at layer $\\ell$, $T$ is the simulation time steps, $\\mathrm{clip}(\\mathbf{x}^\\ell, -T, T)$ constrains $\\mathbf{x}^\\ell$ to $[-T, T]$, and the symbol $\\left\\lfloor \\cdot \\right\\rceil$ denotes rounding to the nearest integer.\n\nThe output $\\mathbf{s}^\\ell$ represents the signed spike firing rate. During inference, $\\mathbf{s}^\\ell$ decomposes into $T$ ternary spike signals where spike polarity follows the sign of $\\mathbf{s}^\\ell$:\n$$\n\\mathbf{s}^\\ell[t] = \\mathrm{sign}(\\mathbf{s}^\\ell) \\cdot \\mathbf{1}(t \\leq |\\mathbf{s}^\\ell| \\cdot T)\n$$\n\n$$\n\\mathbf{s}^\\ell = \\frac{1}{T} \\cdot \\sum_{t = 1}^{T} \\mathbf{s}^\\ell[t]\n$$\nwhere $\\mathbf{s}^\\ell[t] \\in \\{-1, 0, 1\\}$. The front-loaded firing pattern concentrates all $|\\mathbf{s}^\\ell| \\times T$ spikes within the first $|\\mathbf{s}^\\ell| \\times T$ time steps, with spike polarity determined by the sign of $\\mathbf{s}^\\ell$.\n\n## W2 and Q1:  This work lacks experiments on neuromorphic datasets.  Giving more experiments on CIFAR10, CIFAR100, CIFAR10DVS, DVSGesture.\n**A**:  We evaluate our BSA on small-scale datasets, including CIFAR10, CIFAR100 and temporal neuromorphic datasets(CIFAR10-DVS and DVS128 Gesture). The detailed results on the QKformer structure for these small-scale datasets are presented in the following： \n\n| Dataset | Methods | Architecture | Param (M) | Time Step | Top-1 Acc (%) |\n|---------|---------|-------------|-----------|-----------|---------------|\n| CIFAR10 | S-Transformer[1] | S-Transformer-2-512 | 10.28 | 4 | 95.60 |\n|         | QKformer[2] | HST-4-384 | 6.74 | 4 | 96.18 |\n|         | QKformer[2]+BSA(ours) | HST-4-384 | 6.74 | 4 | 96.07 |\n| CIFAR100 | S-Transformer[1] | S-Transformer-2-512 | 10.28 | 4 | 78.4 |\n|          | QKformer[2] | HST-4-384 | 6.74 | 4 | 81.15 |\n|| QKformer[2]+BSA(ours) | HST-4-384 | 6.74 | 4 | 81.76 |\n| DVS128 | S-Transformer[1] | S-Transformer-2-256 | 2.57 | 16 | 99.3 |\n|        | QKformer[2] | HST-2-256 | 1.50 | 10, 16 | 98.3, 98.6 |\n|        | QKformer[2]+BSA(Ours) | HST-2-256 | 1.50 | 10, 16 | 98.5, 98.7 |\n| CIFAR10-DVS | S-Transformer[1] | S-Transformer-2-256 | 2.57 | 16 | 80.0 |\n|             | QKformer[2] | HST-2-256 | 1.50 | 10, 16 | 83.8, 84.0 |\n|        | QKformer[2]+BSA(Ours) | HST-2-256 | 1.50 | 10, 16 | 84.1, 84.6|\n\nThe experimental results demonstrate that BSA combined with QKformer can achieve overall performance improvements. However, as discussed in our analysis, the advantages of BSA are not prominently exhibited in classification tasks.\n## W3: The authors should provide more details about tracking and segmentation tasks\n**A**: Due to space constraints, detailed implementation specifics for segmentation and tracking tasks are provided in the Appendix, including comprehensive hyperparameter settings and network configurations. Code will be made available on GitHub.\n\n**Segmentation Implementation:** For ADE20K (150-category semantic segmentation benchmark), we employ the MMSegmentation framework [3] with 160,000 training iterations. The AdamW optimizer uses a learning rate of 2×10⁻⁴ and weight decay 0.05, with 1,500-iteration warm-up followed by linear decay. Experiments utilize 4 NVIDIA A100 80GB GPUs.\n\n**Tracking Implementation:** SDTrack-Base employs C=64 channels, while SDTrack-Tiny uses C=32 with reduced Stage 4 (6 blocks) and Stage 5 (10C channels, 2 blocks). Tracking heads differ in channel dimensions: Tiny outputs 256 channels versus Base's 512 in the first layer, with progressive halving in subsequent layers. Both maintain 128×128 template and 256×256 search image inputs. Training involves ImageNet-1K pretraining followed by fine-tuning on event-based tracking datasets via pair matching tasks, without data augmentation.\n\n## Reference:\n\n[1] Wang, Y., Shi, K., Lu, C., Liu, Y., Zhang, M., & Qu, H. (2023, August). Spatial-Temporal Self-Attention for Asynchronous Spiking Neural Networks. In IJCAI (pp. 3085-3093).\n\n[2] Zhou C, Zhang H, Zhou Z, et al. Qkformer: Hierarchical spiking transformer using qk attention[J]. Advances in Neural Information Processing Systems, 2024, 37: 13074-13098.\n\n[3] Contributors, MMSegmentation. \"MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.\" 2020.\n\n[4] Yao M, Qiu X, Hu T, et al. Scaling spike-driven transformer with efficient spike firing approximation training[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025."}]

Author rebuttal:

Meta-review: This paper introduces an SNN-based compression method that leverages Synaptic Pruning and Efficient Coding principles to reduce model size while preserving performance. Reviewers acknowledged the relevance of the problem and the potential benefits for neuromorphic deployment. The method is well-motivated, clearly presented, and demonstrates promising results with consistent performance retention under compression. Despite some limitations in dataset scale and scope of evaluation, the overall contributions are meaningful and timely for the community. Overall, the recommendation is acceptance.
Decision: Accept (spotlight)