Toward Relative Positional Encoding in Spiking Transformers
Authors: Changze Lv
Yansen Wang
Dongqi Han
Yifei Shen
Xiaoqing Zheng
Xuanjing Huang
Dongsheng Li
Venue: NeurIPS 2025 spotlight ()
Keywords: Spiking Neural Networks
Relative Positional Encoding
Spiking Transformers
Primary area: neuroscience_and_cognitive_science
TL;DR: We design relative positional encoding methods for spiking Transformers

Abstract: Spiking neural networks (SNNs) are bio-inspired networks that mimic how neurons in the brain communicate through discrete spikes, which have great potential in various tasks due to their energy efficiency and temporal processing capabilities.
SNNs with self-attention mechanisms (spiking Transformers) have recently shown great advancements in various tasks, and inspired by traditional Transformers, several studies have demonstrated that spiking absolute positional encoding can help capture sequential relationships for input data, enhancing the capabilities of spiking Transformers for tasks such as sequential modeling and image classification. However, how to incorporate relative positional information into SNNs remains a challenge.
In this paper, we introduce several strategies to approximate relative positional encoding (RPE) in spiking Transformers while preserving the binary nature of spikes.
Firstly, we formally prove that encoding relative distances with Gray Code ensures that the binary representations of positional indices maintain a constant Hamming distance whenever their decimal values differ by a power of two, and we propose **Gray-PE** based on this property.
In addition, we propose another RPE method called **Log-PE**, which combines the logarithmic form of the relative distance matrix directly into the spiking attention map.
Furthermore, we extend our RPE methods to a two-dimensional form, making them suitable for processing image patches.
We evaluate our RPE methods on various tasks, including time series forecasting, text classification, and patch-based image classification, and the experimental results demonstrate a satisfying performance gain by incorporating our RPE methods across many architectures.
Our results provide fresh perspectives on designing spiking Transformers to advance their sequential modeling capability, thereby expanding their applicability across various domains.
Our code is available at https://github.com/microsoft/SeqSNN.

Toward Relative Positional Encoding in Spiking Transformers
Changze Lv Yansen Wang2t
Dongqi Han
Yifei Shen?
Xiaoqing Zheng't Xuanjing Huang' Dongsheng Li
College of Computer Science and Artificial Intelligence, Fudan University
Microsoff Research Asia
(cz1v24)@m fudan. ddd.. {zhengxq, jjhhang)@fudan eedueeedddddduduuuuuaau. (yansenwang, hongqihan, dongsli jemicrosoft....mmmm com
Abstract
Spiking neural networks (SNNs) are bio-inspired networks that mimic how neurons in the brain communicate through discrete spikes, which have great potential in various tasks due to their energy efficiency and temporal processing capabilities. SNNs with self-attention mechanisms (spiking Transformers) have recently shown great advancements in various tasks, and inspired by traditional Transformers, several studies have demonstrated that spiking absolute positional encoding can help capture sequential relationships for input data, enhancing the capabilities of spiking Transformers for tasks such as sequential modeling and image classification. However, how to incorporate relative positional information into SNNs remains a challenge. In this paper, we introduce several strategies to approximate relative positional encoding (RPE) in spiking Transformers while preserving the binary nature of spikes. Firstly, we formally prove that encoding relative distances with Gray Code ensures that the binary representations of positional indices maintain constant Hamming distance whenever their decimal values differ by power of two, and propose Gray-PE based on this property. In addition, we propose another RPE method called Log-PE, which combines the logarithmic form of the relative distance matrix directly into the spiking attention map. Furthermore, we extend our RPE methods to two-dimensional form. making them suitable for processing image patches. We evaluate our RPE methods on various tasks, including time series forecasting, text classification, and patch-based image classification, and the experimental results demonstrate satisfying performance gain by incorporating our RPE methods across many architectures. Our results provide fresh perspectives on designing spiking Transformers to advance their sequential modeling capability, thereby expanding their applicability across various domains. Our code is available at //ghtps://gitte thub. wwt.moimccicccceecccc cttmm/miccooooo//egSM
1 Introduction
Spiking Neural Networks (SNNs) I are class of bio-inspired models designed to emulate the communication process of biological neurons, which transmit information through discrete spikes. In contrast artificial neural networks (ANNs) that operate continuous values, SNNs process information in the form of spikes occurring at precise moments in time. The temporal characteris- ties of spikes make SNNs particularly well-suited for tasks involving sequential data dynamic
'The work conducted during the internship of Changze Lv at Microsoft Research Asia. Conresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

environments, such as sensory processing 2 31 patch- image classification
4 5 time-series
forecasting [ and natural language processing 9
In the vanilla Transformer architecture 12), positional encoding serves as critical mechanism for modeling sequential dependencies in input data. Beyond absolute positional encoding, relative positional encoding (RPE) [3] [4] has emerged as an effective approach to represent inter-element distances, enabling models to capture relational patterns within sequences dynamically. Although RPE has demonstrated effectiveness in improving language modeling [3] and visual recognition tasks
[5]. its integration into SNNS remains underexplored. Existing methodologies for implementing positional encoding in spiking Transformers either suffer from ambiguous spike representations across positions [6). or neglect to integrate relative positional relationships entirely
7 Directly adapting current RPE techniques, such as Attention with Linear Biases (ALiBi) [13] and Rotary Position Embedding (RoPE) [1,, to spiking Transformers encounters significant challenges. Specifically, spiking neural architectures exhibit intrinsic difficulty in decoupling relative positional information from their sparse, event-driven representations. This limitation, empirically demonstrated in Section 5.2 underscores the necessity for rethinking RPE integration to align with neuromorphic computing principles, such as temporal sparsity and spike-based communication.
In this paper, we first propose that the Hamming distance [77] which quantifies the number of ones resulting from the XOR operation between two binary strings, serves as an appropriate metric for measuring relative distances when both the query and key matrices are binary. Consequently, we refine the spiking self-attention mechanism NOR (XNOR) logic operations. Then we present two novel approximation strategies for integrating RPE into spiking Transformers, while strictly preserving the binary activation dynamics inherent to spiking neurons. First, we propose Gray-PE, method exploiting the properties of Gray Code [18] to binarize relative positional distances. We theoretically prove that encoding relative distances via Gray Code ensures a constant Hamming distance between the binary representations of positional indices whose decimal differences equal 2", where n 2 (See Theorem IT. This property guarantees that any pair of positions separated by a relative distance of 2n in decimal space exhibits invariant Hamming distances in their Gray Code-encoded representations. Such invariance stabilizes positional relationship modeling for power-of-twe intervals, addressing critical limitation in existing spiking neural architectures. Second, we propose Log-PE, a method adapting insights from ALiBi [33 and Rectified RoPE [19). Log-PR integrates non-negative logarithmic transformation of the relative distance map directly into the spiking attention map, inducing a decaying sensitivity to positional relationships akin to windowed attention mechanisms. Moreover, extend the proposed RPE methods to their two-dimensional form, making them suitable for processing image patches.
by replacing dot- dot-product operations with exclusive-
To systematically evaluate the efficacy of our proposed RPE methods, we benchmark them across three cross-domain tasks: time series forecasting, text classification, and patch-based image classifica- tion. We employ three representative spiking Transformer architectures as backbones: Spikformer [4], the Spike-driven Transformer [5]. and QKFormer [20). Experimental results demonstrate consistent performance gains across all tasks when integrating our RPE approaches, affirming that explicit modeling of relative positional relationships addresses a critical limitation in existing spiking Trans- former designs. Furthermore, we conduct experiments on ablation study, long sequence modeling, and sensitivity analysis to validate the inner properties of our proposed RPE method.
This work establishes framework for integrating relative positional encoding (RPE) into spiking Transformers, advancing their applicability in neuro- neuro-inspired machine learning paradigms. Our primary contributions are summarized as follows:
Two RPE Methods for Spiking Transformers. To our knowledge, this study is among the first to explore RPE adaptations for spiking architectures systematically. While Gray-PE and Log-PE operate as principled approximations constrained by binary spike dynamics, they address critical gap positional modeling for neuromorphic computation.
Theoretical Foundations and Empirical Analysis. In addition to empirical validation, we provide theoretical guarantees demonstrating that our methods can partially encode relative positional information. Furthermore, we offer necessary analysis on the internal properties RPE and their robustness facing long sequences.
Consistent Performance Gains Across Architectures and Tasks. Our proposed RPE methods consistently improve the performance of spiking Transformers across various sequential tasks, including time-series forecasting and text classification.

---

(e) Spiking Neuron (LIF)
(b) Spiking Self-Attention
Hamming
Spske Marnos
Due
10011
0 0 O
Wi--ii
(dl
Riwry
Cuty
>+ Uptee
19
X
Linear
EN
2011
30.1
Cary
3011
300
2100
31.0
Figure 1: Illustration of preliminary knowledge. (a) Spike dynamics of LIF neurons. (b) Illustration of vanilla spiking self-attention in Spikformer [4]. (c) An example of Hamming Distance between two spike trains. (d) The calculation process of the classic Reflected Gray Code.
2 Related Work
Positional encoding serves as an indispensable mechanism for preserving the order of input elements in sequential modeling tasks. Traditional absolute positional encoding assigns static, predefined embeddings to individual tokens based on their sequential indices. In contrast, relative positional encoding (RPE) dynamically models pairwise distances between tokens, enabling the self- attention mechanism prioritize interactions based on their relative proximity. RPE allows the model to generalize across different sequence lengths and better capture relationships between tokens.
Despite the importance of PE in sequence- aware architectures, its application to SNNs is limited. Existing implementations, such as Spikformer 4 and Spike-driven Transformer (5 16 21, incorpo- rate a combination of convolutional layers, batch normalization, and spiking neuron layers to derive learnable positional encodings. However, we argue that this approach functions more similarly to spike-element-wiss residual connection [ than a conventional positional encoding module. A prin- cipled PE module should offer unique representations for positions, but the ppike-position matrices generated by these methods may lead to identical spike representations for different positions.
CPG-PE, proposed by 77 introduces spiking absolute positional encoding inspired by central pattern generators [22] generating unique periodic binary spike patterns for each position. However, their approach is based on absolute positional encoding and, thus. does not capture the time- transaaaalla invariance property in many sequential modeling problems, which, however, is an important advantage of relative positional encoding methods.
Preliminary
3.1 Spiking Neurons
We take the leaky integrate-and-He (LIF) neuron [1] as our building brick of SNNs, which is governed by the input current [tt,, influencing the membrane potential U[t and the spike output S(t] each time step t. The dynamic of the LIF neuron is captured by the following system of equations:
Ult) = H [1] S(t) + S(t), S(t] ((H(t) Utn)),
H Ult - 1] - (U[t - 11ft-1]-44 (reeves))))
(2)
where 7 is the membrane time constant. The spike S(t) will be triggered when the membrane potential (t) exceeds threshold Unh. right after which U[t] will be reset Ureset-
3.2 Spiking Self-Attention
Spiking self self-attention (SSA) is spiking version of self-attention 12 which was proposed in Spikformer 4 The vital design is to utilize discrete spikes approximate the vanilla self-attention mechanism. It can be written as:
Q. V = SN (BN(X WQ.kv)) E (0,1)TXXXD
(3)
where SA a spike neuron layer described in Equation The input is denoted X E (0, 1x where T' is the number of time steps. BN represents batch normalization, and is a scaling factor.

---

The attention map AttnMap is then computed as the dot product between Q and K"
SSA (Q, K, V)= SN(BN(( Q-K .v W)).
AttnMap
As result, the attention map AttnMap E 16 where No denotes the set of non-negative integers. The outputs of the SSA, as well as Q, K, and V, are all spike matrices containing only values of 0 and 1. The parameters Wo, Wx. Wv, and W are all learnable parameters.
Recent studies, including Spike-Driven Transformer (SDT) 16 21, SpikingResFormer 25
and QKFormer [20]. have proposed various modifications to the standard SSA mechanism. For our empirical evaluation, we selectively employ architectures demonstrating compatibility with our proposed relative position encoding methods.
3.3 Relative Positional Encoding
Relative positional encoding (RPE) in Transformers primarily introduces bias terms into the self- attention mechanism that dynamically encode pairwise token distances. A common implementation of RPE, as demonstrated in prior work [I5] 24], is formalized follows:
(9) dz
Q Q
Attention Q K, V) = Softmax
+ Ris
V.
(5)
AtinMap
Here, Ri.j represents the relative positional bias between the i-th query and the j-th key positions.
Beyond additive bias terms, another widely adopted form of RPE leverages relative positional embeddings directly in the attention computation, where query-position and key-position interactions parameterized separately. For example, RoPE [14] can be expressed as
(QR,) (KR,) )T
Attention(Q, K, V) = Softmax
.V,
(6)
Vdle
AttnMap
where R. and R2 are position-depenndet rotation operators applied to the i-th query and j-th key vectors, respectively.
A critical aspect of RPE is its adherence to distance consistency: the magnitude of R.,, is determined exclusively by the relative positional offset j ensuring that the model systematically differentiates between proximal and distant tokens. This property enhances the model's capacity to capture long- range dependencies and generalize across variations sequence length and structure.
3.4 Hamming Distance
The Hamming distance (17) between two binary strings of equal length is the number of bit positions at which the corresponding bits are different. Formally, for two binary strings A and B of length m,
m
{
if Ar B.,
dz (A, (A, B) Σ 8(A,, B.), where 8(A,, B,)
(7)
otherwise.
Hamming distance is suitable for measuring the relative distances when Q and K are spike matrices.
3.5 Gray Code
Gray Codes [18], also known as reflected binary codes, are binary numbering systems where adjacent values differ by precisely one bit. For non-negative integer æ, the standard binary reflected Gray Code G(x) is defined by the following bitwise operation:
G(x) @ (x > 1),
(8)
where denotes the bitwise XOR operation, and >> denotes the arithmetic right shift.
Since the preliminary knowledge involved extensive and loosely connected, we have provided Figure I to help readers visually grasp the key concepts of each section.

---

KNOR-Baaed SSA
(b) Gray-PE
(: 1 NAAND
[0 10,1
Σ
(Q@K): dime]
uunqueere
smuquerre
dn(6()), 2*)) = (2
[1. 1f n=
any position
1 13rstxte
(: TJSTta0
dim- 0
Position Index
2
(QBK)
XNOR (Broadoas)
du (G(O), G(2"))
(G)) G(1 X
Rig 7stxt=o
Implementation on XNOR-Based SSA:
Summation Alung Dimenssio
-[
((
K
GO
Summation D
Att M4 Pxtxa
Concalenate Dimension D
7: Time Step
Kc) Log-PE
(d) Two- Two-Dimensional Form for Image Patches
Sequence Length
A-1
Widths
D: Channel Dimension
0
1
3
Distance between Patch 0 and
4 for Gray-PE x
Not Operattoo
R2
Gray-PE V
@: XOR Operation
AttnMap
8
9
11
Implementation
G(): Gray Function
12
13
15
-[[
Q
G(b) G(w)
Hamming Distance
Farther distaray, scrulller Resi Heights
K
G(b) ((w)
Overview of Our Method. (a) XNOR-based spiking self-attention. We illustrate the computation flow for Q and K a PyTorch-style notation. (b) Gray-PE. Position indices differing by 2 exhibit consistent Hamming distance on their Gray code representations. Gray-PE is implemented by concatenating G(l) along the D dimension on both Q and K. (c) Log-PE. pre- assigned relative distance encoding map R,j No is added to the original attention map AttnMap. (d) 2D Form of Gray-PE. A 2D RPE more suitable than the ID version for image patches, as captures the spatial relationships more effectively.
Figure 2:
4 Method
4.1 Design Principles
Relative position encoding (RPE) aims to encode the relative distances between positional indices within sequence. In many spiking Transformers, such as Spikformer, Spike-Driven Transformer, and FKFomerr, both the Q and matrices are binary. Consequently. their relative distances can be computed using the Hamming distance, which corresponds the number of ones resulting from the XOR operation between Q and K. To better align with this Hamming distance-based similarity measure, replace the traditional dot-product spiking self-attention (SSA) mechanism with XNOR-based SSA. Inspired by RPE strategies in Transformers, we propose two approaches for incorporating relative distance information into spiking attention mechanisms: (1) Gray-PE: Gray- Code-based positional encoding concatenated to Q and K, and (2) Log-PE: logarithmic positional encoding applied directly to the attention map.
4.2 XNOR-Based Spiking Self-Attention
In the original Transformer 12), the attention map is computed via the dot product between the query and key matrices, AttnMap Q KTT which effectively captures similarity of and K. As mentioned above, in order to capture the relative distances of spiking matrices while effectively measuring the similarity, design the XNOR-based SSA. Unlike the dro-pur operation, XNOR accounts for both spiking state (1) and the resting state (0).
Formally, modify Equation 4 as follows:
AttnMap Σ
(Q + K),
where denotes the Not operation, denotes the XOR operation, and / represents the channel dimension. Note that every token in Q will perform XOR with every token in K. over the
5

---

channel dimension D to get Att E NTxLxx, shown in Figure Equation 4 should be set a smaller value or treated as a learnable parameter, ensuring that the firing rate of SN does not become excessively large. We will empirically demonstrate that this XNOR modification does not negatively impact the performance of the vanilla spiking self-attention.
(a). The scale factor o in
4.3 Gray-PE
We propose that the Gray Code can serve as an approximate approach to relative positional encoding for spiking Transformers. This is supported by the following Theorem T
Theorem 1. (Proof in Appendix A For two position indices differing by 2 2 0), their Gray
Code representations have a consistent Hamming distance. Specifically, V position i, we have:
[1 if =
du (G(( ((i
(10)
2 1.
As illustrated in Figure (b), the Hamming distance du (G(0), G(I)) and du (G G(2)) both equal 1 because their relative distance is i.e.. 2", n O. Similarly, du (G(0), G(2)) du (G(1), G(3)), and d ( (O),, G(4)) du (G(), G(5)) ). their relative distances are the power of 2. That said, Gray Code ensures the consistency of relative distance representations for every 2n (n 2 0) relative distance.
For implementation, we concatenate the Gray Code representations of each position index to both the query matrix Q E N2 LxD and key matrix K E No NTTxLDD, leaving the remaining operations unchanged. We concatenation instead of addition because Q and K are spike matrices, and addition would compromise their binary nature. Formally, the attention map AttnMap will be:
D
AttnMap Σ
(IQ G()) @ K G())),
(11)
2=
where G(-) represents the function that converts integers into their binary Gray Code representations. The vector I denotes an array of position indexes, specifically [0, 2, 1], where L is the sequence length of Q and K. denotes concatenation the channel dimension D.
Notably, the binary nature of Gray Code (comprising only and 1) aligns intrinsically with the spike-based computation paradigm, avoiding the need for floating-pon ppint operations that impose significant implementation overhead on neuromorphic hardware.
4.4 Log-PE
Although Gray-PE can partially capture relative distances, it faces significant challenges when the input sequence long or when the downstream task is highly sensitive to long-range dependencies. For instance, when L 2 102, the distinguishable range of relative distances under Gray-PE becomes constrained by its power-of-two quantization mechanism. To mitigate this, we propose Log-PE that integrates logarithmic positional bias into spiking-based self-attention. Specifically, we simulate Equation 5 and follow ALiBi [3] to directly add pre-assigned relative position map, denoted as Rij. to the attention map produced by SSA:
AttnMap = Σ ((Q0K)
RRijj where R = [Ra,] = [loog2 p
(12)
Here, [.] denotes the round-up function, is the sequence length, and i,j is position indices.
Figure 2 (c) shows illustration of Log-PE. Since the original AttnMap is matrix composed of non-negative integers, we aim to ensure accurate relative distance consistency while preserving the effectiveness of spiking self-attention Theoretically, if we set the R,, パー
could obtain complete RPE for the spiking Transformers. However, we choose not to pursue this solution, because for long sequence lengths L, the large values of +I would catastrophically overshadow the original spiking attention activations (See Appendix B Therefore, using the logarithmic form R.,; represents a compromise that balances the values between the spiking attention map and complete- RPE, while partially capturing relative position information.

---

4.5 Two-Dimensional Form for Image Patches
CNN-based SNN models, such as Spiking VGG 25 and SEw-ResNet [3, do not incorporate the concept of "positional encoding" in their spike representations. Vision Transformer (26] reformulated traditional image classification into a patch-based approach, dividing images into smaller patches. Unlike ID positional encoding, which only considers the linear sequence of patches, 2D RPE accounts for both the horizontal and vertical positions of the patches in the image grid. This ensures that the model can recognize the relative positions along a single axis and the crucial interactions between patches across both dimensions. We show our 2D form in Figure 2 (d), In our implementation, we assign horizontal and vertical positions with independent dimensions to store the Gray Code. Formally, the attention map AttnMap is:
AttnMap = Σ (QQ Il G(h) || G(w)] @ [K] || G(h) || G(w)))- 1=0
(13)
Here, h is the array of position indices, specifically h = [0, 1.2...... h - 1]. where h denotes the maximum patch index along the height axis. Similarly, w is along the width axis. As for the 2D form Log-PE, we can add Rhh i.j and R' G, on AttuMap, replacing the sequence length L in Equation 12 with h or However, in our pre-experiments, we found that spiking Transformers with Log-2D) failed to converge due to the excessive magnitude. Therefore, we abandon the 2D form of Log-PE.
Experiments
5.1
Datasets
To evaluate the RPE capabilities of the compared models, we conduct experiments on two sequential tasks: time-series forecasting and text classification. Following [61. choose 4 real-world datasets for time-series forecasting: Metr-la (27), Pems-bay [27], Electricity (28). Solar [28]. For text classification, we follow [7] and conduct experiments on six benchmark datasets: Movie Reviews (2). SST-2 33). SST-5, Subj, ChnSenti, and Waimai. Additionally, to demonstrate the versatility of
our RPE method in image processing, we perform patch-based image classification experiments on two static datasets, CIFAR and Tiny-ImageNet, and one neuromorphic dataset, CIFARIO-DVSS 2].
The details of these datasets, metries, and training hyperparameters provided in Appendix D
5.2 Time-Series Forecasting
We follow the SeqSNN 66 framework to conduct time-series forecasting experiments. Specifically, take Spikformer 4 Spikingformer [3T], Spike-driven Transformer (SDT) V1 (5), and the current visual state-of-the-art (SOTA) model, QKFormer 200. as the backbone architectures, We modify the SSA mechanism outlined in Section 4.2 to create two variants: Spikformer-XNOX and QKFormer- XNOR. SDT adopts variant of SSA, which makes it only able to integrate Log-PE but not Gray-PE. We present the performance of the compared SNN models with various positional encoding methods in Table [
The key findings are as follows:
(1) Directly applying RPE methods to spiking Transformers is ineffective. Specifically, Spikform- ers that are directly equipped with RoPE or ALiBi exhibit poor performance across all benchmarks. As discussed in Section [ argue that this limitation stems from the binary nature of spiking neurons during the computation of Q and K, which makes it difficult to disentangle positional information from sparse spiking activations.
(2) The XNOR modification does not impact the performance of the original SNN models. The average performance of Spikformer with Conv-PE is nearly identical to that of Spikformer- with Conv-PE. This suggests that our XNOR modification of the SSA does not affect the performance of the original SNN models.
(3) Gray-PE and Log-PE, enable spiking Transformers to achieve the best performance among their variants. CPG-PE is spiking version of absolute PE designed for SNNs. Spikformer and Former, when equipped with our proposed Gray-PE and Log-PE, consistently outperform all other corresponding variants.
7

---

Table 1: Experimental results of time-series forecasting on 4 benchmarks with various prediction lengths 6, 24, 48, 96. "PE" stands for positional encoding. "R" denotes relative PE, while "A" denotes absolute PE. "w/" denotes "with". The best results for each series of spiking Transformers are highlighted in bold font. t (L) indicates that the higher (lower) the better. Results highlighted with shading are ours. All results averaged across 3 random seeds.
Models
Type
Metrita
Penssayy LI
24
65
06
Electricity
9g
asr.
548
.30
-65
400
417
205
3173
482
533
.511
17.4
-311
-120
5417
Traesfrmer of ALIBI
K R
RSE.
1996
125
335
.200
.314
585
2513
-222
-207
-157
2509
-100
1551
446
679
.351
-526
2003
-504
1227
74L
100
1955
-20
.273
-800
.830
-970
.422
1968
747
72L
Tousfomer
x
331
7:5
122
00:1
3021
355
2013
715
1776
373
3722
193
Spikfoence Con--E (Orizinal)
A
-222
.223
1093
1897
-023
-144
.001
10/3
Spitfiemee of ALIB
0773
7756
-333
1.00
7500
.536
700
6770
1.04
7.50
1.00
1.01
(30)
1.01
1,83
1.00
1.03
-.43
1.00
313
1973
Syikfonner wi RiPE
R
2033
-035
-757
-3500
-555
-083
-296
.250)
-100
101.
1964
1025
-531
-256
-531
.441
-1.4
-090
2344
/335
.551
36.-
0440
-354
.454
-840
(123
Syikfrener sf CPG-IE
-230
1405
1027
-257
(190
505
535
-000
516
Spikfrener--e JUNH wi' Coen-irt
A
RSE.
K-
1039
713
-733
-313
8315
-000
010
350
-773
-015
-355
.250
.018
1995
1928
1125
.278
3021
.221
1227
791
1935
.255
[50)
10.1
.7.
376
7115
-325
.354
1942
123
Spikformer Cray-iFF
5:4
T24
6994
6775
1036
1:01
774
780
473
7.4
Spikfrener- XNOR mi Loge PE
R
RSE.
R--
710
708
876
.490
1776
(40)
022
-360
541
.005
-044
7758
734
125
.778
.300
3144
3400
7700
5500
Sptingfience soho ME (Unngmat)
500
730
612
106
.153
3700
155
700:
611
0775
6579
-256
100
151
013
091
765
17:1
2013
10"
.755
-340
-57
Sphirgffemeee- XNOM vo' Lingeatt:
RSEL
722
8.37
-156
200
.820
-716
.881
645
640
-584
-377
302
.211
1535
1911
.75
420
5000
573
.078
.265
71.
Sykkigfoenee XNOR Log
540
573
4119
1532
1.66
1162
AS1
775
2770
1336
531
SD Cuar (Organd))
1914
(236
-356
1489
1873
-206
-021
/532
.371
-570
SEDEVE tn CIG-T
A
RSF:
2011
372
318
775
-115
7116
5778
174)
1223
8:10
6617
1716
185
2330
(.21
534
710
637
7111
[6.]
-561
187)
-055
7115
370
1755
127
Log
-74
230
07-4
.918
-957
1:2
.250
716
734
QRFFomer of CPGUPE
0.9
740
.704
3314
-00
1085
.225
-315
-376
-012
-201
1511
1991
.285
-022
7551
2012
191
415
7094
307
.200
21.1
1851
J115
385
.152
-985
112
OKTOOmer-NNIR an Crny-itt
745
1411
318
374
-10
715
100
656
127
270
517
710
031
07.
077
785
722
QKFouuee XNOR wi PE
BSE.
2435
735
1351
885
1121
274
457
315
564
264
2256
2210
3336
51.4
(4) For long input sequences, Log-PE more effective than Gray-PE in capturing relative positional information. The input sequence length for Metr-la and Pems- is 12, whereas for Solar and Electricity, it is 168. On the equipped with Log-PE consistently outperform those with Gray-PE across nearly all prediction length settings. This result indicates that Log-PE PE more effective for processing long input requeeeeuueee sequences.
datasets Solar and Electricity, spiking Transformers
5.3 Text Classification
We conduct experiments to assess the efficacy of spiking Transformers with Gray-PE and Log-PE in text classification tasks. By comparing them against alternative PE techniques, we demonstrate their superior ability to model complex linguistic structures and contextual dependencies. Our experimental setup strictly adheres to the methodology outlined in 77. and the results are shown in Table 2
Table 2: Accuracy (%) on text classification benchmarks. Note that QKFormers fail to converge in the text classification task. Experimental results are averaged across 5 random seeds.
Model
Spike
Type
Param(M]
MR
English Datasee (Length- 122)
SST-2
Subj
SST-3
Chinew Dhitase Lnggh -
ChnSenti
Waimal
Asg.
Fine- tuned BEST
109.8
87.63in.4
02.91ac.47
95.90-9.16
50.41.0.15
80,
90.27
54.33
Spatoemer win FE:
100.55
81.21 aa
31 60lan
20.872003
17.25
Spiacemerr w/ CPG-PE
A. 110.4
62. 222.2229-0
82.00 ss
92 50
43.625
96 96.54gn
48.49
79.41
Spikformer XNOR wio PC
109.8
75 75.90
81.74:a.a am
01 50
41.88113e
85 duddanna
58. 08.661
77.20
Spiitormer XNUR wi Gray- PE
R 189.8
88.7548-45
84.52-9.3 29
02 50lann
4106in
87.418836
88. 88.000
80.13
Spikformer-XNea XNOR wi Log PE
R 189.8
83.88.0.4
84.64.0.3
92 50.0.0
44.52
87. $5.90..0. .4.34
88.05.0.25
80.35
Based on the results in Table 2 it is evident that our proposed Gray-PE and Log-PE PE significantly outperform the other spiking positional encoding methods across several key benchmarks. Both Gray- PE and Log-PE demonstrate superior accuracy on the English and Chinese datasets, with particularly notable improvements on MR, SST-2, Subj, and ChnSenti. However, the performance of RPE on the Waimai dataset is not as strong as that of CPG-PE. We attribute this to the nature of the dataset, which consists of user reviews often containing informal language, typos, or mixed expressions. This noise can hinder the model's ability to extract meaningful patterns. These results highlight the advantages of our proposed spiking RPE techniques, especially in handling the dependencies and varying word order in text classification tasks. Unlike spiking absolute PE, i.c., CPG-PE, which
8

---

struggles to adapt to the nuances of language, Gray-PE and Log-PE provide more flexible and context-sensitive representation, improving the model's ability to classify sentences accurately.
5.4
Patch-based Image Classification
Table 3:
Accuracy (%) on image classification benchmarks. Numbers with denote our implemen- tations. The best and second- best results are highlighted in bold and underlined formats, respectively.
The results with shading ours. Results are averaged across 4 random seeds.
Model
Splie
PF
1spe
Faram
CIFARIO
LMD
Auc
Faram LXU
CIPARIO-DYS
Arc
Faram (SD
CIPAR100
Aue
Param LXT
Tiny- Jay--ImageNet
Arc
Avg.
Viston Transtommer
9.32
96.73
9.56
$1.02
9.40
62.18
Spikformer w/ (..mmFFE (Ongima)
9.32
63.80
2.3V
78.10
9.30
77.04"
9:10
18.10
76.5.
Spikformer CPG-PE
8.17
95.05
2.06
18.00
77.82
48.52"
74.95
SpikfevmesXNNRR PP ID
78.12
74.05
Spikfermer-XNDD Cmay
1.90
78.70
804
78.45
73.39
In this section, we evaluate ViT-based SNNs, Spikformer, which adopts a patch-splitting processing approach. To enhance compatibility with this framework, extend Gray-PE into 2D form and integrate it into the patch-based architecture. The experimental results are summarized in Table 3 We draw conclusions that:
(1) Gray-PE enhances the performance of Spikformer while maintaining parameter efficiency. Both 1D and 2D variants of Gray-PE consistently improve classification accuracy. Notably, Gray- PE surpasses spiking absolute PE (CPG-PE). indicating its superior ability to model inter-pateh dependencies within images, even an approximation of RPE.
(2) The 2D variant of Gray-PE demonstrates superior performance over its 1D counterpart in processing image patches. Empirical comparisons between Spikformers equipped with Gray-PE ID and 2D reveal that the wo-dimeenionall form is highly effective. Specifically. Gray-PE 2D achieves an average accuracy improvement of 0.44% over Gray-PE ID.
Furthermore, we present the image classification performance of the state-of-the-art QKFormer integrated with our proposed RPE methods in Appendix
5.5 Capability of Processing Long Sequences
In this section, assess the effectiveness of our proposed relative positional encoding methods in handling long sequences within spiking Transformers. To this end, we use two text classification datasets characterized by long input samples: AGNEWS 32 and IMDB [33 Following 34 fix
the sequence max length to 1024 for AGNEWS and 2048 for IMDB. We train the Spikformer model using various positional encoding strategies on these datasets, and present the results in Table 4
As shown in Table 4 although Spik- former models lag behind the fine- tuned BERT in overall performance, both Log-PE and Gray-PE demon- strate effectiveness when handling long input sequences. Notably, Log- PE yields substantial performance im- provements, suggesting its strong suit- ability for processing long texts. This outcome is expected, as Log-PE is specifically designed to accommodate long-range dependencies.
Table 4: Accuracy (%) on 2 long text classification bench- marks. We set the sentence length to 1024 for AGNEWS and 2048 for IMDB.
Model
Spike
Type
AGNEWS
IMDB
Avg.
Fine-tuned BERT
94.50
92.10
93.30
Spilformer w/ Conv-Pt (Original)
A
83.54
79.06
$1.46
Spikformer w/ CPG-PE
84.70
79.47
82.00
Spikformee-XN
Gray-PE
81.92
Spikformer-XNO Log.Pt PB
R
86.77
80.46
83.62
5.6 Discussion on Hardware-Friendlines and Computing Efficiency
Although traditional SSA benefits from highly optimized matrix multiplication (GEMM) on GPUs, we would like to clar- ify that our XNOR-based SSA also retains computational efficiency for the following reasons: First, the core of XNOR-based baaed SSA relies on XNOR and bit-count opera- tions, which are natively supported by dig-
Table 5: Evaluation of both time consumption and GPU
memory usage for SNNs Electricity dataset.
Time Consumption CPL Memory Usage
Model
s/epoch
MB
Spikformer (Original)
137 48
10572.56
Spikforme-XXNO
130 66
10608.32
Spikformer w/ PPG.PE
140.56
10963.88
9

---

ital hardware and neuromorphic processors. These are much cheaper than foating-point multipli- cations and additions in terms of energy and hardware complexity. Secondly, many neuromorphic accelerators (e.g.. Loihi 35 TrueNorth (66] natively support spike-based bitwise logic, making our XNOR mechanism better aligned with the target deployment platform than conventional floating- point matrix products. Lastly, while matrix multiplication benefits from BLAS acceleration, XNOR and summation over dimensions are also highly parallelizable, and can be efficiently implemented using tensor intrinsics (e.g., bitwise_ xnor, popcount, reduce_sume
We benchmarked both time consumption and GPU memory usage for SNNs in time-series fore- casting task, mainly on the Electricity dataset with 24 of horizon length, as shown in 5 For more analysis on the hardware-friendlinee of Log-PE, please refer the Appendix
5.7 Analysis and Ablation
In this section, we analyze the following aspects: (1) The influence of internal properties in Gray-PE, (2) Ablation studies on XNOR and Log-Ph PE (shown in Appendix B).
Consider that: If the number of bits used for encoding relative positions in Gray Code is b, then the total num- ber unique encodings possible 2 We set the maximum sequence length is L, so relative distances range from 0 1. According the pigeonhole principle, if L 1 > 2", there will be least two distances that are rep- resented identically. This issue can be mitigated by increasing b cover the range of relative distances up to L 1. From (ag, we observe that for long-sequence datasets, such Solar and Electricity (Length = 168), number of bits should be at least 7 to avoid Gray-PE missing relative positional information. However, for shorter datasets like Metr-la (Length 12) and ChnSenti (Length 32), 5 bits are sufficient.
Time Seite
"hat Clandiiattin
0.75
0.05
Hi
5012
0.55
Oriseti
0.49
De Mumber or Ciay Code
De Dekumber Gnay Code
(b)
Figure 3: Spikformer-XNOR with Gray-PE across various bit numbers ranging from 2 to 12 on (a) time-series forecast- ing tasks and (b) text classification tasks.
6 Conclusion
In this work, we have designed several RPE methods for spike Transformers. Our approach preserves the spiking nature of SNNs while effectively representing relative positions. Experimental evaluations on time series forecasting, text classification, and image classification demonstrate significant perfor- mance improvements. These empirical results, together with theoretical analysis of the proposed RPE methods, highlight the potential to enhance the versatility and applicability of SNNs across various domains. Future work and limitations are discussed in Appendix F
Broader Impact
This work aims to advance the field of spiking neural networks. We hope our work can open new avenues for embedding relative positional encoding within SNNs, thereby expanding their applicability across a wide range of domains. We do not see negative societal impacts of this work.
Acknowledge
The authors would like thank the anonymous reviewers for their valuable comments. This work was partially supported by the National Natural Science Foundation of China (No. 62076068).
10

---

References
[1]
Wofgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural Networks, 14:1659-1671, 1997.
[2] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar An event- stream dataset for object classification. Frontiers in Neuroscience, 11, 2017.
[3] Wei Fang, Zhaofei Yu, Yanqing Chen, Tiejun Huang. Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In Neural Information Processing Systems, 2021.
[4] Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
[5] Man Yao, JiaKui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, XU Bo, and Guoqi Li. Spike- driven transformer. In Shirth- seventh Conference on Neural Information Processing Systems, 2023.
[6] Changze Lv, Yansen Wang, Dongqi Han, Xiaoqing Zheng, Xuanjing Huang, and Dongsheng Li. Efficient and effective time-series forecasting with spiking neural networks. In Forty-first International Conference on Machine Learning (ICML), 2024.
[7] Changze Lv, Dongqi Han, Yansen Wang, Xiaoqing Zheng, Xuanjing Huang, and Dongsheng Li. Advancing spiking neural networks for sequential modeling with central pattern generators. Advances in Neural Information Processing Systems, 17-26445-,6940, 2024.
[8] FENG SHIBO, Wanjin Feng, Xingyu Gao, Peilin Zhao, and Zhiqi Shen. Ts-lif: A temporal segment spiking neuron network for time series forecasting. In The Thirteenth International Conference on Learning Representations, 2025.
[9] Rui-Jie Zhu, Qihang Zhao, Guoqi Li, and Jason Eshraghian. SpikeGPT: Generative pre-trained language model with spiking neural networks. Transactions on Machine Learning Research, 2024.
[10] Changze Lv, Jianhan Xu, and Xiaoqing Zheng. Spiking convolutional neural networks for text classification. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
[11] Xingrun Xing, Zheng Zhang, Ziyi Ni, Shitao Xiao, Yiming Ju, Yequan Wang, Jiajun Zhang, Guoqi Li, al. Spikelm: Towards general spike-driven language modeling via elastic bi-spiking mechanisms. In Forty-first International Conference on Machine Learning, 2024.
[12] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[13] Ofir Press. Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2017.
[14] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputinng 168:127063, 2024.
[15] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[16] Man Yao, JiaKui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, Bo XU, and Guoqi Li. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next gext-generation neuromorphic chips. In The Twelfth International Conference on Learning Representations, 2024.
11

---

[17] Richard W Hamming. Coding and information theory. Prentice-Hale Inc., 1986.
[18] Frank Gray. Pulse code communication. United States Patent Number 2632058, 1953.
[19] Jianlin Su. ReRoPE, 2023.
[20] Chenlin Zhou, Han Zhang, Zhaokun Zhou, Liutao Yu, Liwei Huang. Xiaopeng Fan, Li Yuan, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. QKFormer: Hierarchical Spiking Transformer using Q-K Attention. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[21] Man Yao, Xuerui Qiu, Tianxiang Hu, Jiakui Hu, Yuhong Chou, Keyu Tian, Jianxing Liao, Luziwei Leng, Bo Xu, and Guoqi Li. Scaling driven transformer with efficient spike firing approximation training. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2025.
[22] Eve Marder and Dirk Bucher. Central pattern generators and the control of rhythmic movements. Current biology, (1(23):986--999)) 2001.
[23] Xinyu Shi, Zecheng Hao, and Zhaofei Yu. Spikingresformer: bridging resnet and vision transformer in spiking neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5610-5619, 2024.
[24] Yanbin Hao, Diansong Zhou, Zhicai Wang, Chong- Wah Ngo. and Meng Wang. Posmlp-viden spatial and temporal relative position encoding for efficient video recognition. International Journal of Computer Vision, 132((1))5820-5844 2024.
[25] Abhromil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13:95, 2019.
[26] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[27] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. In International Conference on Learning Representations, 2018.
[28] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41 International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 95-104, 2018.
[29] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, 2005.
[30] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In EMNLP. 2013.
[31] Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Han Zhang, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven driven residual learning for transformer-bawe spiking neural network. arXiv preprint urXiv: 11954, 2023.
[32] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[33] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the asxociation for computational linguistics: Human language technologies, pages 142-150.
2011.
12

---

[34] Zongyi Li, Jianhan Xu, Jiehang Zeng, Linyang Li, Xiaoqing Zheng, Qi Zhang, Kai Wei Chang, and Cho-Jui Hsieh. Searching for an effective defender: Benchmarking defense against adversarial word substitution. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 3137-3147, 2021.
[35] Mike Davies, Narayan Srinivasa, Tsung- Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. leee Micro, 38(1):82-99)) 2018.
[36] Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, al. Truenorth: Design and tool flow of 65 mw I million neuron programmable neurosynaptic chip. IEEE transactions on computer-aided design of integrated circuits and systems, 34 (10): 1537-1557, 2015.
[37] Zhiyang Chen, Yousong Zhu, Chaoyang Zhao, Guosheng Hu, Wei Zeng, Jinqiao Wang. and Ming Tang. Dpt: Deformable patch-baser transformer for visual recognition. In Proceedings of the 29th ACM international conference on multimedia, pages 2899-2907, 2021.
[38] Usman Muhammad, Md Ziaul Hoque, Weiqiang Wang, and Mourad Oussalah. Patch-based discriminative learning for remote sensing scene classification. Remote Sensing, 14(23):5913, 2022.
[39] Li Zhu, Chenglong Jiang. and Minghu Wu. A patch information supplement transformer for person re-identification. Electronics, 12(9):1997, 2023.
[40] Hongyi Wang, Yingying Xu, Qingqing Chen, Ruofeng Tong, Yen-Wei Chen, Hongjie Hu, and Lanfen Lin. Adaptive decomposition and shared weight volumetric transformer blocks for efficient patch-free 3d medical image segmentation. IEEE Journal of Biomedical and Health Informatics, 27(10)4885 4865, 2023.
[41] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412,6981 2014.
[42] Changze Lv, Tianlong Li, Jianhan Xu, Chenxi Gu, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Spikebert: A language spikformer learned from bert with knowledge distillation. 2023.
[43] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2018.
[44] Xinhao Luo, Man Yao, Yuhong Chou, Bo Xu, and Guoqi Li. Integer-valued training and spike-driven inference spiking neural network for high- performance and energy -negg-effficient object detection. European Conference on Computer Vision, pages 253-272. Springer, 2024.
[45] Zhanfeng Liao, Yan Liu, Qian Zheng, and Gang Pan. Spiking nerf: Representing the wolldwrld geometry by discontinuous representation. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pages 13798, 2024.
13

---

A Proof of Theorem 1
This section provides a detailed mathematical proof of Theorem T We use the standard Reflected Binary Code (RBC) (x)) as our Gray Code:
Definition 1. The Reflected Binary Code (Gray Code) of an integer x is defined as:
G(x) = x @ (x > 1),
(14)
where denotes the bitwise XOR operation, and >> denotes the arithmetic right shift.
and we restate Theorem hher:
Theorem 1. For any non-negative integer n, and for any pair of decimal integers a and b = 2", the Hamming distance between their Gray Code representations G(a)) and G(b) is consistently:
[1 in =0
du dn(G(o), (2 if 1
(15)
Here, Hamming distance is the number of different bits between two binary representations.
Proof. For n consider b a 2 We analyze the XOR of their Gray Codes:
G(a) @ G(a + 2") |a e (a > 1)] + [a + 2") @ ((a + 2") > 1)]
(16)
Using the associativity and commutativity of XOR, we regroup:
G((a) + G(a + 2") = [a (a + 2 @ [(a > 1) e ((a + 2n) > 1)]
(17)
Let us denote:
A = a (a A2 (a >> 1) + ((a 2n) > 1).
(18)
Case 1: n== 0
In this case, b = 1, and it is well known that adjacent integers in the Gray code differ by exactly one bit. Therefore, we have du (G(a), G(b)) = du (G(a), G(a 1)) = 1.
Case 2: n 2
We consider two subcases based the bit at position n in
Subcase A: Bit is 0
Then a + 2" flips bit n from 0 to I, with no carry. Hence:
A 2", A2 = 2n-1
(19)
Therefore,
G( @ G(b)
2" @ 2n
(20)
This value has exactly two bits set (at positions n and n 1), so the Hamming distance
Subcase B: Bit n in a is
Then adding 2n to a causes carry from bit upwards. Let e be the smallest index greater than such that bit is 0; bits n through e are all 1. Then:
A
+ (a + 2n)
(21)
(2),
has ones bits n through e. We denote ((2) for binary representation. Similarly,
A = (a> ((++))>>)-
Σ 2 L...0 (2),
(22)
1=
c-x+1 x-1
14

---

has ones at bits n through 1. Thus,
G(a) @ G(b)
A + 2
Σ ビ=--1
2
2" 2
(23)
has ones only positions and 1, and all other bits canceled due to alignment. The result has exactly two bits set, so the Hamming distance is 2.
Combining all cases, we conclude that for any non-negative integer n!:
1 if 0, if 2 1.
G(a 2*))
(24)
This rigorously proves the observed property of Gray Codes concerning the Hamming distance between numbers differing by powers of two.
B Ablation Study on Log-PE and XNOR
In this section, compare the performance of vanilla spiking Transformers with Log-PE, XNOR variants with Log-PE, and models equipped with complete relative positional encoding (C-RPE). As discussed in Section 4.4 C-RPE is implemented by setting Ra, = iー
h-1 and adding directly to
the attention scores.
As shown in Table ST both Spik- former and QKFormer with C-RPE perform significantly worse than their Log-PE counterparts, with some vari- ants failing to converge entirely. This degradation attributed the overly large positional encodings disrupt- ing the training dynamics. Further- more, observed from vanilla Spik- former with Log-PE, we confirm that the dot product, which does not use Hamming distance to measure relative distance, is not suitable for RPE meth- ods.
Table $1: Ablation study on XNOR and Log-PE. We take the time-series forecasting performance of SNNs Metr-la and Electricity as examples. C-RPE denotes Complete RPE. (J) indicates that the higher (lower) the better. denotes failure to converge.
Metisla (L = 12)
Electricity (L 168)
Model (Prediction Length 24)
RT
RSB
R'T
RSB
Spiktormer Log-PE
481
.733
.710"
1.03"
Spikformer- -NOR w/ Log-PE
535
7719
.97
.30
Spikformer XNOR w/ C-RPE
158*
967*
7710*
1.03
QKomme w/ Log-PE
475
788
.700
LO3
QKFormer XNOR Log PB
541
775
.974
.285
QKForner XNOR C-RPE
430
.811
.71''
1.3'
C Performance of QKFormers on Image Classification
In this section, we conduct experiments current SOTA spiking Transformer, QKFormer 20
Table S2: Accuracy (%) of QKFormer on image classification benchmarks. Numbers with denote implementations. "PE" stands for positional encoding. "R" denotes relative PE, while "A" denotes absolute PE. Results are averaged across 4 random seeds.
Model
Spiie
PE
Type
CIFAR10
CIPARIO-DVS
CIFAR100
Tiny ImageNet
(M1)
Kemer Com (Ongin))
0.74
30.32
0.75
QKfoemer w/ CPG PC
7.01
06.50
1.58
82.00
7.04
80.52
7.08
56.75
78.89
OKToemer-XNDD XNOR w/ Gray-
R 6.02
06.22
1.11
82.20
6.04
80.88
6.06
57.21
70.05
Bekkomme--NUxs w/ Gray- 20
R
6,02
96.36
1.41
80.10
6.84
8082
6,06
$7.90
79.56
As shown in Table S2 we find that: QKFormer exhibits insensitivity to positional encoding in image classification. FKFormer exhibits minimal performance gains, or even degradation, when augmented with PE techniques, including both CPG-PE and Gray-PE. We attribute this to its attention design, which aggregates queries along the channel dimension before dot product with keys. This design inherently biases the model toward spatially specific features while suppressing temporal dependen- cies. Previous studies (37 40 have shown that patch-based image classification primarily focuses on spatial (i.c., channel- information rather than the sequential dependencies between patches.
15

---

This contrasts with sequence modeling tasks such as time-series forecasting and text classification, where capturing inter-token dependencies crucial. In image classification, our positional encoding encourages the model to emphasize sequential relationships between patches, which introduces conflict with the QKFormer's attention mechanism, ultimately hindering performance in this domain.
D Experimental Settings
D.I Datasets
D.1.1 Time-series Forecasting
We strictly follow the dataset settings of CPG-PE 77..
The datasets we used are as follows: Metr-la Average traffic speed data collected from highways in Los Angeles County. Pems-bay [27]: Average traffic speed data from the Bay Area. Electricity 28 Hourly electricity consumption data in kilowatt- (kWh) of 321 clients. Solar [28]: Solar power production. The detailed statistical characteristics and distribution ratios for each dataset are presented below:
227
Table S3: The statistics of time-series datasets.
Dataser
Samples
Variables
Obscrvation Length
Train- Valid- Train-Valid- Test Ratio
Metr-la
34, 272
207
12, (short-term)
(0.7.0.2,0.1) (0.7.0.2.0.1)
Pems--uy
52, 116
325
12, (short-term))
Solar-energy
52, 560
137
168, (long- term)
(0.6,0.2,02 0.2) (0.6,0.2, 0.2)
Flectricity
26,304 304
321
168, (long-term)
D.1.2 Text Classification
For text classification, we follow 10] to conduct experiments on six easy discrimination tasks, covering both English and Chinese datasets. Here are the datasets we used in text classification experiments:
AGNEWS [32 is large-scale text classification benchmark derived from AG's corpus of news articles, containing 120, 000 training samples and 7, 600 valid samples evenly distributed across four categories World, Sports, Business, and Science/Technologe IMDB [33) is benchmark for binary sentiment classification, containing 50, 000 movie reviews labeled as positive or negative, split evenly into training and test sets to evaluate natural language understanding and opinion mining models. The MR dataset, which stands for Movie Review, contains movie-review documents labeled based on their overall sentiment polarity (positive or negative) subjective rating (29] SST-5 includes 11, 855 sentences from movie reviews for sentiment classification across five categories: very negative, negative, neutral, positive, and very positive [30]. SST-2 the binary version of SST-5, containing only two classes: positive and negative. The Subj dataset is designed to classify sentences as either subjective or objective"] ChnSenti consists of approximately 7, 000 Chinese hotel reviews, each annotated with positive or negative label" Waimai contains around 12, 000 Chinese user reviews from a food delivery platform, intended for binary sentiment classification (positive and negative
D.1.3 Image Classification
Here are the datasets we used in image classification experiments:
The CIFAR dataset is one of the most widely used henchmarks for image classification, comprising a collection of 60, 000 color images, each with a resolution of 32 x 32 pixels. These images are partitioned into 50, 000 training samples and 10, 000 test samples. The dataset includes 10 classes, each containing 6, 000 images, and spans a variety of object categories such as airplanes, cars, birds, and cats. The relatively low resolution of the images makes the dataset challenging benchmark for evaluating model performance in small-scale image classification tasks.
The Tiny- ImageNe dataset is a simplified subset of the original ImageNet, designed for efficient experimentation in image classification and deep learning research. consists of 200 object classes,
https://www.cg cornell. http://ww..ooee...ee///eeeeeeeee ott.e//e.eueeeeeooooo/iieierrrer
ChnsontiCorp_hhaaall.ooo
httos://raw.gtthuuueeee
16

---

each containing 500 training images. 50 validation images, and 50 test images (totaling 100, 000 training, 10, 000 validation, and 10, 000 test images). All images are downsampled to a resolution of 64 x 64 pixels, balancing computational feasibility with visual complexity Designed for efficient deep learning research, it reduces computational costs while maintaining diversity.
The CIFARIO-DVS dataset represents a neuromorphic adaptation of the original CIFARIO set, where static images have been converted into dynamic representations that simulate the recording capabilities of a Dynamic Vision Sensor (DVS) camera. Unlike traditional cameras, a DVS captures changes in the scene as individual events, rather than capturing full-frame images at fixed time intervals. This conversion results in a dataset that is more aligned with how biological vision systems process information. The CIFARIO-DYS dataset consists of 9, 000 training samples and 000 test samples, with higher resolution of 128 x 128 pixels compared to the original CIFARIO 10. The event-driven nature of this dataset presents unique challenges in terms of processing and model adaptation, as it requires handling sparse, asynchronous event streams rather than dense, synchronous pixel data. This dataset particularly valuable for testing models designed for neuromorphic systems and event weentbbased vision tasks, offering more realistic and biologically plausible approach to image classification.
D.2 Time-series Forecasting
Metrices The metrics used in time-series forecasting are the coefficient of determination (R²) and the Root Relative Squared Error (RSE).
M
C
L
ym
Y mm )2
n²= MCL m=
(25)
Y )
E=1
D==l
M
Ym
Ym
RSE
(26)
Ym
YI2
In these formulas, M represents the size of the test set, C' denotes the number of channels, and L signifies the length the predictions. is the average of Y The term Y refers to the I-th future value of the e-th variable for the m-th sample, while Yc, I represents the mean of y I across all samples. The symbols Y and y are used to denote the predicted values. Compared to Mean Squared Error (MSE) or Mean Absolute Error (MAE), these metrics exhibit greater resilience to the absolute values of the datasets, making them especially valuable in time- teeessriess forecasting tasks.
Model Architecture All SNNs take 4 time steps for spiking neurons. We construct all Spikformer as 2 blocks, setting the feature dimension as 256, and the hidden feature dimension FFN as 1024. As for QKFormer, we set the block number as 4, 2 of which are QK blocks and the other 2 are Spikformer blocks.
Training Hyper-parameters we set the training batch size 32 and adopt Adam 41 optimizer
with a cosine scheduler of learning rate x 10-4 An early stopping strategy with a tolerance of 30 epochs is adopted. For other configurations, we honestly follow the SeqSNN [rameworkk/propopee by 16). We conducted time-series forecasting experiments on 24G-V100 GPUs. On average, a single experiment takes about 1 hour under the settings above.
D.3 Text Classification
Model Achirecture All Spikformers are with 12 encoder blocks and 768 feature embedding dimension. We have substituted layer normalization of SpikeBERT [42 with batch normalization in our directly-trained Spikformer models for text classification tasks.
Training Hyper-parameters We directly trained Spikformers with arctangent surrogate gradients on all datasets. We use the BERT-Tokenizer in HuggingfacQ to tokenize the sentences to token
https://hugginggaacecom
17

---

sequences. We pad all samples to the same sequence length of 256. We conducted text classification experiments on RTX-3090 GPUs, and set the batch size 32, optimizer as AdamW 43 with weight decay of 5 10-3, and set cosine scheduler of starting learning rate of 5 x 10-2. What's more, in order to speed up the training stage, we adopt the automatic mixed precision training strategy. On average, single experiment takes about 1.5 hours under the settings above.
D.4 Image Classification
Model Architecture For all Spikformer models, we standardized the configuration include 4 time steps. Specifically, for the CIFARIO and CIFARI00 datasets, the models were uniformized with encoder blocks and feature embedding dimension of 384. For the CIFAR dataset, the models were adjusted to have 2 encoder blocks and a feature embedding dimension of 256. For all OKFormers, we set the block number as 4, where blocks are QK blocks and the other are Spikformer blocks.
Training Hyper-parameters We honestly follow the experimental settings in Spikformer 4 and QKFormer [20]. whose source code and configuration files are available at ht //gg thwb.......................... com/ and bettps//ggihhe epochs are quite big (300 400 epochs) in their settings, we choose to one 80G-A100 GPU, and it takes about 3 hours to conduct a single experiment, on average.
tt-Zhou/spikkorrr
com/ zwouchenlin2099 QKFormer] As the training
E Analysis on Hardwaree-rieendlines of Log-PE
First, Log-PE doesn't need to perform logarithmic operations directly on hardware during inference.
Specifically, the relative position bias is defined as R.,, [ogg [-->|+ 1]. where [-] denotes
L-1
the ceiling cround-up) function, and L is the maximum sequence length. Since this bias depends only on the relative positions and the predefined sequence length, the entire bias matrix can be computed offline and stored ahead of time, eliminating the need for any runtime computation.
Secondly, even if one wishes to compute the logarithmic transformation on hardware, this can be efficiently achieved using lookup table (LUT) implementation. Given an unsigned integer input of N bits, we partition the input range into K intervals. Each interval is approximated using piecewise linear function y = ax b, with the parameters (a, b) stored the LUT. The total LUT storage cost is: K (N + 2P) bits ~ K-(N-2P) bytes, where P is the bit width of the parameters.
This strategy is similar existing SNN approximations for exponential/leaky functions and has been successfully deployed in many types of neuromorphic chips, such as Intel Loihi (35). Hence, the hardware implementation of Log-PE is efficient, low-cost, and practically feasible.
F
Limitations and Future Work
F1
Limitations
Despite the promising enhancements introduced by our relative positional encoding method for spiking Transformers, several limitations must be acknowledged. Firstly, the current implementation may encounter scalability issues when applied to extremely long input (such as ultra-long texts with the length of 10240) sequences. Additionally, while Gray-PE and Log-PE effectively preserve binary spike representations, they may limit the flexibility and adaptability of the encoding scheme across diverse data modalities and task requirements. Furthermore, our evaluations have heen confined to specific applications such as time series forecasting, text classification, and image patch classification, which may not fully capture the method's 's performance in other domains, such as object detection and real-world geometry representation (45]
44
F.2
Future Work
Future work should focus on optimizing the Gray Code-based RPE to enhance its scalability and efficiency, enabling its deployment in larger and more intricate SNN models. Exploring alternative encoding strategies or hybrid approaches could provide greater flexibility and improve the robustness
18

---

of positional encoding across various data types and tasks. Expanding the scope of evaluation to include wider range of applications would offer a more comprehensive understanding of the method's effectiveness. Additionally, integrating Gray Code-based RPE with other advanced neural network components, such as attention mechanisms or neuromorphic hardware, could further elevate the performance and practical utility of SNNs. These efforts will contribute to the advancement of more versatile and powerful biologically inspired neural network architectures.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have clarified our claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations and future work in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

We have provided the full set of assumptions and a complete (and correct) proof in the Method Section and Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have shown our experiment results in the Experiment Section. We have submitted our source code in the Supplementary Material. We will upload our code and data to GitHub upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have submitted source code in the Supplementary Material. We will upload our code and data to GitHub upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht
/ //nips. cttps.cccpbllll celllicgguiiio ded//odeSummmm/m ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have shown our experimental settings and implementation details in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our reported results are all averaged over several random seeds. We have reported the standard deviation of the results in Table 2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the compute resource in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:://wwuurppo.ccc///////eeeee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive societal impacts and negative societal impacts of the work in the Broader Impact Section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The datasets used in the paper are all public datasets. Please refer to Appendix for details of datasets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com commdaaasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy httee////eerrrsssccccoofrreeeees for what should or should not be described.
26


Reviews:
[{"review_id": "AevZQbkp2Z", "reviewer": "Reviewer_7uot", "summary": "This paper develops two novel methods (Gray-PE and Log-PE) to integrate relative positional encoding into spiking Transformers, overcoming a key challenge. \n1. ​**​Gray-PE:​**​ Encodes positions using Hamming distance that keeps digital patterns similar for close positions.\n2. ​**​Log-PE:​**​ Adds relative distance patterns using logs to the attention mechanism.\nThese methods effectively encode sequence positions while preserving the essential binary nature of spikes.", "questions": "**Question 1: Computational Overhead of Gray-PE**  \nGiven the formulation of Gray-PE, which expands the channel dimensionality of both Q and K, a quantitative assessment of its computational overhead would be valuable. Specifically:  \n- What is the magnitude of channel dimension expansion introduced by concatenating Gray codes?  \n- Has the authors conducted empirical measurements to quantify the resultant increase in computational complexity (e.g., FLOPs, memory footprint, or latency)?  \n\n**Question 2: Boundary Effects in Log-PE**  \nThe proposed logarithmic positional encoding (Log-PE) employs the formulation:  $ R_{i,j} = \\left\\lceil \\log_2 \\left( \\frac{L-1}{|i-j|+1} \\right) \\right\\rceil $, where the maximum bias scales with $\\lceil \\log_2(L-1) \\rceil$ (e.g., $R_{\\max}=10$ for $L=1024$). Could such values overwhelm the original attention map magnitudes? Most positional encoding schemes introduce subtle biases without dominating semantic features.  Has the distribution of the original attention map been changed significantly after using Log-PE? \n\n**Other questions**  \nTable 1 seems too complex to understand the details. The author should import the accessibility.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. Provide theoretical justification for Gray-PE and Log-PE, especially preserving the essential binary nature of spikes. .\n2. Empirically demonstrate consistent performance improvements across diverse tasks (time series, text classification, image classification) and architectures. \n\nWeakness:\nSee questions.", "was_revised": true, "final_justification": "I hope the authors can include the ImageNet results in the final version. I have no further questions.", "rebuttal": "Thank you for your thoughtful comments and suggestions, which are valuable for enhancing our paper. We are pleased that our contributions are well recognized.\n\n## 1. Computational Overhead of Gray-PE (Q1)\n\n1. What is the magnitude of channel dimension expansion introduced by concatenating Gray codes?\n\nThanks for the question. The number of additional channels introduced by Gray-PE depends on the maximum sequence length. Specifically, to uniquely encode positions up to length $L$, we require $\\lceil \\log_2 L \\rceil$ bits. For example, for $L = 1024$, only 10 additional channels are needed.\nWe empirically analyze this relationship in Section 5.6, Figure 3(b), showing that a small number of bits (typically 5–10) is sufficient to preserve relative distance information for common SNN sequence lengths.\n\n2. Has the authors conducted empirical measurements to quantify the resultant increase in computational complexity (e.g., FLOPs, memory footprint, or latency)?\n\nIn response to your suggestion, we have conducted empirical measurements on the Electricity time-series forecasting task (prediction horizon = 24), using equal batch size and input length across all models. The results are summarized below:\n\n|Model|Time Consumption (s/epoch)|GPU Memory Usage (MB)|\n|-|-|-|\n|Spikformer (Original)|137.48|10572.56|\n|Spikformer-XNOR|139.66|10608.32|\n|Spikformer-XNOR w/ Gray-PE|140.56|10963.88|\n\nThese results show that **XNOR-based SSA and Gray-PE introduce negligible additional overhead** in both time and memory. The memory footprint increase is primarily due to the additional Gray code channels (≤10), and the runtime increase is <2.3% per epoch.\n\n## 2. Boundary Effects in Log-PE (Q2)\n\n1. Could Log-PE values overwhelm the original attention map magnitudes?\n \nThis is indeed an important point. The answer is No. As mentioned in **Line 201, Section 4.4**, we deliberately adopt a logarithmic (rather than linear or inverse) transformation to avoid overwhelming the spike-based attention map. In fact, we attempted a \"complete\" relative position encoding (e.g., $R_{i,j} = \\frac{L-1}{|i-j|+1}$), but found it led to severe degradation in model performance due to value collapse in the attention map.\nWe report this result in Table S1 in Appendix B, confirming that Log-PE is a balanced approximation that preserves performance and numerical stability.\n\n2. Has the distribution of the original attention map been changed significantly after using Log-PE?\n\nWe designed Log-PE to mimic the ALiBi bias pattern that largest values on the diagonal (i.e., close positions), gradually decreasing toward the edges (i.e., distant positions).\nAs such, it preserves the relative structure of the attention map, and empirical results across multiple tasks (see Tables 1–4) show no degradation in performance or convergence, further confirming the compatibility of Log-PE with the spiking attention mechanism.\n\n## 3. Table 1 is hard to understand (Q3)\n\nThanks for your feedback about the potential clarity issue regarding Table 1. We would like to explain Table 1 as follows:\n\n1. Table 1 shows the experimental results of various SNN models in time-series forecasting.\n2. Special Notations: **PE** denotes positional encoding. \"R\" denotes relative PE, while \"A\" denotes absolute PE. \"w/\" denotes \"with\", while \"w/o\" denotes \"without\". $\\uparrow$ denotes that the larger the value, the better the metric.\n3. The $L$ in Metr-la ($L=12$) indicates that the input length of Metr-la dataset is $L$. The number $6, 24, 48, 96$ is the prediction length.\n4. Sin-PE denotes the original sin/cos-based absolute positional encoding [1] in Transformer. RoPE denotes rotary positional encoding [2], which is a classic relative PE. CPG-PE denotes the spiking absolute PE proposed by [3]. Spikformer, SDT-V1, and QKFormer are all variants of spiking Transformers.\n5. Through Table 1, we want to show several conclusions: (1) Results of \"Transformer with RoPE\" and \"Spikformer with RoPE\" show that directly applying RPE methods to spiking Transformers is ineffective. (2) Results of \"Spikformer with Conv-PE\" and Spikformer-XNOR with Conv-PE show that the XNOR modification does not impact the performance of the original SNN models. (3) Obviously, Gray-PE and Log-PE, enable spiking Transformers to achieve the best performance among their variants. (4) For long input sequences (Solar and Electricity), Log-PE is more effective than Gray-PE in capturing relative positional information.\n\nWe will also update the main text to discuss Table 1 more clearly. Thanks again for these valuable comments, and please kindly let us know if you have any further suggestions. \n\n[1] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30.\n\n[2] Su J, Ahmed M, Lu Y, et al. Roformer: Enhanced transformer with rotary position embedding[J]. Neurocomputing, 2024, 568: 127063.\n\n[3] Lv C, Han D, Wang Y, et al. Advancing spiking neural networks for sequential modeling with central pattern generators[J]. Advances in Neural Information Processing Systems, 2024, 37: 26915-26940."}, {"review_id": "iorxn3PCcD", "reviewer": "Reviewer_eL4X", "summary": "This manuscript explores relative positional encoding (RPE) in spiking Transformers. The authors first propose replacing dot-product attention with an XNOR-based one that better reflects the distance between binary spike trains. They further propose two novel RPE methods tailored to the spiking domain: Gray-PE, which leverages the gray code to encode relative positions while preserving binary compatibility, and Log-PE, which applies a logarithmic transformation to relative distances. Comprehensive evaluations across multiple tasks confirm the effectiveness and clear performance advantages of the proposed methods.", "questions": "Q1. The motivation for replacing the standard dot-product with an XNOR-based attention is not fully explained. What specific advantages does XNOR offer over dot-product in spiking self-attention?\n\nQ2. What is the rationale for applying a logarithmic transformation to the relative distance in Log-PE, rather than using the raw offset $|i-j|$ directly?\n\nQ3. The paper claims that Log-PE addresses issues associated with ALiBi, but experimental comparisons were limited to RoPE. Could the authors include ALiBi as a baseline to support this claim with experimental evidence?\n\nQ4. Is the logarithmic operation used in Log-PE friendly to hardware implementation?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n\nS1. The manuscript is well-structured and easy to follow. Key concepts are clearly explained.\n\nS2. To my knowledge, this is the first work to investigate RPE in SNNs. The proposed methods appear novel.\n\nS3. Experimental evaluations look extensive, covering multiple tasks (time-series forecasting, text classification, image recognition) and several spiking Transformer backbones.\n\n\n## Weaknesses:\n\nWhile the contributions are promising, I have several concerns regarding the XNOR-based self-attention mechanism and the Log-PE method in the Questions section. Offering a clearer rationale for these design choices would significantly strengthen the paper. I would be inclined to raise my rating if the authors can satisfactorily address these points during the discussion period.", "was_revised": true, "final_justification": "Issues Resolved:\n\n1. The authors have provided a clear and thorough explanation for their architectural choices.\n\n2. The authors have included new experimental comparisons to ALiBi.\n\n3. The authors have clarified the hardware feasibility of the proposed method.\n\nThe authors’ comprehensive rebuttal and new results have fully addressed my concerns. The work is novel, well-structured, and now well-justified. I am therefore increasing my score and recommend acceptance.", "rebuttal": "Thanks for your thoughtful comments and suggestions, which are valuable for enhancing our paper. We appreciate the positive assessment of our work. The following are responses to individual concerns:\n\n## 1. Offering a clearer rationale for the design choices would significantly strengthen the paper. What specific advantages does XNOR offer over dot-product in spiking self-attention? (W1 & Q1)\n\nWe thank the reviewer for pointing out the need to clarify the rationale behind our architectural choices. Our overall design philosophy is to develop a **coherent and spike-compatible framework** that jointly considers encoding and attention mechanisms in the binary domain. Below, we outline the interconnected motivations for each design choice:\n\n* **Gray-PE**: We adopt Gray code as a binary-friendly positional encoding scheme due to its **uniform Hamming distance behavior** (Theorem 1). Specifically, positions differing by powers of two exhibit stable Hamming distances between their Gray code representations, making it well-suited to approximate relative distances in a spike-efficient manner.\n* **Log-PE**: Inspired by the ALiBi-style design, we initially introduced a full relative position matrix into attention. However, this naïve integration led to severe numerical imbalance in the spike domain, as shown in Table S1 (Appendix B). We therefore apply a **logarithmic transformation** to compress the dynamic range while still preserving coarse-grained distance patterns—effectively balancing biological plausibility and attention stability.\n* **XNOR-Based Attention**: In standard self-attention, similarity between Q and K is computed using the dot-product. However, in spiking Transformers, Q and K are binary matrices (0/1), and dot-product loses its semantic interpretability—particularly in the context of relative positions. Instead, we leverage Hamming distance, which provides a natural dissimilarity measure between two binary vectors. Since $$ d_H(Q, K) = \\sum (Q \\oplus K), \\quad \\text{then} \\quad \\text{similarity}(Q, K) = \\sum \\neg(Q \\oplus K),$$ the XNOR operation becomes a natural surrogate for dot-product in the binary spike domain. Crucially, the effectiveness of Gray-PE depends on using XNOR/XOR as the similarity operator—Theorem 1 only holds under this operation. Moreover, XNOR produces a larger similarity scale compared to dot-product, which helps counteract the numerical dominance of relative position encodings. We also note that XNOR-based attention is gaining traction in recent spiking Transformer architectures \\[1, 2], further validating our design direction.\n\nOverall, these components are not arbitrarily chosen, but rather co-designed to maintain consistency within the binary spike computation paradigm.\n\n[1] Xiao Y, Wang S, Zhang D, et al. Rethinking Spiking Self-Attention Mechanism: Implementing a-XNOR Similarity Calculation in Spiking Transformers[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 5444-5454.\n\n[2] Zou S, Li Q, Ji W, et al. SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and O(T) Complexity[C]//Forty-second International Conference on Machine Learning, 2025\n\n## 2. What is the rationale for applying a logarithmic transformation to the relative distance in Log-PE, rather than using the raw offset directly? (Q2)\n\nAs mentioned in our response to W1 and Line 201, Section 4.4, we initially experimented with direct relative offset terms (e.g., $R_{i,j} = \\frac{L-1}{|i-j|+1}$) inspired by ALiBi. However, such absolute values overpowered the attention map, leading to collapsed spike activations and degraded performance (see Table S1 in Appendix B).\nWe chose a logarithmic transformation to compress the dynamic range of relative positions, enabling the model to:\n* Encode near positions with higher resolution;\n* Avoid numeric overflow in the spike-based attention;\n* Preserve spike-compatibility through integer-based additive bias.\n\nTo intuitively illustrate this difference, we provide two examples: the matrix resulting from the XNOR operation on the original $Q$ and $K$ exhibits overall values around 230 (without scaling). Given the sequence length of 168, the maximum value of $R_{i,j} = \\frac{L-1}{|i-j|+1}$ can reach as high as 167, which significantly disrupts the original attention mechanism. In contrast, the maximum value of $R_{i,j}$ in our Log-PE formulation is only 8, thus minimally interfering with the original attention. This compromise ensures the stability of Log-PE while still preserving effective relative position modeling.\n## 3. The paper claims that Log-PE addresses issues associated with ALiBi, but experimental comparisons were limited to RoPE. Could the authors include ALiBi as a baseline to support this claim with experimental evidence? (Q3)\n\nThanks for your suggestion! We follow your advice to conduct the following results:\n\n|Model|Metric|Metr-la||||Pems-bay||||Solar||||Electricity||||Avg.|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|||6|24|48|96|6|24|48|96|6|24|48|96|6|24|48|96||\n|Transformer w/ ALiBi|$R^2$|0.725|0.558|0.409|0.293|0.782|0.727|0.690|0.677|0.924|0.845|0.741|0.665|0.984|0.980|0.976|0.968|0.747\n||$RSE$|0.556|0.700|0.814|0.885|0.507|0.569|0.606|0.615|0.281|0.393|0.527|0.602|0.250|0.271|0.339|0.422|0.521\n|Spikformer w/ ALiBi|$R^2$|0.665|0.483|0.380|0.104*|0.760|0.644|0.348|0.064*|0.080*|0.080*|0.080*|0.080*|0.710*|0.710*|0.710*|0.710*|-\n||$RSE$|0.622|0.768|0.833|1.02*|0.529|0.709|0.870|1.04*|1.01*|1.01*|1.01*|1.01*|1.03*|1.03*|1.03*|1.03*|-\n\n\nNumbers marked with $^*$ indicate non-convergent results. Transformers equipped with ALiBi exhibit performance comparable to those using Sin-PE and RoPE. However, directly injecting ALiBi-style bias into spiking self-attention disrupts the attention map, often resulting in training instability and convergence failure—particularly on datasets with long input sequences, such as Solar and Electricity ($L=168$).\n\nThese results will be included in the updated version of the paper and appendix.\n\n## 4. Is the logarithmic operation used in Log-PE friendly to hardware implementation? (Q4)\n\nFirst, we would like to clarify that it is not necessary to perform logarithmic operations directly on hardware during inference. Specifically, the relative position bias is defined as $\\mathbf{R}\\_{i,j} = \\begin{bmatrix} \\lceil \\log\\_{2} \\left( \\frac{L-1}{|i - j| + 1} \\right) \\rceil \\end{bmatrix}$, where $\\lceil \\cdot \\rceil$ denotes the ceiling (round-up) function, and $L$ is the maximum sequence length. Since this bias depends only on the relative positions and the predefined sequence length, the entire bias matrix can be computed offline and stored ahead of time, eliminating the need for any runtime computation.\n\nSecondly, even if one wishes to compute the logarithmic transformation on hardware, this can be efficiently achieved using a **lookup table (LUT)** implementation. \n* Given an unsigned integer input of $N$ bits, we partition the input range into $K$ intervals.\n* Each interval is approximated using a piecewise linear function $y = ax + b$, with the parameters $(a, b)$ stored in the LUT.\n* The total LUT storage cost is: $K \\cdot (N + 2P) \\ \\text{bits} \\approx \\frac{K \\cdot (N + 2P)}{8} \\ \\text{bytes},$ where $P$ is the bit width of the parameters.\n\nThis strategy is similar to existing SNN approximations for exponential/leaky functions and has been successfully deployed in many types of neuromorphic chips, such as Intel Loihi [1]. Hence, the hardware implementation of Log-PE is efficient, low-cost, and practically feasible.\n\n\n[1] Davies M, Wild A, Orchard G, et al. Advancing neuromorphic computing with loihi: A survey of results and outlook[J]. Proceedings of the IEEE, 2021, 109(5): 911-934.\n\nThanks again for these valuable comments from your detailed review! We will be more than happy to address any further feedbacks."}, {"review_id": "5wHPI4QH7u", "reviewer": "Reviewer_TvqC", "summary": "This paper addresses the challenge of encoding relative positional information in SNN Transformers while preserving the binary nature of spike signals. It proposes using Gray code to encode relative position distances and theoretically proves that this encoding method maintains a stable Hamming distance. Additionally, a logarithmic-based relative positional encoding method is introduced. The proposed approaches are evaluated across various tasks, achieving significant performance improvements.", "questions": "1. Regarding the Hamming distance calculation formula between two integers provided in Theorem 1 of Section 4.3, I don’t quite understand why the distance between $d_H(0,1)$ and $d_H(1,2)$ is 1. If they are converted into binary spike sequences, shouldn’t the Hamming distance of $d_H(1,2)$ be 2?\n\n2. Is it possible that the addition of Log-PE to the attention map and Gray-PE to the QK terms could compromise the low-power characteristic of SNNs, which relies on purely additive operations?", "limitations": "See weaknesses and questions.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The research problem is highly valuable. Relative positional encoding has become a common technique in ANNs and is known to significantly improve performance. However, due to the binary nature of spike outputs in SNNs, it is challenging to directly transfer the methods used in ANNs for relative positional encoding. Therefore, exploring how to incorporate relative positional encoding into binary SNNs is a meaningful and important research direction.\n\n2. The proposed method is quite interesting. By using Gray code encoding, it enables the representation of relative positional information within binary spike sequences. Additionally, a novel self-attention operator is designed to address the challenge of incorporating relative positional encoding into SNNs.\n\n3. After incorporating relative positional encoding, the model achieves good performance across various types of tasks.\n\nWeaknesses:\n\n1. The paper also mentions that previous works in SNNs have already applied absolute positional encoding. However, why is it challenging to incorporate relative positional encoding? Although such works are indeed scarce at present, it would be helpful if the authors could elaborate more clearly on the specific difficulties involved.\n\n2. The paper lacks sufficient analysis regarding the computational speed of the newly proposed XNOR-Based SSA operator compared to the traditional SSA operator. Traditional SSA can be efficiently implemented using matrix multiplication, enabling parallel computation with high efficiency. In contrast, the newly proposed operator relies primarily on element-wise multiplication between tensors, which raises concerns about its computational efficiency.", "was_revised": false, "final_justification": "", "rebuttal": "We are cheerful that our contribution is well recognized. And thanks for your valuable suggestions that truly enhance the quality of our paper and make it more understandable for the wider community.\n\nResponses to your concerns are presented as follows:\n\n## 1. Why is it challenging to incorporate relative positional encoding? (W1)\n\nWe thank the reviewer for raising this important question. While absolute positional encoding (APE) has been explored in previous Spiking Transformer designs, incorporating relative positional encoding (RPE) remains a significant challenge—not only technically, but also fundamentally, given the nascency of this research area.\n\nFirst, Spiking Transformers are a newly emerging architecture, with most architectural formulations only appearing in the last one to two years. The exploration of how to adapt techniques from ANNs—especially position encoding schemes—to the spike-based domain is still in its infancy. Identifying RPE mechanisms that are compatible with spike-based representations, preserve temporal dynamics, and remain hardware-friendly requires extensive evaluation, filtering, and experimentation. Our work serves not only to provide one such solution, but also to illuminate a viable design space for future studies in this direction.\n\nFrom a technical perspective, RPE is particularly challenging in SNNs due to the following:n Constraint**: Unlike standard Transformers, both Q and K in spiking Transformers are represented as binary spike matrices. This means that many conventional RPE methods—such as RoPE (which relies on sinusoidal transformations in continuous space) and ALiBi (which introduces real-valued additive bias terms)—cannot be directly applied without violating the discrete, binary nature of spike-based computation.\n* **Sparsity and Temporal Dynamics**: Spiking models are temporally sparse and compute in an event-driven manner. Thus, injecting continuous-valued RPE biases into spike-based attention maps requires additional processing (e.g., floating-point operations), which undermines the neuromorphic compatibility and energy efficiency of SNNs.\n* **Hardware Considerations**: Neuromorphic chips (e.g., Intel Loihi, IBM TrueNorth, and the Speck chip) are optimized for bitwise logical operations and typically lack hardware support for continuous arithmetic. Therefore, any RPE design must preserve the bitwise, energy-efficient computation pattern, which rules out most RPE mechanisms in conventional Transformers.\n\nThese factors collectively explain why designing spike-compatible RPE is non-trivial and why existing literature has so far focused primarily on absolute encodings. We will update the manuscript to further clarify our motivations. Thanks for the comment!\n\n## 2. The paper lacks sufficient analysis regarding the computational speed of the newly proposed XNOR-Based SSA operator compared to the traditional SSA operator. (W2)\n\nThis is a good point! Although traditional SSA benefits from highly optimized **matrix multiplication (GEMM)** on GPUs, we would like to clarify that our **XNOR-based SSA** retains computational efficiency for the following reasons:\n* **Bitwise Operations**: The core of XNOR-based SSA relies on XNOR and bit-count operations, which are natively supported by digital hardware and neuromorphic processors. These are much cheaper than floating-point multiplications and additions in terms of energy and hardware complexity.\n* **Hardware Compatibility**: Many neuromorphic accelerators (e.g., Loihi, TrueNorth) natively support spike-based bitwise logic, making our XNOR mechanism better aligned with the target deployment platform than conventional floating-point matrix products.\n* **Parallelizability**: While matrix multiplication benefits from BLAS acceleration, XNOR and summation over dimensions are also highly parallelizable, and can be efficiently implemented using tensor intrinsics (e.g., `bitwise_xnor`, `popcount`, `reduce_sum`).\n* **Empirical Evidence**: We evaluate computational speed and GPU memory of SNNs on Time-series forecasting (Electricity, horizon=24, equal batch size and input length) as follows:\n\n|Model|Time Consumption (s/epoch)|GPU Memory Usage (MB)|\n|-|-|-|\n|Spikformer (Original)|137.48|10572.56|\n|Spikformer-XNOR|139.66|10608.32|\n|Spikformer-XNOR w/ Gray-PE|140.56|10963.88|\n\n## 3. Regarding the Hamming distance calculation formula between two integers provided in Theorem 1 of Section 4.3, I don’t quite understand why the distance between $d_H(0,1)$ and $d_H(1,2)$ is 1. If they are converted into binary spike sequences, shouldn’t the Hamming distance of $d_H(1,2)$ be 2? (Q1)\n\nThank you for pointing this out. The confusion stems from a missing clarification in our notation. The statement in the main paper should be:\n> “As illustrated in Figure 2(b), the Hamming distance $d_H(G(1), G(2)) = 1$...”\n\nThat is, we are referring to the Hamming distance between the Gray code representations of position 1 and 2, not their raw binary or integer distance. The key property of Gray code is that values with a difference of 1 have a single-bit difference, i.e., Hamming distance = 1, which is leveraged to encode relative positions in a hardware-friendly way.\n\nWe will correct this notation and clarify the explanation in the final version to avoid ambiguity.\n\n## 4. Is it possible that the addition of Log-PE to the attention map and Gray-PE to the QK terms could compromise the low-power characteristic of SNNs, which relies on purely additive operations? (Q2)\n\nWe designed both Gray-PE and Log-PE with energy-efficiency in mind, and do not compromise the low-power characteristics of spiking neural networks:\n* **Gray-PE** simply concatenates precomputed binary Gray code vectors to Q and K. The resulting attention map is still computed via XNOR and bit-count operations, which are lightweight and consistent with spike-based processing.\n* **Log-PE** introduces a precomputed integer bias map added to the integer-valued attention map. The addition is carried out once and involves no floating-point operations, which maintains low precision and energy efficiency.\n\nOverall, our methods respect the core design principles of SNNs: binary spike operations, minimal use of arithmetic, and hardware alignment, ensuring that the proposed RPE methods remain highly suitable for low-power neuromorphic applications.\n\nWe appreciate again your deliberated review and the valuable feedbacks! In case there is any remaining concerns, we will be more than happy to make further discussion."}, {"review_id": "ZXjWdM2gcZ", "reviewer": "Reviewer_uN9o", "summary": "This paper addresses the challenge of incorporating relative positional encoding (RPE) into spiking neural networks (SNNs), specifically spiking Transformers. The authors identify that existing RPE methods like RoPE and ALiBi, designed for continuous-valued networks, perform poorly when directly applied to binary spike-based systems. They propose two novel approaches: Gray-PE, which uses Gray Code properties to encode relative distances with consistent Hamming distances for power-of-2 intervals, and Log-PE, which adds logarithmic positional bias directly to attention maps. The paper also introduces XNOR-based spiking self-attention to better align with binary spike representations. Comprehensive experiments across time-series forecasting, text classification, and image classification demonstrate consistent performance improvements over existing spiking positional encoding methods.", "questions": "Is directly concatenating Grey codes into Q and K for Hamming distance computation equivalent to first computing the Hamming distance between Q and K separately, then adding a fixed distance matrix? If equivalent, then concatenating Grey codes into Q and K becomes redundant, because replacing the matrix multiplication between Q and K with Hamming distance computation already compromises the linear attention property of SSA. Concatenating Grey codes into Q and K would only introduce redundant computations (calculating Hamming distances between Grey codes), making it preferable to precompute the fixed distance matrix and perform addition instead. Furthermore, this approach is suboptimal because Grey codes can only distinguish whether two positions differ by 1, resulting in poor discriminative power for relative positions. This limitation also explains why Grey-PE performance is inferior to Log-PE performance in most cases.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. Addresses a genuine research gap: The problem of RPE in spiking networks is underexplored but important for advancing neuromorphic computing capabilities.\n2. Hardware compatibility: Methods preserve binary spike nature, making them suitable for neuromorphic hardware implementation.\n\n**Weaknesses**\n1. Limited technical novelty: The core techniques (Gray Code, logarithmic encoding) are well-established. The main contribution is adaptation rather than fundamental innovation.\n2. This method undermines the linear attention property inherent in SSA: SSA constitutes a linear attention mechanism due to its computational order of $Q(K^TV)$, yielding an approximate time complexity of O(N) when N>>d, where N denotes the token count and d represents the token dimensionality. However, the present study substitutes the matrix multiplication between Q and K with Hamming distance calculations, thereby violating SSA's linear attention characteristics. Future designs of large-scale spiking neural network models will inevitably require further exploitation of SNNs' low computational complexity properties, making the retention of linear attention mechanisms imperative—a requirement that conflicts with the methodology presented in this paper.\n3. Experimental limitations: The dataset selection in this study deviates from mainstream choices, as widely-used benchmark datasets such as GLUE in the NLP domain and ImageNet in the computer vision field were not adopted in this paper.", "was_revised": true, "final_justification": "Thanks to the authors for the response. I have no further questions.", "rebuttal": "Thanks for raising comments and suggestions regarding many important issues of our work. We address your concerns with the following responses.\n\n## 1. Limited technical novelty: The core techniques (Gray Code, logarithmic encoding) are well-established. The main contribution is adaptation rather than fundamental innovation. (W1)\n\nWhile it is true that both Gray Code and logarithmic encoding are long-standing concepts, our contribution is not a direct reuse, but a principled rethinking and repurposing of these techniques under the fundamentally different constraints posed by SNNs.\n\nOur work tackles a largely unexplored problem: how to design relative positional encoding (RPE) that is compatible with the binary, event-driven nature of SNNs and implementable on neuromorphic hardware. To the best of our knowledge, this is the first systematic attempt to bring RPE into Spiking Transformers in a theoretically grounded and hardware-compliant manner.\n\nOur contributions lie in both theoretical and architectural innovation:\n1. Theoretical innovation: We prove (Theorem 1) that Gray Code ensures a consistent Hamming distance between relative positions differing by $2^n$, which is critical in preserving relative distance information in spiking binary attention.\n2. Methodological innovation: We design two spike-compatible, non-floating-point RPE schemes that are implementable under neuromorphic principles and show superior empirical performance across multiple SNN backbones and tasks. Prior RPEs like RoPE/ALiBi are fundamentally incompatible with binary spiking attention due to their reliance on continuous or additive positional biasing.\n\nMore importantly, our work serves a pioneering role in a very nascent area: Spiking Transformers are still in their infancy, with most work in the past year focusing on building basic viability. There remains a clear gap in adapting core mechanisms from ANN Transformers—like positional encoding—to this new paradigm. The fact that seemingly \"established\" concepts need to be carefully reengineered for this setting speaks to the novelty and significance of our contributions.\n\n## 2. This method undermines the linear attention property inherent in SSA. (W2)\n\nWe agree that preserving the low-complexity nature of SNNs is crucial for future large-scale applications. We would like to clarify and respond to this concern:\n1. SSA in Spiking Transformers Is Not Strictly Linear in Practice. While the original SSA mechanism in Spikformer and SDT adopts a formulation that can theoretically be reduced to O(N) under the assumption N ≫ d, this condition is not met in most practical SNN applications, especially in the datasets and architectures we evaluate. In time-series forecasting, the sequence length N is typically 12 or 168, much smaller than the embedding dimension d = 256. In text classification, even for long sequences such as AGNews (N=1024), the dimensionality d = 768 is comparable, and the assumption N ≫ d does not hold. In vision tasks, where Spikformer operates on image patches, N is often small (e.g., 49 for 7×7 patch grids), again violating the N ≫ d condition.\nFurthermore, we note that even in the original Spikformer implementation, the authors compute the attention using the standard form $QK^TV$ rather than the factored $K^TV$ variant, acknowledging this practical constraint.\n3. XNOR-based SSA Preserves Efficiency and Hardware-Friendliness. Our proposed XNOR-based SSA is computationally lightweight and hardware-efficient. Bitwise XNOR and summation operations over binary matrices are inherently more efficient than floating-point matrix multiplications on neuromorphic hardware or digital circuits.\nImportantly, our modification does not increase the asymptotic time complexity and can be efficiently parallelized at the bit level. In this sense, while the attention mechanism deviates from the \"linear attention\" formulation, it remains computationally tractable and well-suited to energy-efficient SNN deployment.\n\nIn summary, we acknowledge the value of linear attention in scaling models to longer sequences, but in the current spiking architectures and tasks, the practical implications are minor. Our proposed mechanism maintains the core benefits of spiking computation—namely sparsity, bit-level efficiency, and compatibility with neuromorphic hardware—while significantly improving sequence modeling through principled relative positional encoding.\n\n## 3. Experimental limitations: The dataset selection in this study deviates from mainstream choices, as widely used benchmark datasets such as GLUE in the NLP domain and ImageNet in the computer vision field were not adopted in this paper. (W3)\n\nWe appreciate the suggestion regarding broader dataset usage.\nHowever, to fairly validate the cross-domain generalizability of our RPE methods in SNNs, we selected datasets following previous work on sequential tasks in SNNs. Specifically,\n1. We follow [1][2][3] to conduct time-series forecasting experiments (e.g., Metr-la, Pems-bay, Solar).\n2. We follow [2][4] to conduct text classification, including six diverse datasets (English/Chinese, short/long), offering rich linguistic diversity;\n3. We follow [2][5] to conduct image classification, including not only standard datasets (CIFAR, Tiny-ImageNet) but also a neuromorphic dataset (CIFAR10-DVS), which is specifically relevant to the SNN community.\n\nWhile GLUE and ImageNet are standard in ANN research, they are computationally intensive and less suitable for initial SNN benchmarking due to current hardware and convergence limitations. We highlight that our methodology is orthogonal to specific datasets and can be easily applied to GLUE/ImageNet in future work.\n\n[1] Lv C, Wang Y, Han D, et al. Efficient and Effective Time-Series Forecasting with Spiking Neural Networks[C]//International Conference on Machine Learning. PMLR, 2024: 33624-33637.\n\n[2] Lv C, Han D, Wang Y, et al. Advancing spiking neural networks for sequential modeling with central pattern generators[J]. Advances in Neural Information Processing Systems, 2024, 37: 26915-26940.\n\n[3] Fang Y, Ren K, Shan C, et al. Learning decomposed spatial relations for multi-variate time-series modeling[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(6): 7530-7538.\n\n[4] Lv C, Xu J, Zheng X. Spiking Convolutional Neural Networks for Text Classification[C]//The Eleventh International Conference on Learning Representations, 2023\n\n[5] Li Y, Geller T, Kim Y, et al. Seenn: Towards temporal spiking early exit neural networks[J]. Advances in Neural Information Processing Systems, 2023, 36: 63327-63342.\n\n## 4. Is directly concatenating Gray codes into Q and K for Hamming distance computation equivalent to first computing the Hamming distance between Q and K separately, then adding a fixed distance matrix? (Q1)\n\nThe short answer is: Yes, the two are equivalent under the assumption that both Q and K are binary vectors, and the Gray codes are deterministically assigned based on position.\n\nMathematically:\n1. Concatenation form: $\\text{AttnMap}\\_{i,j} = \\sum\\_d \\neg([Q\\_i | G(i)] \\oplus [K\\_j | G(j)])$\n2. Additive form: $\\text{AttnMap}\\_{i,j} = \\sum\\_d \\neg(Q\\_i \\oplus K\\_j) + R\\_{i,j}$\n\nIn the binary domain, these two are mathematically equivalent since $\\text{XNOR}(A | B, C | D) = \\text{XNOR}(A, C) + \\text{XNOR}(B, D)$ in terms of Hamming matching. Under this setting, $R_{i,j} = \\sum_d \\neg(G(i) \\oplus G(j))$.\n\nWe chose the concatenation-based formulation for the following reasons:\n1. Clarity of Presentation: Our primary motivation is to make the functionality of Gray codes as a form of relative positional encoding more transparent to readers. By embedding $G(i)$ and $G(j)$ directly into Q and K, it becomes more intuitive to see how the Hamming distance between Gray codes captures relative position. Nevertheless, we acknowledge that this could be misinterpreted as being fundamentally different from additive RPE, and we will add a clarification and mathematical equivalence discussion in the Method section of the revised version.\n2. Memory Efficiency: Concatenating $G(i)$ and $G(j)$ directly avoids the need to explicitly store the full pairwise distance matrix $R$, whose size grows quadratically with the sequence length. This is particularly important for neuromorphic hardware, where memory is limited and efficient use of bitwise operations is critical.\n3. No Additional Computation Overhead: Since XNOR is a bitwise operation, our approach introduces no extra computational cost. The Gray code bits are simply treated as additional spike dimensions during the existing Q-K matching. In contrast, adding a precomputed $R_{i,j}$ would require separate accumulation logic and increase implementation complexity in a hardware-constrained setting.\n\n## 5. This approach is suboptimal because Gray codes can only distinguish whether two positions differ by 1. (Q2)\nWe believe this is a misunderstanding. Gray codes can distinguish whether two positions differ by $2^n$, and we offer a detailed proof in Appendix A.\n\nWhile it is true that adjacent Gray Codes differ by 1 bit, our method does not rely solely on this property. Our core idea is that relative distances differing by $2^n$ (See Theorem 1 and Appendix A) yield consistent and distinct Hamming distances, which enables structured representation of relative distances.\n\nMoreover, in short-to medium-length sequences, the discriminative capacity of Gray-PE is sufficient, as shown in our experiments on Metr-la, SST-2, CIFAR, etc. For long sequences, we explicitly recognize this limitation and propose Log-PE as a complementary method, which handles large positional spans via a decaying distance-sensitive bias.\n\nWe hope the responses above can fully address your concerns. Please kindly let us know if there are any remaining concerns. We will be glad to have further clarifications. Thanks again for your time and effort on the review process!"}]

Author rebuttal:

Meta-review: This paper introduces Gray-PE and Log-PE, two novel methods for incorporating relative positional encoding into spiking Transformers. The proposed approaches enhance performance across diverse tasks and architectures while preserving the important binary spike-based nature.

Reviewers broadly agree that the paper is well-motivated, clearly written, and supported by solid theoretical and empirical evidence. During the rebuttal, key concerns were raised regarding computational overhead, hardware compatibility, motivation for design choices, and the lack of large-scale benchmarks. The authors addressed these issues thoroughly, providing clear clarifications. After rebuttal, all reviewers agree that most of their concerns has been resolved.

Given all the strengths of this work, and the successful rebuttal of reviewers’ concerns, I recommend acceptance for this paper.
Decision: Accept (spotlight)