Title: Toward Relative Positional Encoding in Spiking Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Spiking Neural Networks

Relative Positional Encoding

Spiking Transformers
Abstract: Spiking neural networks (SNNs) are bio-inspired networks that mimic how neurons in the brain communicate through discrete spikes, which have great potential in various tasks due to their energy efficiency and temporal processing capabilities.
SNNs with self-attention mechanisms (spiking Transformers) have recently shown great advancements in various tasks, and inspired by traditional Transformers, several studies have demonstrated that spiking absolute positional encoding can help capture sequential relationships for input data, enhancing the capabilities of spiking Transformers for tasks such as sequential modeling and image classification. However, how to incorporate relative positional information into SNNs remains a challenge.
In this paper, we introduce several strategies to approximate relative positional encoding (RPE) in spiking Transformers while preserving the binary nature of spikes.
Firstly, we formally prove that encoding relative distances with Gray Code ensures that the binary representations of positional indices maintain a constant Hamming distance whenever their decimal values differ by a power of two, and we propose **Gray-PE** based on this property.
In addition, we propose another RPE method called **Log-PE**, which combines the logarithmic form of the relative distance matrix directly into the spiking attention map.
Furthermore, we extend our RPE methods to a two-dimensional form, making them suitable for processing image patches.
We evaluate our RPE methods on various tasks, including time series forecasting, text classification, and patch-based image classification, and the experimental results demonstrate a satisfying performance gain by incorporating our RPE methods across many architectures.
Our results provide fresh perspectives on designing spiking Transformers to advance their sequential modeling capability, thereby expanding their applicability across various domains.
Our code is available at https://github.com/microsoft/SeqSNN.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of incorporating relative positional encoding (RPE) into spiking neural networks (SNNs), specifically spiking Transformers. The authors identify that existing RPE methods like RoPE and ALiBi, designed for continuous-valued networks, perform poorly when directly applied to binary spike-based systems. They propose two novel approaches: Gray-PE, which uses Gray Code properties to encode relative distances with consistent Hamming distances for power-of-2 intervals, and Log-PE, which adds logarithmic positional bias directly to attention maps. The paper also introduces XNOR-based spiking self-attention to better align with binary spike representations. Comprehensive experiments across time-series forecasting, text classification, and image classification demonstrate consistent performance improvements over existing spiking positional encoding methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. Addresses a genuine research gap: The problem of RPE in spiking networks is underexplored but important for advancing neuromorphic computing capabilities.\n2. Hardware compatibility: Methods preserve binary spike nature, making them suitable for neuromorphic hardware implementation.\n\n**Weaknesses**\n1. Limited technical novelty: The core techniques (Gray Code, logarithmic encoding) are well-established. The main contribution is adaptation rather than fundamental innovation.\n2. This method undermines the linear attention property inherent in SSA: SSA constitutes a linear attention mechanism due to its computational order of $Q(K^TV)$, yielding an approximate time complexity of O(N) when N>>d, where N denotes the token count and d represents the token dimensionality. However, the present study substitutes the matrix multiplication between Q and K with Hamming distance calculations, thereby violating SSA's linear attention characteristics. Future designs of large-scale spiking neural network models will inevitably require further exploitation of SNNs' low computational complexity properties, making the retention of linear attention mechanisms imperative—a requirement that conflicts with the methodology presented in this paper.\n3. Experimental limitations: The dataset selection in this study deviates from mainstream choices, as widely-used benchmark datasets such as GLUE in the NLP domain and ImageNet in the computer vision field were not adopted in this paper.\n\nQUESTIONS:\nIs directly concatenating Grey codes into Q and K for Hamming distance computation equivalent to first computing the Hamming distance between Q and K separately, then adding a fixed distance matrix? If equivalent, then concatenating Grey codes into Q and K becomes redundant, because replacing the matrix multiplication between Q and K with Hamming distance computation already compromises the linear attention property of SSA. Concatenating Grey codes into Q and K would only introduce redundant computations (calculating Hamming distances between Grey codes), making it preferable to precompute the fixed distance matrix and perform addition instead. Furthermore, this approach is suboptimal because Grey codes can only distinguish whether two positions differ by 1, resulting in poor discriminative power for relative positions. This limitation also explains why Grey-PE performance is inferior to Log-PE performance in most cases.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of encoding relative positional information in SNN Transformers while preserving the binary nature of spike signals. It proposes using Gray code to encode relative position distances and theoretically proves that this encoding method maintains a stable Hamming distance. Additionally, a logarithmic-based relative positional encoding method is introduced. The proposed approaches are evaluated across various tasks, achieving significant performance improvements.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The research problem is highly valuable. Relative positional encoding has become a common technique in ANNs and is known to significantly improve performance. However, due to the binary nature of spike outputs in SNNs, it is challenging to directly transfer the methods used in ANNs for relative positional encoding. Therefore, exploring how to incorporate relative positional encoding into binary SNNs is a meaningful and important research direction.\n\n2. The proposed method is quite interesting. By using Gray code encoding, it enables the representation of relative positional information within binary spike sequences. Additionally, a novel self-attention operator is designed to address the challenge of incorporating relative positional encoding into SNNs.\n\n3. After incorporating relative positional encoding, the model achieves good performance across various types of tasks.\n\nWeaknesses:\n\n1. The paper also mentions that previous works in SNNs have already applied absolute positional encoding. However, why is it challenging to incorporate relative positional encoding? Although such works are indeed scarce at present, it would be helpful if the authors could elaborate more clearly on the specific difficulties involved.\n\n2. The paper lacks sufficient analysis regarding the computational speed of the newly proposed XNOR-Based SSA operator compared to the traditional SSA operator. Traditional SSA can be efficiently implemented using matrix multiplication, enabling parallel computation with high efficiency. In contrast, the newly proposed operator relies primarily on element-wise multiplication between tensors, which raises concerns about its computational efficiency.\n\nQUESTIONS:\n1. Regarding the Hamming distance calculation formula between two integers provided in Theorem 1 of Section 4.3, I don’t quite understand why the distance between $d_H(0,1)$ and $d_H(1,2)$ is 1. If they are converted into binary spike sequences, shouldn’t the Hamming distance of $d_H(1,2)$ be 2?\n\n2. Is it possible that the addition of Log-PE to the attention map and Gray-PE to the QK terms could compromise the low-power characteristic of SNNs, which relies on purely additive operations?\n\nLIMITATIONS:\nSee weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript explores relative positional encoding (RPE) in spiking Transformers. The authors first propose replacing dot-product attention with an XNOR-based one that better reflects the distance between binary spike trains. They further propose two novel RPE methods tailored to the spiking domain: Gray-PE, which leverages the gray code to encode relative positions while preserving binary compatibility, and Log-PE, which applies a logarithmic transformation to relative distances. Comprehensive evaluations across multiple tasks confirm the effectiveness and clear performance advantages of the proposed methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\nS1. The manuscript is well-structured and easy to follow. Key concepts are clearly explained.\n\nS2. To my knowledge, this is the first work to investigate RPE in SNNs. The proposed methods appear novel.\n\nS3. Experimental evaluations look extensive, covering multiple tasks (time-series forecasting, text classification, image recognition) and several spiking Transformer backbones.\n\n\n## Weaknesses:\n\nWhile the contributions are promising, I have several concerns regarding the XNOR-based self-attention mechanism and the Log-PE method in the Questions section. Offering a clearer rationale for these design choices would significantly strengthen the paper. I would be inclined to raise my rating if the authors can satisfactorily address these points during the discussion period.\n\nQUESTIONS:\nQ1. The motivation for replacing the standard dot-product with an XNOR-based attention is not fully explained. What specific advantages does XNOR offer over dot-product in spiking self-attention?\n\nQ2. What is the rationale for applying a logarithmic transformation to the relative distance in Log-PE, rather than using the raw offset $|i-j|$ directly?\n\nQ3. The paper claims that Log-PE addresses issues associated with ALiBi, but experimental comparisons were limited to RoPE. Could the authors include ALiBi as a baseline to support this claim with experimental evidence?\n\nQ4. Is the logarithmic operation used in Log-PE friendly to hardware implementation?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops two novel methods (Gray-PE and Log-PE) to integrate relative positional encoding into spiking Transformers, overcoming a key challenge. \n1. ​**​Gray-PE:​**​ Encodes positions using Hamming distance that keeps digital patterns similar for close positions.\n2. ​**​Log-PE:​**​ Adds relative distance patterns using logs to the attention mechanism.\nThese methods effectively encode sequence positions while preserving the essential binary nature of spikes.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. Provide theoretical justification for Gray-PE and Log-PE, especially preserving the essential binary nature of spikes. .\n2. Empirically demonstrate consistent performance improvements across diverse tasks (time series, text classification, image classification) and architectures. \n\nWeakness:\nSee questions.\n\nQUESTIONS:\n**Question 1: Computational Overhead of Gray-PE**  \nGiven the formulation of Gray-PE, which expands the channel dimensionality of both Q and K, a quantitative assessment of its computational overhead would be valuable. Specifically:  \n- What is the magnitude of channel dimension expansion introduced by concatenating Gray codes?  \n- Has the authors conducted empirical measurements to quantify the resultant increase in computational complexity (e.g., FLOPs, memory footprint, or latency)?  \n\n**Question 2: Boundary Effects in Log-PE**  \nThe proposed logarithmic positional encoding (Log-PE) employs the formulation:  $ R_{i,j} = \\left\\lceil \\log_2 \\left( \\frac{L-1}{|i-j|+1} \\right) \\right\\rceil $, where the maximum bias scales with $\\lceil \\log_2(L-1) \\rceil$ (e.g., $R_{\\max}=10$ for $L=1024$). Could such values overwhelm the original attention map magnitudes? Most positional encoding schemes introduce subtle biases without dominating semantic features.  Has the distribution of the original attention map been changed significantly after using Log-PE? \n\n**Other questions**  \nTable 1 seems too complex to understand the details. The author should import the accessibility.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}