Quantized Spike-driven Transformer
Authors: Xuerui Qiu
Malu Zhang
Jieyuan Zhang
Wenjie Wei
Honglin Cao
Junsheng Guo
Rui-Jie Zhu
Yimeng Shan
Yang Yang
Haizhou Li
Venue: ICLR 2025 Poster (2025)
Keywords: Spiking Neural Network+Spike-driven+Quantized Spiking Transformer+ Neuromorphic Computing
Primary area: foundation or frontier models, including LLMs
TL;DR: We proposed a quantized spike-driven Transformer that achieves state-of-the-art results on various vision tasks and maintains a tiny model size.

Abstract: Spiking neural networks (SNNs) are emerging as a promising energy-efficient alternative to traditional artificial neural networks (ANNs) due to their spike-driven paradigm.
However, recent research in the SNN domain has mainly focused on enhancing accuracy by designing large-scale Transformer structures, which typically rely on substantial computational resources, limiting their deployment on resource-constrained devices.
To overcome this challenge, we propose a quantized spike-driven Transformer baseline (QSD-Transformer), which achieves reduced resource demands by utilizing a low bit-width parameter. 
Regrettably, the QSD-Transformer often suffers from severe performance degradation.
In this paper, we first conduct empirical analysis and find that the bimodal distribution of quantized spike-driven self-attention (Q-SDSA) leads to spike information distortion (SID) during quantization, causing significant performance degradation. To mitigate this issue, we take inspiration from mutual information entropy and propose a bi-level optimization strategy to rectify the information distribution in Q-SDSA.
Specifically, at the lower level, we introduce an information-enhanced LIF to rectify the information distribution in Q-SDSA.
At the upper level, we propose a fine-grained distillation scheme for the QSD-Transformer to align the distribution in Q-SDSA with that in the counterpart ANN.
By integrating the bi-level optimization strategy, the QSD-Transformer can attain enhanced energy efficiency without sacrificing its high-performance advantage.
We validate the QSD-Transformer on various visual tasks, and experimental results indicate that our method achieves state-of-the-art results in the SNN domain.
For instance, when compared to the prior SNN benchmark on ImageNet, the QSD-Transformer achieves 80.3\% top-1 accuracy, accompanied by significant reductions of 6.0$\times$ and 8.1$\times$ in power consumption and model size, respectively. Code is available at https://github.com/bollossom/QSD-Transformer.

Published as a conference paper at ICLR 2025
QUANTIZED SPIKE-DRIVEN TRANSFORMER
Xuerui Qiu 1,233 Malu Zhang's Jieyuan Zhang'", ',, Wenjie Wei', Honglin Cao', Junsheng Guo Rui-Jie Zhu Yimeng Shan", Yang Yang', Haizhou Li?
University of Electronic Science and Technology of China, Institute of Automation, Chinese Academy of Sciences,
School of Future Technology, University of Chinese Academy of Sciences, "China Agricultural University, University of California, Santa Cruz, Liaoning Technical University, "Chinese University of Hong Kong (Shenzhen)
ABSTRACT
Spiking neural networks (SNNs) are emerging as a promising energy effrcyeefficeett alternative to traditional artificial neural networks (ANNs) due their spike-driven paradigm. However, recent research in the SNN domain has mainly focused on enhancing accuracy by designing large-scale Transformer structures, which typically rely on substantial computational resources, limiting their deployment on resource-constrained devices. To overcome this challenge, we propose a quantized spike-driven Transformer baseline resource demands by utilizing a low bit-width parameter. Regrettably, the QSD- Transformer often suffers from severe performance degradation. In this paper, we first conduct empirical analysis and find that the bimodal distribution of quantized spike-driven self-attention (Q-SDSA) leads spike information distortion (SID) during quantization, causing significant performance degradation. To mitigate this issue, we take inspiration from mutual information entropy and propose a bi-level optimization strategy to rectify the information distribution in Q-SDSA. Specifically, at the lower level, we introduce an ceermaation-nnhaneee LIF to rectify the information distribution in Q-SDSA. At the upper level, we propose fine- grained distillation scheme for the QSD-Transformer to align the distribution Q-SDSA with that in the counterpart ANN. By integrating the bi-level optimization strategy, the QSD-Transformer can attain enhanced energy efficiency without sacrificing its high-performance advantage. We validate the QSD- SSD-Trassformr on various visual tasks, and experimental results indicate that our method achieves state-of-the-art results in the SNN domain. For instance, when compared the prior SNN benchmark on ImageNet, the QSD-Trranffomee achieves 80.3% top- accuracy, accompanied by signifieant reductions of 6.0x and 8.1 in power consumption and model size, respectively. Code is available at Quantized Spike- driven Transformer.
which achieves reduced
INTRODUCTION
Spiking neural networks (SNNs) have emerged as a promising approach for realizing energy-efficien computational intelligence due to their high biological plausibility (Maass, 1997), sparse spike-driven communication (Roy et al., 2019), and low power consumption on neuromorphic hardware (Davies et al., 2018; Pei et al., 2019; Merolla et al., 2014). Within SNNs, the spiking neuron transmits information via sparse binary spikes, where the binary value of 0 denotes neural quiescence and the value of signifies a spiking event (Shrestha & Orchard, 2018 18; Eshraghian et al., 2023). The unique spike-driven nature is key to achieving low power consumption, where only a subset of spiking neurons activated perform sparse synaptic accumulation (AC) (Yao al., 2023a;b). However, despite their high energy efficiency, the application of SNNs is constrained by their low task accuracy.
"qqual Contribution Corresponding author, saluzhhag@uuestc.dddee

---

Published as a conference paper at ICLR 2025
(a) Quantized Spike-driven Self-attention.
(b) Information Enhanced LIF.
1 +
Multi-bit Training
binary spike input
01101010
0.25
Spikes from pre-layer
Canov
Q-Conn2
Q-Conv3
1 100100
Kmem
Vmem
Spike- Inference
(c) Fine-grained Distillation.
Rectify
Rectify
Rectify
JSD-Transformer
ANN-based Transformer
Baseline
Amem
&mmem
mem
(Student)
(Teacher)
Q-Conv layer
Conv layer
IE-LIF
IE-LIF
IE-LIF
9,1
Q-SDSA
MHSA
kg
KA
Q-FC
VA
FC
Matrix Multiplication
Add
X
L
Information Enhanced
classification head
MHSA
Multi-head Self-attention
classification head
Q SDSA
Quantized Spike-driven Self-attention
Figure 1: Overview of the Transfooffrrrrr. (a) Proposed quantized spike-driven self-attention (Q- SDSA) module, where the membrane potential is rectified and then sent to the cermatiin-enhancee LIF (IF-LIF) neuron. (b) Proposed IE-LIF spiking neuron model, which utilizes the multi-bit spike during training while the binary spike during inference. (c) Proposed fine-grained distillation scheme.
Numerous researchers have made great efforts to improve the performance and expand the application scenarios of SNNs. Building upon the success of Vision Transformers (ViT) (Dosovitskly al., 2020; Touvron et al., 2021; Yu et al., 2023), researchers naturally combined SNNs with Transformers, resulting in significant performance improvements on ImageNet benchmark (Zhou al., 2023b;a; 2024b;a) and diverse application scenarios (Zhang et al., 2022b;c; Lv et al., 2023). Despite their commendable performance, these studies come at the cost of massive model parameters and high computational complexity. For instance, Spikformer v2 (Zhou et al., 2024c) and Spike-driven Transformer v3 Yao et al. (2024) achieve accuracies of 82.4% and 86.2% on the ImageNet dataset, respectively. These models have 173M parameters, necessitating 1384MB memory, and requiring 28.4G synapse-operatione per second for inference. This places significant demands on the storage and computational capabilities of neuromorphic chips, thereby limiting their deployment on edge devices. Therefore, there is an urgent need for a low-bit and high-performance Spike-based Transformer.
Numerous efforts have been made to compress and accelerate neural networks on edge computing devices, e.g., pruning (Han et al., 2015; Shen et al., 2023), quantization (Qin et al., 2021; Deng et al., 2023), and knowledge distillation (Hinton al., 2015; Xu et al., 2023). Among these, quantization is particularly suitable for hardware deployment as it can reduce the bit-width of network parameters and activations, enabling efficient inference. The post-training quantization (PTQ) approach (Jacob et al., 2018) calculates quantization parameters directly based on pre-trained full-precision models, which may limit the model's performance to suboptimal level without fine-tuning. In particular, the model obtained from this approach may suffer from dramatic performance drops when quantized to ultra-low bits (c.g., 2, 4 bit). contrast to PTQ, Quantization-ANe Aware Training (QAT) (Krishnamoorthi, 2018) performs quantization during the learning process and generally achieves great performance with high compression ratios. However, the field of SNNs, research on QAT methods has primarily focused convolutional neural networks (CNNs), with low-bit Spikformer remaining largely unexplored.
2

---

Published as a conference paper at ICLR 2025
In this paper, we first construct quantized spike-driven Transformer (SSD-Transformer) baseline (Yao et al., 2023a), which directly quantizes 32-bit weights to low bit-width during training. Despite exhibiting significant energy efficiency, this simple method can lead to severe performance degra- dation. Through detailed analysis of the baseline, we reveal that quantizing the attention module will reduce the representation capability of the self-attention maps, which is defined as the spiking information distortion (SID) problem. This is the main reason for the performance degradation. To address this issue, we propose a bi-level optimization strategy for the baseline, aiming at rectifying the distribution in quantized spike-drive drike-driven self-attention maps (Q-SDSA) from both the neuron and network levels. The overview of the QSD-Transfore is shown in Fig. and our main contributions can be summarized as:
We construct a lightweight spike-driven Transformer baseline through quantization, called QSD-Transfors The QSD- Transformer quantizes the synaptic weights from a 32-bit to a low-bit representation (typically 2, 3, and, 4 bits). leading to reduced model size and significant energy efficiency advantages.
We reveal that the proposed baseline suffers from performance degradation due to the SID) problem in Q-SDSA. Inspired by information entropy, propose a bi-level optimization strategy to solve this issue. This strategy introduces an information-enhanee grained distillation to rectify the distribution of Q-SDSA, leading to enhanced performance.
LIF and a fine-
We validate the QSD- TrD-srasfformer various visual tasks, e.g., classification, object detec- tion, semantic segmentation, and transfer learning. Experimental results indicate that our method outperforms existing spiking Vision Transformers by substantial margin, while also boasting a compact model size and extremely low power consumption.
2 RELATED WORKS
Spiking vision transformer. Spikformer (Zhou al., 2023b) pioncered direct training with pure SNN architecture, introducing a linear self-attention mechanism that eliminates multiplication by activating Query, Key, and Value with spiking neurons and replacing softmax with spiking neurons. Its successor (Zhou et al., 2024c) integrated masked image modeling (He et al., 2022), achieving an 82.25% accuracy on ImageNet with 172 M parameters, the highest among SNNs. SpikingResformer (Shi et al., 2024) introduces a novel spike self-attention mechanism along with a judicious scaling approach, enabling effective extraction of local features. However, none of these models preserved the spike-driven nature until the spike-driven Transformer (Yao et al., 2023b), which introduces the sparse addition to self-attention using only masking operations. Its successor (Yao al., 2023a) focused on the meta-design of the spiking vision Transformer, including architecture, spike-driven self-attention, shorteut connections, etc. The proposed spike-driven Transformer (Yao al., 2023a) set up direct training SNN backbone for improving performance across tasks like image classification, segmentation, and object detection, hinting at impacts on neuromorphic chip design. Hence, this study, we adopt the pure addition spike-driven Transformer v2 for quantization baseline.
Model compression. Numerous compression techniques have been explored to compress large-scale SNNs, including: (1) Pruning (Han et al., 2015; Kusupati et al., 2020; Savarese et al., 2020) in SNNs generally draw on traditional pruning methods from ANNs to suit the spatial and temporal domains (Chen al., 2022; Shi et al., 2023; Shen et al., 2024). While successful on simpler datasets and shallow networks, achieving high performance becomes more challenging with complex datasets and deeper networks. (2) Knowledge distillation (Hinton al., 2015; Guo al., 2023; Touvron et al., 2021) involves the transfer of knowledge from large-scale ANNs or SNNs to smaller-scale SNNs, aiming to compress models and reduce energy consumption. However, these methods (Takuya et al., 2021; Tran et al., 2022; Xu et al., 2023) often distill only the final output of the model, leading to incomplete knowledge transfer and suboptimal performance in SNNs. (3) Quantization (lacob et al., 2018; Krishnamoorthi, 2018), particularly for hardware deployment, is advantageous as it reduces the bit-width of network parameters and activations, enabling efficient inference. Recent research on quantization methods (Stromatias et al., 2015; Deng et al., 2023; Kheradpisheh et al., 2022) for SNNs has predominantly focused on weight binarization within Conv-based architectures. For instance, Deng al. (Deng et al., 2023) utilized QAT (Krishnamoorthi, 2018; Jacob et al., 2018) to reduce the weight size of Conv-based SNN, which demonstrated high compression performance with acceptable accuracy loss recognition tasks. Despite the significant potential of QAT in reducing the
3

---

Published as a conference paper at ICLR 2025
memory and computational costs of Conv-based SNN (Deng et al., 2023), directly applying QAT on Spikformer leads to poor performance. The core challenge is the significant distribution discrepancy between the binary spike patterns and the normal distribution in the self-attention of Spikformer and ANN Transformer, which causes severe information distortion, leading to performance degradation.
3 PRELIMINARY
In this section, we first introduce the spiking neuron model. Then, we construct a quantized spike- driven Transformer (QSD-Transformer) baseline, which quantizes the synaptic weight from 32-bit to low bit-width, thereby demonstrating significant energy efficiency advantages.
Spiking neuron model. In this paper, we choose the widely-employed iterative Leaky Integrate- and-Fire (LIF) model (Wu et al., 2018; Guo et al., 2024), which can be described by the following mathematical equations:
v1( h - 1] + f(w 1 1)),
(1)
s'[1 - [v 0),
(2)
h111 rv s' s'4),
where the time constant, is the time step, w' is the weight matrix of layer f, f() is the operation that stands for convolution (Conv) or fully connected (FC), x is the input, and (-) denotes the Heaviside step function. When the membrane potential v exceeds the firing threshold 1 the LIF neuron will trigger spike s; otherwise, it remains inactive. After spike emission, the neuron invokes the reset mechanism, where the soft reset function employed. the membrane potential following the reset function.
OSD-Transformer baseline. We select the purely spike-driven Transformer v2 (SD- -D-Trassformer v2) (Yao al., 2023a) to perform quantization, and the LSQ (Bhalgat et al., 2020) method is employed to quantize the 32-bit weights to low bit-width (e.g.. 2, 3, 4 bits). The quantization function is defined as:
((w)= clip
2 --
-
-1
w a'w Q(w),
Oww
where w is the 32-bit weight, 6 the bit assigned to the quantized weight (i.e., Q(w)), and ow the scaling factor used to mitigate the quantization error. Moreover, c2, a, b} confines x within range [a, b). and 1-7 denotes the rounding operator. These two operations make the quantization function non-differentiable, so we adopt the straight-througl estimator (STE) (Bengio et al., 2013) to assist the gradient backpropagation. Eq. is performed for all weight layers the baseline model. Building upon this, the calculation for certain layer e {FC, Conv} in our baseline is expressed as:
(x(x) w SN(() aw Q(w) (x(x).
(5)
Here, S.(.)) represents the spiking neuron layer, which converts the floating-point input x into the binary spike. Hence, the QSD-Transformer employs binary spike activities and low bit-width weights for Conv and FC operations. This replaces the original computationally intensive operations, leading significant energy efficiency improvements. Following Eq. 5, the quantization for the spike-driven self-attention (Q-SDSA) can be further described as:
(6) where, 9s (g) and kg, V. are obtained in the same way. can observed from Eq. 6 that our Q-SDSA module reduces the computational number by the linear attention mechanism with D²N where N is the token numbers and D is the channel dimensions. However, quantizing the attention module will diminish its representation capacity, leading to performance degradation. In the next section, we will provide a detailed explanation of this issue and propose methods to address it.
= Q (x), k = 2xom=2(x), v = (x), Q-SDSA(q. = (N,(k,
4 METHOD
In this section, we first reveal that the performance degradation of our baseline is due to the limited representational capacity of Q-SDSA. Inspired by information entropy theory, we propose bi-level
4

---

Published as a conference paper at ICLR 2025
optimization strategy to address this issue. At the lower level, we introduce the ceeemattion-nhhanee leaky integrate-and aeerrate-nn-fiie fire (IE-LIF) neuron, which maximizes information entropy by adjusting the spike distribution. At the upper level, we present the fine- ane-grained distillation scheme, which minimizes conditional entropy by aligning the information of Q-SDSA with that of ANNs.
All Quantized (2). Quantize FC Quantize Attention Quantoa
quary
0.25
0.25
AT-022 4.50)
0.23
AT-0.31.
[1]
0.20
0.20
0.20
70/2
0.10
0.10
0.12
13)
75.7
0.06
0.05
0.65
[0]
79.7
0.0
79.9
quary
key
value
(1)
57.6
1.00
1:00
1.00
SMN
122
700
0.50
0:50
0.53
(3)
0:25
0.25
0.25
(4)
70.9
71.
0.00
0.00
(a)
(b)
Figure 2: (a) Accuracy of quantizing different modules in the SD-Transformer v2 and its same ANN Transformer. (b) The distribution of the attention module (blue shadow), and the probability density function curve of normal distribution (red line). Experiments are conducted on ImageNet, and layers in SD-Trrasformer v2 and its same ANN Transformer are selected for illustration.
4.1 PERFORMANCE DEGRADATION ANALYSIS
Despite its efficiency advantages, the QSD-Transformer baseline suffers from performance degra- dation, as shown in Fig. (a) (All quantized). In contrast, the quantized ANN Transformer can balance both efficiency and performance (Wu al., 2022). Hence, we quantize each module of the architecture, i.c., FC, Conv, and Attention, to identify which one has the biggest impact on performance degradation.
We illustrate the ablation results in Fig. 2 (a), where all weights are quantized to 4-bit. Obviously, the attention layer in SNNs is highly sensitive to quantization, but it is not observed in ANNs. This is attributed to the different information distributions in the self-attention map, as depicted in Fig. 2 (b). It can be observed that the information within Q-SDSA displays a bimodal distribution, whereas the information in ANN adheres to a normal distribution. Due to the utilization of both low-bit weights and binary spikes, the information representation capability of the Q-SDSA is severely limited compared to that of ANNs. We define it as the spike information distortion (SID) problem.
To solve the SID issue, we draw on the quantized Transformer in the ANN domain (Liu et al., 2021) that has struck good balance between efficiency and performance by maintaining the noraml distribution of activity. This prompts us to adjust the information distribution in the Q-SDSA to match that of ANNs. To achieve this, take inspiration from the information entropy theory (Paninski, 2003) and formulate it as the mutual information entropy maximization problem.
Definition 1. Addressing the performance degradation of the to maximizing the mutual information entropy between it and the quantized Transformer in ANNs. The optimization goal for the OSD-Transformer is defined below.
baseline is equivalent
max Ipp =H(p²) H(ps 1
where dd and A are the attention score value in SNN and ANN respectively, and @ is the parameters of the QSD-Qranaaa Transformer. However, directly optimizing Eq. 7 is challenging, so we regard as a bi-level optimization problem (Colson et al., 2007; Sinha et al., 2017). It is achieved by minimizing the conditional information entropy H(ps A and maximizing the information entropy H Hpp)), which is defined as:
min H(p 11)) s.t. S arg max H(pS). 05
(8)
8d
5

---

Published as a conference paper at ICLR 2025
To accomplish it, first propose the iiformatiinnenhannee LIF (IE-LIF) neuron, aiming to maximize the information entropy pd at the lower level. We further introduce fine-grained distillation (FGD) scheme, aiming to minimize the conditional entropy H(p" ApA)
) at the upper level.
4.2 IPPORMATION-ENHATTE
LIF NEURON
As mentioned in Fig. 2 (b), the information in the self- self-attention map of the OSD-Transfonen follows a binomial distribution, which limits the representational capacity of the attention module. Therefore, propose the ceermation-enhancee LIF (IE-LIF) neuron and adjust the information distribution of
Q-SDSA at the lower level, focusing on maximizing the information entropy H(p²).
Proposition 1. Given the SNN Transformer and ANN Transformer models, where the distributions of the query (q), key (k), and value (v) follow binomial Brr, T) and normal N (H, o) distributions, respectively, is postulated that as the SNN's time step T tends infinity, there exist parameters H, o, and such that the average entropy over time of the SNN's attention scores H(E? (t))
equals ANN attention scores' entropy H(pA).
Proof can be found in Appendix B. According to Proposition 1., within infinite time steps T. the attention matrix values (e.g.. qa v3))) in the Transfoororm have the same information representation as those in the ANN Transformer. However, numerous time steps T will inevitably lead to lateney and huge energy consumption. Recently, Hao et al. (2023b) and Hu et al. (2023) achieved high-performance conversion by transforming quantized ANNs into SNNs and using fewer time steps. This prompted us to train directly using multi-bit values. We only need to ensure that inference is spike-driven. Thus, we propose the concept of IE-LIF, in which Eq. 2 can be written as:
a [t]
clippv [t), 0, 0,b]
(9)
b
where a is the multi-bit output of IE-LIF, and b represents the maximum integer value emitted by the IE-LIF, which is equipped with the baseline. Since Eq. 9 is non-differentiable, we employ the straight-through estimator (STE) (Bengio et al., 2013) to retain the gradient derivation during backpropagation.
Previous SNNs have utilized multi-bit spikes (integers) (Hao et al., 2024; Ponghiran & Roy, 2022) continuous values (Wu et al., 2021) to reduce quantization error, thereby alleviating the shortcomings of binary spikes. However, this approach raises concerns because it can undermine the inherent spike-driven characteristics of SNNs. We propose solution where IE-LIF uses multi-bit values during the training phase and subsequently converts these values to binary spikes for inference, as depicted in Fig. (c). Moreover, the output x [t] of each layer in the SNN is represented as:
+ wf. a((t) w Σ s
(10)
where a [t represents multi-bit spikes during training with one timestep and is denoted as {0, 0.25, 0.5, 0.75, 1)7=1, while s'[ti represents binary spikes during inference and is extended to virtual timesteps denoted as {0, 10.11 1- with a maximum integer value set to 4 in this paper.
With the introduction of IE-LIF, our next step involves low-level optimization maximize the entropy of attention scores p We first observed that the membrane potentials {qmem Kmem: Vmem) in the IE-LIF model within Q-SDSA approximately follow normal distribution A (P, o) with p as also noted in (Guo et al., 2022a;b). Then we provide the formula for calculating the maximum information entropy.
Proposition 2. Given a random variable x following a normal distribution N(u, σ), the information entropy H(x) achieves its maximum value of 1 log (x).
Proof can be found in Appendix C. According to Proposition 2., the maximum information entropy of membrane potential is H (Pmem) = 1 log 2 (Pmem). acting as the upper limit for the information entropy of spike attention scores. Through the application of IE-LIF, the information entropy of spike attention scores becomes a discrete representation of membrane potential's 's information entropy:
H(ps) = Σ
(((pmem ( (p - ))) loog (G(pmem (00mem))((unn (Pmem -ppmmmm-))))) 1;)).
(11)
O=

---

Published as a conference paper at ICLR 2025
where (((mme)) is the Gaussian distribution function of membrane potential Premm and 6(-) is the Dirac delta function and J-x ((z) dr 1. However, since Pmem ~ (0,00) when Eq. 9 is applied to membrane potentials Pmem. spikes are emitted only when Pnnm exceeds O. This may lead to the distributions of attention scores resembling the right tail of a discrete normal distribution, causing mismatched attention scores between SNNs and ANNs. Hence, we propose membrane potential rectify function (MPRF) Φ (.), which can be expressed as:
Pmem (u((pem))
Pmem (pmem)
+y+0 a,
(12)
o (Pmem)
where Punm and Pinm represent the membrane potential of Q-SDSA before and after applying the MPRF, and Y, a are the learnable hyperparameters. The MPRF is only executed when l Q-SDSA. After MPRF, attention score distribution aligns more closely with the desired normal distribution, reducing the mismatch between SNNs and ANNs. At this point, the information entropy of SNN attention scores is p = E ((Pmee))(PPmmss })) Furthermore, the MPRF can be incorporated into the weights w' during inference, details of which can found in Appendix D.
4.3 FINE-GRAINED DISTILLATION
The IE-LIF neuron has maximized the information entropy (p Building upon this, we achieve the optimization goal of Eq. 8 by proposing a fine-grained distillation (FGD), which adjusts the distribution of Q-SDSA at the upper level to minimize the conditional entropy H(p (p^).
The proposed FGD achieves minimal conditional entropy by minimizing the norm distance between isr and f with the optimal solution being ~ - p It utilizes appropriate distillation activations and meticulously designed similarity matrices to effectively leverage knowledge from the teacher model. Therefore, the FGD scheme is defined as:
L
LLOD Σ
F pd
Fpoll
where
pxp Fp
(13)
| x xpxpTI1 1
p 1=1
where I. denotes the number of layers in the Transformer, H represents the number of heads, and ||- indicates E2 normalization. During backpropagation, gradient updation drives the attention matrix of the DSD-Transformer and the same ANN Transformer closer, thereby minimizing H(p" (p4). The overall training loss function of our QSD-Transformer is defined as:
T
L = LEE(
s s((l),)) + ALrOD,
(14)
where sf is the output of our TSD-TTranffommrr, LCE is the cross-entropy loss (Rathi & Roy, 2021) for ensuring task performance, and LTGD is the proposed distillation loss for enhancing information entropy. X a coefficient balance these two loss functions, and set to 2 in this paper.
5 EXPERIMENT
In this section, we validate the QSD- DSD-Transformer on various vision tasks, including image classifica- tion, object detection, semantic segmentation, and transfer learning. Then, we ablate the proposed scheme to prove the effectiveness of our method. For further detailed information on datasets, power calculations, experimental setups, and hyperparameters, refer Appendix E and G.
ImageNet classification. We evaluate the QSD- TrDnsfsssfors'sss effectiveness in image classification using the challenging ImageNet- 1K dataset (Deng et al., 2009). The comparison results are sum- marized in Table 1. Notably, with only 6.8M parameters, the QSD- TSD--Tranfformr achieves a top-1 accuracy 80.3% the SNN domain, showeasing significant advantages in both accuracy and efficiency. Specifically, OSD-Transformer vs. SD-Transformer v2 (Yao et al., 2023a)
Spik- iegResformer (Shi et al., 2024): Param, 6.8M vs. 55.4M vs. 60.4M; Power: 8.7mJJ vs. 52.4mJ 9.7mJ: Acc, 80.3% 79.7% 78.7% 7%. When compared to the SOTA model in the SNN field, SD- Transformer v2 (Yao et al., 2023a), our method achieves a 0.6% improvement in accuracy while reducing parameter by 87.58% and power by 83.40% In summary, the QSD- Transforrrr
7

---

Published as a conference paper at ICLR 2025
Query
key
value
0S0 frandbeesgooouur 424
32 S5.148
0.2
0.29
0.25
Accuraty
040
0.0
0.5
0.00
0.
0.5
0.00
60
0.3
Transforme
query
key
value
76-
.00.51...00
Spiidormer
9.3
75
02
02
74-
01
10
20
30
40
50
Power(m]]
60
05
1.0
15
05
0.0
0.5
1.0
35
60
0.2
(a)
(b)
Figure 3: (a) Comparative results of accuracy, power, and parameters on ImageNet. (b) Comparison of information distribution in Q-SDSA before and after using the proposed IE-LIF and FGD scheme.
Table ImageNet classification results (Deng al., 2009). Bits' denotes the width of the weight and activity, respectively. Power is the estimation of energy consumption same as Yao et al. (2023b).
indicates self celf-impllmmenatioon results with open-source code (Yu al., 2023).
Spike driven
Time
Param
Power
Acc.
Method
Architecture
Bits
Step
(M)
(mJ)
(%)
Transformer (Yu et al. 2023)
CAtormer"
32-32
NIA
15.1
40.3
79.9
QCFS (Bu et al,, 2021)
ResNet-34
32-1
256
21.8
73.4
MST (Wang et al., 2023)
Swin-T
32-1
128
28.5
77.9
SFW-ResNet-34
32-1
25.6
4.9
67.8
SEW-ResNet (Fang al., 2021)
SEW-ResNet-152
12.9
69.2
MS-ResNet-B3
32-1
MS-ResNet(Ilu et al., 2024b)
MS-ResNet- 104
32-1
77.3
10.2
75.3
Spikformer-8-512
32-1
29.7
11.6
73.4
Spikformer (Zhou et al., 2023b)
Spikformer-8- 768
32-1
66.3
21.5
74.8
AD-Transtormer---3
32-1
29.7
4.5
74.6
AD-Transformer (Yao et al., 2023b)
SD Transformeer-8--. 8-766
32-1
4
66.3
6.1
76.3
SpikingResformer.
32-1
11.1
4,2
74.3
SpikingResformer (Shi al., 2024)
SpikingResformer-LL
32-1
4
60.4
9.7
78.7
SD Tramsfomme v2-T
32-1
15.1
16.7
74.1
To-nrrasffrrmr (Yao et al., 2023a)
SD Transformer v2-M
32-1
31.3
32.8
77.2
SD-1 To-Transformer v2-L
55.4
52.4
79.7
SD- Transformer v2-T
77.5
QSD TaD-Transfrrmrr
SD- Transformer v2-M
78.9
AD-nransfrrmer v2-L
8.7
80.3
Table 2: Object detection results COCO 2017 (Lin et al., 2014).
Spike driven
Time Step
Param
Power
mAP@0.5
Method
Architecture
Bits
(M)
(mJ)
(%)
Transformer el al., 2023)
32-32
890.6
Transformer (Zhu al., 2020)
DETR
32-32
N/A
41.0
860.2
$7.0
Spiking-Yon Yolo (Kim
2020)
ResNet-18
32-1
3500
10.2
25.7
Spike Calibration (Li al., 2022)
ResNet-18
32-1
512
17.1
45.3
EMS-SNN (Su al., 2023)
EMS EMS-ResNettl
32-1
26.9
50.1
SD-Transfomer v2 (Yao ct al., 2023a)
SD- To-Trnsformer v2-M
32-1
75.0
140.8
51.2
SD- 3D-Trassformer v2-T
16.9
45.1
48.1
Toaassfsfrrrrr
SD- TD-Trassformrr v2-M
117.2
establishes better results both accuracy as well as efficiency on ImageNet-IK in the SNN domain and especially shines in efficiency.
Object detection. We evaluate the efficacy the QSD-QQQDTaatt Transformer on the object detection task and select the classic and large-scale COCO (Lin et al., 2014) dataset
benchmark for evaluation. Similar to the previous work (Yao et al., 2023a), we convert the mmdetection (Chen et al., 2019) codebase into spiking version by IE-LIF and then use it execute model. We employ the
8

---

Published as a conference paper at ICLR 2025
Table 3: Semantic segmentation results on ADE20K (Zhou et al., 2017).
Spike -driven
Time
Param
Power (mJ)
MIoU
Method
Architecture
Bits
Step
(M)
(%)
Segforee (Xie el al., 2021)
Segformer
32-32
X
N/A
3.8
38,9
37.4
DecpLaab-V3 (Zhang et al., 2022a)
DecpLabbVV V3
32-32
x
N/A
68.1
1240.6
42.7
SD-Trunstormer v2 (Yao et al., 2023a)
SD-Traasformer v2-M
32-1
/
59.8
183.6
35.3
SD> Ir-nstasermmrr V2-T
4-1
3.3
17.5
39.0
QSD-Transformer
SD-Tramsformer v2-M
4-1
/
9.6
37.9
40,5
Table 4: Transfer learning results on CIFARIO, CIFAR100 and CIFARIO-DVS.
CIFARIO
CIFARI00
CIFARIO-DVS
Param (M)
Method
T
Acc. (%)
T
Acc. (%)
T
Acc. (%)
Spikformer (Zhou et al., 2023b)
29.1
4
97.0
4
83.8
Spikingee (Shi al., 2024)
17.3
97.4
85.9
10
84.8
1.8
97.8:10.1
86.6
10
88.
6.8
4
98.4
4
87.6
10
0.1
SSD-Transformer as the backbone to extract features, along with Mask R-CNN (He et al., 2017) for object detection. The backbone is initialized with the pre-trained OSD-Transformer on ImageNet- IK, and other added layers are initialized with Xavier (Glorot & Bengio, 2010). The comparison results are summarized in Table 2. Obviously, the Transffofrre outperforms the existing state-of-the-an methods in the SNN domain by significant margin. More specifically, our method exceeds the performance of SD-Transformer v2 by 5.8% terms of the mAP@0.5 metric, while utilizing fewer than half the parameters. In conclusion, our approach demonstrates efficacy in object detection tasks and has established a new benchmark for detection within the SNN domain.
Semantic segmentation. We validate the efficacy of the QSD- -SD-Transformer on the semantic segmen- tation task and select the challenging ADE20K (Zhou et al., 2017) dataset. Similar to the procedures in object detection, we converted the mmsegmentation (Contributors, 2020) codebase into spiking version and utilized it to execute our model. The QSD- Transformer serves as the backbone for feature extraction, integrated with Semantic FPN (Kirillov et al., 2019) for segmentation. The initialization similar to that in the object detection task. The backbone is initialized with pre-trained model on ImageNet- IK, and the added layers are initialized using Xavier (Glorot & Bengio, 2010). Since SD-Transformer v2 is the only work in the SNN field reporting results on ADE20K, we compare our approach with advanced deep models. As depicted in Table 3, our method significantly outperforms SD-Transformer v2 (Yao et al., 2023a) across all comparison metrics, achieving an 83.94% reduction in parameters, 79.36% decrease in power, and a 5.2% increase in MIoU. Moreover, our method achieves a comparable MIoU to the advanced DeepLab-Vy V3 in the ANN domain while substantially reducing both parameters and power.
Transfer learning. We demonstrate the efficacy of the OSD-Transformed
transfer learning tasks. We evaluate the model's transfer learning capability on both static datasets (CIFAR) (Krizhevsky et al., 2009) and the neuromorphic dataset (CIFAR 10-DVS) (Li et al., 2017) using five repeated experiments with different random seeds. To assess this ability, we fine-tune the pre-trained weights from the ImageNet- IK dataset on these selected datasets. Compared with existing transfer learning methods in SNNs, such as SpikingResformer and Spikformer, the proposed DSD-Transforme demonstrates state-of-the oh- results. It achieves 98.4% accuracy on CIFAR-10. 87.6% accuracy on CIFAR- 100, and 89.8% accuracy on CIFAR 10-DVS, surpassing Reskrmnee by 1.0%, 1.7%, and 5.0%, respectively. Thus, our method achieves the best performance across various computer vision tasks.
Ablation study. We first ablate two components of the OSD-Trranforrest namely the IE-LIF and FGD schemes, to verify the effectiveness the proposed method. Additionally, we quantized the weights to 4, 3, and 2 bits to study the impact of bit width on performance. Experiments are performed on the ImageNet dataset. The results are shown in Table 5, where the QSD-Transformer baseline without the IE-LIF neuron and FGD scheme achieves accuracy of 70.0%. In contrast, using the IE-LIF neuron increases the accuracy by 5.8%. With both the IE-LIF neuron and FGD scheme, the accuracy further reaches 77.5%. Therefore, both the proposed IE-LIF neuron and the FGD scheme

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study of the IE-LIF, FGD, and different bits.
Architecture
IE-LIF
FGD
Weight Bits
Acc.(%)
4
70.0
4
75.8
SD-Trans SD-Transformer v2 (Yao et al., 2023a)
/
4
77.5
3
76.9
75.0 2
4
64.1
4
70.1
Spikformer (Zhou et al., 2023b)
4
75.5
3
74.1
2
73.1
improve performance, and their combined use bring more significant accuracy. Moreover, we also investigate the impact of bit-width on performance. It can be seen from Table that the accuracy decreases with bit width reduction. Notably, when the weights quantized to 2-bit, our method still achieves 75.0% accuracy.
Next, delve into the application of our method within the Spikformer architecture (Zhou al., 2023b) to validate its robustness and scalability. Specifically, we initially established a Spikformer-8- 384 model as quantization baseline under the conditions of time step of and 4-bit quantization of the weights. Subsequently, we conducted ablation experiments of various modules and weight bit-widths on the ImageNet dataset. As shown in Table 5, direct quantization of Spikformer indicates significant performance drop of 6. 1.4% under standard quantization conditions. Then, by applying our IE-LIF spiking neurons, we were able to enhance the accuracy by 6.0%. Furthermore, the accuracy was further improved to 75.5% by combining the IE-LIF neurons with the FGD scheme. also investigated the impact of bit-width on performance. Notably, our method maintains 73. % accuracy with 2-bit weight quantization. The above results demonstrate that our method can be robustly applied to various spiking-bast -based Transformer models.
Finally, our ablation studies the activity bit and training time step T the IE-LIF model reveal that augmenting the activity bit b substantially boosts per- formance. As depicted Table 6, elevating from I to 4 with 7 results in 9.9% increase in accu- racy; conversely, with b = 1. raising 'I' from I to 4 yields more modest 2.4% improvement. This dis- parity arises because augmenting the activity bits b enhances the model's information capacity and mit- igates quantization errors, whereas increasing the training time step T has a less pronounced impact, likely due to the redundancy inherent in spike trains.
Table Ablation study of the activity bits and training time step the Tra-Transformrr.
Bits (b)
Timestep (T)
Acc.(%)
67.6
68.5
70.0
2
71.6
2
77.4
77.5
Furthermore, extending the time step incurs significant memory and energy costs, which is not the case for increasing the activity bit b. These findings underscore that the quantization performance is more sensitive to the activity bits than the time step configuration.
CONCLUSION
In this paper, we first introduce the lightweight spike-driven transformer, namely the QSD- Transformer, which quantifies the weights from 32-bit low-bit. By employing both low-bit weights and 1-bit spike activities, TSD-Trrasformer has demonstrated significant energy efficiency, Despite exhibiting efficiency benefits, the QSD-Transformer suffers from performance degradation. We reveal that this is attributed to the SID problem and propose bi-level optimization strategy to solve this challenge. At the lower level. we propose the IE-LIF neuron, which generates multi-bir bit spikes in training while maintaining spike-driven behavior during inference. At the upper level, introduce the FGD scheme, which optimizes attention distribution between the Q-SDSA and its ANN counterpart. Extensive experiments show that method achieves state-of-the-ar results in both
10

---

Published as a conference paper at ICLR 2025
performance and efficiency various vision tasks, paving the way for the practical deployment of spike-based Transformers in resource- -eoource--imttedd platforms.
ACKNOWLEDGEMENT
This work was supported in part by the National Natural Science Foundation of China under grant U20B2063, 62220106008, and 62106038, the Sichuan Science and Technology Program under Grant 2024NSFTD0034 and 2023YFG0259. The research is supported by Shenzhen Science and Technology Program ZDSYS2023062099 230020000660990000066 and Shenzhen Science and Technology Research Fund (Fundamental Research Key Project Grant No.
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308. 3432, 2013.
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 696-697, 2020.
Tong Bu, Wei Fang, Jianhao Ding, PENGLIN DAI, Zhaofei Yu, and Tiejun Huang. Optimal ann-smn conversion for high-accuracy and ultra-low- lattenc spiking neural networks. International Conference Learning Representations, 2021.
Tong Bu, Jianhao Ding, Zhaofei Yu, and Tiejun Huang. Optimized potential initialization for low- latency spiking neural networks. Proceedings of the AAAI Conference Artificial Intelligence, 2022.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 1906:9906000 07150.775,,, 2019.
Yanqi Chen, Zhaofei Yu, Wei Fang, Zhengyu Ma. Tiejun Huang, and Yonghong Tian. State tran- sition of dendritic spines improves learning of sparse spiking neural networks. In International Conference on Machine Learning, pp. 3701-3715. PMLR, 2022.
Yuan Shih Chow and Henry Teicher. Probability theory: independence, interchangeability martin- gales. Springer Science & Business Media, 2012.
Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research, 153:235-256 2007.
MMSegmentation Contributors. Mmsegmentation: Openmmlab semantic segmentation toolbox and benchmark, 2020.
Mike Davies, Narayan Srinivasa, Tsung-Har Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro, 38(1):82-99) 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database, 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. IEEE, 2009.
Lei Deng, Yujie Wu, Yifan Hu, Ling Liang, Guoqi Li, Xing Hu, Yufei Ding, Peng Li, and Yuan Xie. Comprehensive compression using admm optimization and activity regularization. IEEE Transactions on Neural Networks and Learning Systems, 34(6):2791 2023. doi: 10.1 1..100//INN.SS222233000000
Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ann-snn conversion for fast and accurate inference in deep spiking neural networks. arXiv preprint arXiv:2105. 105. 1654, 2021.
11

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Jason K Eshraghian, Max Ward, Emre O Neftci, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mohammed Bennamoun, Doo Seok Jeong, and Wei D Lu. Training spiking neural networks using lessons from deep learning. Proceedings of the IEEE, 2023.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances Neural Information Processing Systems, 11-21056-21063 2021.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Yufei Guo, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Yinglei Wang, Xuhui Huang, and Zhe Ma. Im- loss: information maximization loss for spiking neural networks. Advances in Neural Information Processing Systems, 35:156- 166, 2022a.
Yufei Guo, Yuanpei Chen, Liwen Zhang, YingLei Wang, Xiaode Liu, Xinyi Tong, Yuanyuan Ou, Xuhui Huang, and Zhe Ma. Reducing information loss for spiking neural networks. In European Conference on Computer Vision, pp. 36-52. Springer, 2022b.
Yufei Guo, Weihang Peng, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Xuhui Huang, and Zhe Ma. Joint a-snn: Joint training of artificial and spiking neural networks via self-distillation and weight factorization. Pattern Recognition, 142:109639, 2023.
Yufei Guo, Yuanpei Chen, Xiaode Liu, Weihang Peng, Yuhan Zhang, Xuhui Huang. and Zhe Ma. Ternary spike: Learning ternary spikes for spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 12244-12255 2024.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Zecheng Hao, Tong Bu, Jianhao Ding. Tiejun Huang, and Zhaofei Yu. Reducing ann-snn conversion error through residual membrane potential. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11- 2023a.
Zecheng Hao, Jianhao Ding, Tong Bu, Tiejun Huang, and Zhaofei Yu. Bridging the gap between anns and snns by calibrating offset spikes. arXiv preprint arXiv:2302, 2023b.
Zecheng Hao, Xinyu Shi, Zhiyu Pan, Yujia Liu, Zhaofei Yu, and Tiejun Huang. Lm-ht snn: Enhancing the performance of to counterpart through learnable multi-hierarchical threshold model. arXiv preprint arXiv:2402.00411 2024.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2961-2969, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.023: 2015.
Mark Horowitz. 1.1 computing's ss energy problem (and what we can do about it). In 20/4 IEEE International Solid-State State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14, 2014. doi: 1LL.L///SSCC.2214.7777777
JiaKui Hu, Man Yao, Xuerui Qiu, Yuhong Chou, Yuxuan Cai, Ning Qiao, Yonghong Tian, Bo Xu, and Guoqi Li. High-pert performnform temporal reversible spiking neural networks with o(l) training memory and o(1) inference cost. arXiv preprint arXiv:2405. 16466, 2024a.
12

---

Published as a conference paper at ICLR 2025
Yangfan Hu, Qian Zheng, Xudong Jiang, and Gang Pan. Fast-snn: Fast spiking neural network by converting quantized ann. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks toward deep residual learning. IEEE Transactions on Neural Networks and Learning Systems, 2024b.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalemichenko. Quantization and training of neural networks for efficient integer-arithmeeticonlly inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018.
Saeed Reza Kheradpisheh, Maryam Mirsadeghi, and Timothée Masquelier. Bs4nn: Binarized spiking neural networks with temporal coding and learning. Neural Processing Letters, 54(2): 2022.
Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for energy-efficient object detection. Proceedings of the AAAI Conference Artificial Intelligence, 34:11270-11277, Apr. 2020.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6399-6408, 2019.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.0883 2018
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Aditya Kusupati, Vivek Ramanujan. Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade. and Ali Farhadi. Soft threshold weight reparamcterization for learnable sparsity. In International Conference on Machine Learning, pp. 5544 5555. PMLR, 2020.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifarl0-dvs: an event-stream dataset for object classification. Frontiers Neuroscience, 11:309, 2017.
Yang Li, Xiang He, Yiting Dong, Qinggun Kong, and Yi Zeng. Spike calibration: Fast and accurate conversion of spiking neural network for object detection and segmentation. arXiv preprint arXiv:2207.02702. 2022.
Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In International Conference on Machine Learning, pp. 6316-6325. PMLR, 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Zechun Liu, Wenhan Luo, Baoyuan Wu, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Binarizing deep network towards real-network performance. International Journal of Computer Vision, 088202-2119 2020.
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. In Advances in Neural Information Processing Systems. volume 34, pp. 28092-288 28103. Curran Associates, Inc., 2021.
Changze Lv. Tianlong Li, Jianhan Xu, Chenxi Gu, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Spikebert: language spikformer trained with two-stage knowledge distillation from bert. arXiv preprint arXiv:2308. /5/22, 2023.
Wolfgang Maass. Networks of spiking neurons: The third generation of neural network models. Neural Networks, 1559)1165,,167,, 1997.
13

---

Published as a conference paper at ICLR 2025
Paul A Merolla, John V Arthur, Rodrigo Alvarez-leaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking- neuron integrated circuit with a scalable communication network and interface. Science, 345 (6197))668-61) 2014.
Priyadarshini Panda, Sai Apama Aketi, and Kaushik Roy. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience, 14:653, 2020.
Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6): 191-1253, 2003.
Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):10--11)) 2019.
Wachirawit Ponghiran and Kaushik Roy. Spiking neural networks with improved inherent recurrence dynamics for sequential learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8001-8008, 2022.
Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. Binary neural networks: survey. Patlern Recognition, 105:107281, 2020. ISSN 0031-3203.
Haotong Qin, Yifu Ding, Mingyuan Zhang, YAN Qinghua, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu. Bibert: Accurate fully binarized bert. In International Conference Learning Representations, 2021.
Xuerui Qiu, Man Yao, Jieyuan Zhang, Yuhong Chou, Ning Qiao, Shibo Zhou, Bo Xu, and Guoqi Li. Efficient 3d recognition with event-driven spike sparse convolution. arXiv preprint arXiv:2412.07330,, 2024a.
Xuerui Qiu, Rui-Jie Zhu, Yuhong Chou, Zhaorui Wang, Liang-Jian Deng, and Guoqi Li. Gated attention coding for training nigh-peeformance and efficient spiking neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 38(1):601-610, Mar. 2024b.
Nitin Rathi and Kaushik Roy. Diet-snn: low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions Neural Networks and Learning Systems, 2021.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607-617, 2019.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. Advances in neural information processing systems, 33:1 1010-11300000 2020.
Yimeng Shan, Malu Zhang, Rui-jie Zhu, Xuerui Qiu, Jason K Eshraghian, and Haicheng Qu. Advancing spiking neural networks towards multiscale spatiotemporal interaction learning. arXiv preprint arXiv:2405. /3672, 2024.
Jiangrong Shen, Qi Xu, Jian Liu, Yueming Wang, Gang Pan, and Huajin Tang. Esl-snns: An evolutionary structure learning strategy for spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 86-93, 2023.
Jiangrong Shen, Wenyao Ni, Qi Xu, and Huajin Tang. Efficient spiking neural networks with sparse selective activation for continual learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 611-619, 2024.
Xinyu Shi, Jianhao Ding, Zecheng Hao, and Zhaofei Yu. Towards energy efficient spiking neural networks: An unstructured pruning framework. In The Twelfth International Conference on Learning Representations, 2023.
Xinyu Shi, Zecheng Hao, and Zhaofei Yu. Spikingresformer: Bridging resnet and vision transformer in spiking neural networks. arXiv preprint arXiv:2403. 14302, 2024.
14

---

Published as a conference paper at ICLR 2025
Sumit Bam Shrestha and Garrick Orchard. Slayer: spike layer enror reassignment in time. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 1429-1428, 2018.
Ankur Sinha, Pekka Malo, and Kalyanmoy Deb. A review on bilevel optimization: From classical to evolutionary approaches and applications. IEEE transactions on evolutionary computation, 22(2): 276-295, 2017.
Evangelos Stromatias, Daniel Neil, Michael Pfeiffer, Francesco Galluppi, Steve B Furber, and Shih-Chii Liu. Robustness of spiking deep belief networks to noise and reduced bit precision of neuro-inspired hardware platforms. Frontiers in neuroscience, 9:141542, 2015.
Qiaoyi Su, Yuhong Chou, Yifan Hu, Jianing Li, Shijie Mei, Ziyang Zhang, and Guoqi Li. Deep directly-trained spiking neural networks for object detection. Proceedings of the IEEE/CVP International Conference Computer Vision, pp. 6555-6565, 2023.
Sugahara Takuya, Renyuan Zhang, and Yasuhiko Nakashima. Training low-latency spiking neural network through knowledge distillation. In 2021 IEEE Symposium in Low-Power and High-Speed Chips, pp. 1-3. IEEE, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Thi Diem Tran, Kien Trung Le, and An Luong Truong Nguyen. Training low-latency deep spiking neural networks with knowledge distillation and batch normalization through time. In 2022 5th International Conference on Computational Intelligence and Networks, pp. 01-06. IEEE, 2022.
Ziqing Wang, Yuetong Fang. Jiahang Cao, Qiang Zhang, Zhongrui Wang, and Renjing Xu. Masked spiking transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1761-1771, 2023.
Wenjie Wei, Malu Zhang, Jilin Zhang, Ammar Belatreche, Jibin Wu, Zijing Xu, Xuerui Qiu, Hong Chen, Yang Yang. and Haizhou Li. Event-driven learning for spiking neural networks. arXiv preprint ar 20x3: 00270, 2024.
Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European Conference on Computer Vision, pp. 68-85. Springer. 2022.
Yujie Wu, Lei Deng, Guoqi Li. Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in Neuroscience, 12:331, 2018.
Zhenzhi Wu, Hehui Zhang, Yihan Lin, Guoqi Li, Meng Wang, and Tang. Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. IEEE Transactions on Neural Networks and Learning Systems, 33( 33(1 (1:(1)))6299666 6262, 2021.
Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Di He, and Zhouchen Lin. Online training through time for spiking neural networks. Advances in Neural Information Processing Systems, 35:20717-20733 2022.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077-12090 2021.
Qi Xu, Yaxin Li, Jiangrong Shen, Jian K Liu, Huajin Tang, and Gang Pan. Constructing deep spiking neural networks from artificial neural networks with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7886-7895, 2023.
Man Yao, JiaKui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, XU Bo, and Guoqi Li. drikendriven transformer v2: Meta spiking neural network architecture inspiring the design of next- next- gexeerenntttion neuromorphic chips. In The Twelfth International Conference on Learning Representations, 2023a.
15

---

Published as a conference paper at ICLR 2025
Man Yao, JiaKui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer. In Advances in Neural Information Processing Systems, volume 36, pp. 64058, 2023b.
Man Yao, Xuerui Qiu, Tianxiang Hu, Jiakui Hu, Yuhong Chou, Keyu Tian, Jianxing Liao, Luzi- wei Leng, Bo Xu, and Guoqi Li. Scaling spike-driven transformer with efficient spike firing approximation training. arXiv preprint arXiv:2411.1606 2024.
Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R Manmatha, et al. Resnest: Split-attention networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2736-2746, 2022a.
Jiqing Zhang, Bo Dong, Haiwei Zhang, Jianchuan Ding, Felix Heide, Baocai Yin, and Xin Yang. Spiking transformers for event-based single object tracking. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 8801-8810, 2022b.
Jiyuan Zhang, Lulu Tang, Zhaofei Yu, Jiwen Lu, and Tiejun Huang. Spike transformer: Monocular 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part VII, pp. 34 52. Springer, 2022c.
depth estimation for spiking camera. In Computer Vision ECCO
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 633-641, 2017.
Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Zhengyu Ma, Han Zhang, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual learning for transformer-hase spiking neural network. arXiv preprint WtX.:23344.19555 2023a.
Chenlin Zhou, Han Zhang, Liutao Yu, Yumin Ye, Zhaokun Zhou. Liwei Huang, Zhengyu Ma, Xiaopeng Fan, Huihui Zhou, and Yonghong Tian. Direct training high-performance deep spiking neural networks: a review of theories and methods. Frontiers in Neuroscience, 18: 1383844, 2024a.
Chenlin Zhou, Han Zhang, Zhaokun Zhou, Liutao Yu, Liwei Huang, Xiaopeng Fan, Li Yuan, Zhengyu Ma, Huihui Zhou and Yonghong Tian. Qkformer: Hierarchical spiking transformer using qk attention. arXiv preprint Mrxiv:240:: 16552, 2024b.
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng YAN, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. In The Eleventh Interna- tional Conference on Learning Representations, 2023b.
Zhaokun Zhou, Kaiwei Che, Wei Fang. Keyu Tian, Yuesheng Zhu, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer v2: Join the high accuracy club on imagenet with an snn ticket. arXiv preprint arXiv:2401.0200 02020, 2024c.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference Learning Representations, 2020.
16

---

where s' [t] and v(tt] represent the binary and membrane potential of the neuron in layer l, at time is non-differentiable. To overcome this problem, (Wu et al., 2018) propose the surrogate function to make only the neurons whose membrane potentials close to the Published as a conference paper at ICLR 2025
Appendix
A BACKPROPAGATION PROCESS OF SPIKING NEURONS
There exist two primary methods of training hegh-performacce SNNs. One way is to discretize ANN into spike form through neuron equivalence (Li et al., 2021; Bu et al., 2022; Hao et al., 2023a; Ding et al., 2021), i.e., ANN-to-SNN conversion, but this requires long simulation time step and boosts the energy consumption. We employ the direct training method (Wu et al., 2018; Qiu et al., 2024b; Wei et al., 2024) and apply surrogate gradient training.
Then in this section, we introduce the training process of SNN gradient descent and the parameter update method of apatio-tempone backpropagation (STBP) (Wu et al., 2018; Xiao et al., 2022). SNNs' parameters can be taught using gradient descent techniques, just like ANNs, after determining the derivative of the generation process. Moreover, the accumulated gradients of loss L with respect to weights w at layer can be calculated as:
ac Ow
ac
0s 0
[t]
Ov @v
Σ II
av [i+t) OvctI
av i - Os 0
2v
[d]
(v
Oz
[1] [
Ow
ds [i] Ove+l
Ow
T<Hi=t-]
(15)
ds([t)
Moreover, notice that [v
firing threshold receive nonzero gradients during backpropagation. In this paper, the rectangle function, which has been shown to be effective in gradient descent and may be calculated by:
Ös [t]
sign (Iv'lt) 1
a 2
(16)
0v
where a is a defined coefficient for controlling the width of the gradient window.
B PROOF OF THE PROPOSITION 1.
Proposition 1. Given the SNN Transformer and ANN Transformer models, where the distributions of the query (q). key (k), and value (v) follow binomial ((r, T) and normal (u,,o) distributions, respectively, it is postulated that as the SNN's s time step T tends to infinity, there exist parameters
μ, and such that the average entropy over time of the SNN's attention scores H(E ((t))
equals ANN attention scores' entropy A)
Proof. Proposition 1. can be restated as follows:
T
lim 3p, or H H (tt)) H(p T
o
(17)
where A and p represent the query q, key k, and value v in the same architecture ANN (leacher) and QSD-Trrassformer (student), and following the binomial B(r, T)) and normal N a) distribu- tions. @ is the parameters of the student OSSD-Trasffrmerr,
Assume p (2)] 13(3] D are t independent random variables, each following bino- mial distribution. The expectation is B(p it)) = S r') where y [t] is the firing rate of the SNN p r rS(t] where a. Let the characteristic function (Chow & Teicher, 2012) of the
at time t. The variance is given by D(p (t) (t. And Let y |
[ty) = 0 and (Dy
random variable [t] be (yy(n) Then let the random variable 17 [1]+y + +3)+..+y* Then the characteristic function of 77 is:
vio
i
( VT ... [ |
T (18)
in =
VT
Py (o)
VT
Then when SNN's timestep T 00, t/g can be expanded at the point O using the Taylor series:
17

---

Published as a conference paper at ICLR 2025
Pysta( 1-1) 1) (1) + (0)
so (0((
2
#o
(
2
VTo"
(To
+
2!
TT
+o
(To
Since (y))1(0) 1, & (0) = and sy (0) = -0 we have:
j
1
9y
=
1
+o
VTo
21
VTo
T
j
2
T
((vio)
Pyste]
=
+o
+o
TTo
21
TTo
2T
Hence:
lim T-+00
9y8 (c)
j
T
= lim T-20
1
8
o
T
e 52
(19)
VTo
2T
where e-=
happens to be the characteristic function of a random variable following the standard normal distribution N (0, 1), 1y follows the standard normal distribution, which distribution the same to the attention score in ANN Transformer. Hence, as the SNN's time step 'I' tends to infinity,
3u, o, such that H(T p [t)) H(P)).
C PROOF OF THE PROPOSITION 2.
Proposition 2. For a random variable x ~ Nu, o),, the information entropy x reaches its maximum value H(xx) log 2 (x) and is observed to increase with the expansion of variance a.
Proof. For a continuous random variable x obeying a normal distribution, its probability density function p(z) is given by:
(x
p(x) (2T02)
1/2 exp
(20)
20
Consequently, the differential entropy of can be calculated
H(x) = -- = p(x) log p(z)(,
1
(x 11)2
(x 11)
(2T02)11/2 exp
log (2To²)11/2 exp
dx,
202
202
/ /exp.
(x H)²
(x 11)
log (V2TT)
dit
(2ma2)1/2
20
202
(x - 11)2
= (2m02)1/22
log
V2TTO
exp
20²
dx+
(x 41)
(x 1n
(2T02)1/2
exp
202
20² dr,
log (v2TO)
(x - u)²
(x - 112
(x - 4)
exp
dr +
exp
ds,
22 1/2
20
22 1/2
20
202
log (V2TO)
2
(2ma2)1122
V20
exp
2 - / V20
d x - /
22
+
(2TT²211/2 V20
I -- 1
2
(x 11)2
I - 1
exp
22a
20
20
(21)
18

---

Published as a conference paper at ICLR 2025
Moreover, it can be easily proven that
E v² dy = VT.
(22)
Thus,
H(xx) = log (V2TT)
000
v² dy
2
e e- dy,
C
+
VT
Vx xcc
= log (V2TT)
1
1
0
00
e v²
dy
VT
2
(23)
log (2TO
2'
1
log 2TT²²)
+ +1);
2
1
log (TTEO²
2
D THEORETICAL ANALYSIS FUSION MPRF AND WEIGHTS
In Section 4.2, we introduce membrane potential rectify function (MPRF) ('(-) aimed at maximizing the information entropy of the attention score. The inherent homogeneity of convolution operations permits the subsequent batch normalization (BN) and linear scaling transformations to be seamlessly integrated into the convolutional layer with an added bias during deployment. This approach enables the model conduct inference rapidly without incurring additional computational overhead. Specifically, we utilize Eq. 24 to represent the quantized convolution (Q-Conv):
y/Q QConn St
+ bbQ Qonn
(24)
where denotes input binary spike, Wicoon and beccoms are quantized weights and bias of the Q-Conv layer. yq denotes the output of the Q-Conv layer. After employing MPRF, the rectified output should be computed as Eq. 25
8/ -(y() = @(weco. St + b)ccoe); (wl Woocom
St + bb Qcoo.)) p(( pl(QQoom Quana
St + b&com))
of + a
o ((w Qcaan
St + blgame)
gf. (we Qconv o (WQQomm
St + by bccoe)
of =M(w Qcons
Se be bocom
+ a
(25)
Se + bocom
a aW St + by blcc
gt. w WQccms
b b&ccue H( Qcom
S + by bocmmo)
S 1
+ +aa
aW Qconv
St by
bcco
o a St by
blaome)
- b,,
where w and by denote the fusioned weight and bias:
w Wonn
w alw! Qconn
S by 2co
(26)
gt. by bécc u(w e
S + bycom)
bf
Qcone
+ a.
al S
#+bb blccone
E THEORETICAL ENERGY CONSUMPTION ANALYSIS
When disregarding the energy consumption factors related to hardware manufacturing processes, data access, and storage, comparing the computational energy consumption of different models remains
19

---

Published as a conference paper at ICLR 2025
compelling. Such comparisons effectively reflect the intrinsic computational efficiency of various network models. Previous work by (Horowitz, 2014) indicates that, on 45nm process hardware platform, the energy consumption for single multiply accummultee (MAC) operation is 4.6pJ (with 3.7pJ for multiplication and 0.9pJ for addition). Many performance analyses in the research of spiking neural networks (SNNs) (Panda al., 2020; Qiu et al., 2024a; Shan et al., 2024) also reference this data.
E.1 COMPARISION ON MHSA AND SDSA
Given float-point input sequence E NXDD, the float-point Query (q). Key (k), and Value (v) in IN x x are computed using three learnable linear matrices, where N is the token number, and D is the channel dimension. The MHSA scaled dot-product computed as described by (Dosovitskiy et al., 2020):
qk T
MHSA(q, k, v) = softmax
(27)
vd
where d = Ā7 is the feature dimension of one head and H is the number of heads, and v serves as the scaling factor. Typically, MHSA divides q, k and v into heads along the channel dimension. For the jth head, qt. ki and V; in NR After performing the self-attention operation each of the H heads independently, the outputs are concatenated together.
In MHSA, q and k are matrrx-muulipliedd followed by matrix multiplication of their output with v. The computational complexity of MHSA(-) is O(N²D), indicating quadratic relationship with the token number N. For the SDSA modules, the computational complexity is D²), and the energy cost of the Rep-Conv part is consistent with SNN-based convolution. The energy cost of the SDSA operator part is given Table 7.
Table 7: Theoretical FLOPs/SOPs of self-attention modules.
Multi-head Self-attention (MHSA)
Spike-driven Self-attention (SDSA)
Function k, [(g, k v) Scale
MHSA k, v) softmax
gk" vd
SDSA(gs k2, ks) -S& T V)vs) T fri 3 FL T fr2 N
3N
2N2
N²
Softmax Linear
2N
FLfe
T T-fr FL
E.2 THEORETICAL ENERGY CONSUMPTION OF QSD-TRANSFORMER
We first calculate the theoretical energy consumption requires calculating the synaptic operations (SOPs):
SOPs' = fr² x T FLOPS
(28)
where fr', FLOPs', and 7' is the firing rate, float-pointing operations, and timestep of layer f. More- over, the respective number of FLOPs adds ( 16 4) of the number of [2.3,4]-bit multiplications equals the OPs following (Liu et al., 2020; Qin et al., 2020).
The total energy consumption of the network can be calculated using Eq. 29 for non non-quantized models and Eq. 30 for quantized models:
Evotal EMAC FLOPs! + EAC T
FLOPs
sone fr" + Σ FLOPs frr",, (29)
n=2
Eroua EMAC FLOPs1 cone
EAC
SOPS N
Σ
SOPS
(30)
where and M are the total number of Conv and FC layers, EMAC and EAC are the energy costs of MAC and AC operations, and fm fr", FL Soro and FL.OPs the firing rate and FLOPs
20

---

Published as a conference paper at ICLR 2025
of the n-th Conv and m-th FC layer. Previous SNN works (Horowitz, 2014; Rathi & Roy, 2021; Hu et al., 2024a) assume 32-bit floating point implementation in 45nm technology, where EMAC 4.6pJ and EAc 0.9pJ for various operations.
F LIMITATIONS AND FUTURE WORKS
Limitations The limitations of this work include the scalability of low-bit spike-driven Transformer models and the hardware deployment, which we will address in future research. The experimental results presented in this paper are reproducible. Detailed explanations of model training and configu- ration are provided the main text and supplemented in the appendix. Our codes and models will be made available on GitHub after review.
Future Works Since the largest Spikformer-V2 Zhou et al. (2024c) model has not yet released its training code and weights, we will attempt to quantify the Spikformer-V2 model in the future to demonstrate the scalability of our approach. Moreover, we did not take the energy consumption of memory access into account when calculating the thcoretical energy consumption owing the diversity of different dataflow and memory access schemes and the implementation on various hardware platforms. We will deploy our lightweight model into hardware platforms such as Field Programmable Gate Arrays (FPGAs) evaluate the factual performance, where we will optimize the suitable rea--wrtte data streams and memory access schemes to enhance the inference speed of models.
G EXPERIMENT DETAILS
G.1 IMAGENET- IK EXPERIMENTS
ImageNet- 1K dataset is commonly used for computer vision tasks. spans 1000 object classes and contains around 1.3 million training images and 50,000 validation images. For experiments the ImageNet dataset, we used the syper-parameters shown in Table Moreover, we employ our model three different scales, with the specific model configurations detailed in Table 9. We conducted training on eight 40GB A100 GPUs. For the three different model scales- -18MM, 3.8M and 6.8M parameters- allocated 24, 28 and 36 hours of training time, respectively.
Table 8: Hyper Ayreee-parameters for image classification ImageNet- IK and CIFAR10/100.
Hyper-parameter
ImageNet
CIFARTO/10
Timestep (Trainigg/mnerennee
1/4
1/4
Epochs
300
100
Resolution
224x224
128x 128
Batch size
1568
256
Optimizer
LAMB
LAMB
Base Learning rate
6e-4
6e-4
Learning rate decay
Cosine
Cosine
Warmup eopchs
None
Weight decay
0.05
0.05
Rand Augment
9/0.5
9/0.5
Mixup
None
0.8
Cutmix
None
1.0
Label smoothing
0.1
None
G.2 COCO EXPERIMENTS
The COCO dataset aims scene understanding, primarily extracted from complex everyday scenes, where objects in images are precisely localized through accurate segmentation. The COCO dataset comprises 118K training images and 5K validation images. In the COCO experiments, we pre-trained the TrS-ffooffrrm on ImageNet- 1k as the backbone, and then fine-tuned on the COCO dataset for 24 epochs with the Mask R-CNN detector obtain the final model. During the fine-tuning
21

---

Published as a conference paper at ICLR 2025
Table 9: Configurations of different QSD-Transformer models, which is similar our strong baseline Spike-driven Transformer v2 (Yao et al., 2023a).
stage
# Tokens
Layer Specification
1.8M
3.8M
6.8M
Conv
7x7 stride 2
Downsampling
Dim
32
48
64
H
W x
DWConv
7x7 stride
SepConv
2
2
Conv-based SNN block
FC ratio
2
Conv
3x3 stride
Channel Conv
Conv ratio
4
Conv
3x3 stride 2
Downsampling
Dim
64
96
128
H
W
DWConv
7x7 stride
4
X
4
Conv-based SNN block
SepConv
FC ratio
Conv
3x3 stride
Channel Conv
Conv ratio
Conv
3x3 stride 2
Downsampling
Dim
128
192
256
DWConv
7x7 stride
H X 8
W
SepConv
FC ratio
8
Conv-based SNN block
Conv
3x3 stride 4
Channel Conv
Conv ratio
Blocks
Conv
3x3 stride 2
Downsampling
Dim
256
384
512
H
W
SDSA
RepConv
3x3 stride
16 16
Transformer-based
Channel FC
FC ratio
SNN block
Blocks
Conv
3x3 stride I
Downsampling
Dim
360
480
640
H
W
4
16 16
X
Transformer-
Channel
SDSA
RepConv
FC ratio
3x3 stride
SNN block
Blocks
2
stage, we resized and cropped the training and test data to 1333x800, Additionally, we applied random horizontal flipping and resize with ratio of 0.5 to the training data. The batch size set to 12. We used the AdamW optimizer with an initial learning rate of lc-4, and the learning rate was decayed polynomially with a power of 0.9. We conducted training on four 40GB A100 GPUs for duration of 26 hours.
G.3 ADE20K EXPERIMENTS
The ADE20K semantic segmentation dataset comprises over 20K training and 2K validation scene- centric images meticulously annotated with pixel-level object and object parts labels, fostering comprehensive understanding of complex scenes. It encompasses a total of 150 semantic categories, encompassing elements such sky, road, and grass, well discrete entities like person, car, and bed. We also used the SSSD-Transformer pre-trained on ImageNet- IK as the backbone combined with FPN for segmentation experiments. The newly added parameters were initialized using Xavier initialization, and the model was trained on the ADE20K dataset with batch size of 20 for 160K iterations. We utilized the AdamW optimizer with an initial learning rate of x 10-4, 4 and the learning rate was decayed polynomially with power of 0.9. During the initial 1500 iterations, we employed linear decay to warm up the model. The training process was executed on four 40GB A100 GPUs and lasted for 25 hours.
G.4 TRANSFER LEARNING
We performed transfer learning experiments on the static image classification datasets CIFARI0/100 and the neuromorphic classification dataset CIFAR10-DVS The CIFAR10/ datasets each have
22

---

Published as a conference paper at ICLR 2025
50,000 training and 10,000 test images with resolution of 32 32. CIFARIO-DVS consists of 10K event streams created by capturing CIFARIO images using DVS camera.
In these experiments, first loaded pre-trained ImageNet- IK checkpoints and replaced the final fully connected layer to match the number of classes each dataset (c.g., replacing the 1000-FC with 100-FC for CIFAR-100) During fine-tuning, we applied data augmentations like mixup, cutmix, and label smoothing. We used a batch size of 128, the Adam optimizer with a weight decay of 0.01, and cosine-decay learning rate schedule starting at 10 over 100 epochs. The experiments were run on a single 32GB V100 GPU, taking 12 hours for CIFAR-I 10 and CIFAR-100, and 10 hours for CIFAR- 1O-DS...
For LIFARIO-DVS, we added preprocessing steps: dividing the event stream into '" slices, each with an equal number events, and compressing these into three-channel frames representing positive, negative, and all events, transforming the event stream into T frames. We also applied data augmentation to the processed event data, as described in (Wang al., 2023; Shi al., 2024).
23

---

Published as a conference paper at ICLR 2025
QSD-Transformer (16.9M)
GT
Image
Figure 4: Visualization of results on COCO dataset. Our QSD-Transformer excels the target detection task.
24

---

Published as a conference paper at ICLR 2025
QSD-Transformer(333M
GT
Image
streetlight
streetlight
nignboard
signboard!
builiing!
buildiing
Jperson
pola
eencel
bicyyolon
pol
food
FlD
Galling
curtain
Hahht
curtain
walli
pointing
niaing
pointing
Bill
toble
boaket
Billow
susshoon
table
bad
bed
BiSH
signboard
sttretligtt
building
butlcctna
tree
bancon
streetlight
person
celling
Geilin
Sconce
SCOnTeC
painting
painting
che
choir
pillow
pilttoni
bed
Dor 2013
tree
OK
person
person
streetlight:
streetlight
flag
flag
bratidiinga
Binnboard
Trode
buildina
Bigin book
wall!
person
CDF
CDr
Figure Visualization of results on ADE20K dataset. Our QSD-Traanorreer excels in the segmenta- tion task.
25


Reviews:
[{"review_id": "WlvrnBPVU8", "reviewer": "Reviewer_xmWz", "summary": "This paper presents a Quantized Spike-Driven Transformer, addressing the spike information distortion during quantization caused by the bimodal distribution of Quantized Spike-Driven Self-Attention (Q-SDSA). A bi-level optimization strategy is introduced, incorporating information-enhanced LIF and fine-grained distillation to rectify the distribution of Q-SDSA.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper aims to address the problem of performance degradation of the Spike Transformer after quantization, attributing it to the spiking information distortion (SID) problem. The authors presents a loss function based on mutual information maximization to tackle the problem.\n2. The authors conduct experiments across various tasks to demonstrate the effectiveness of the proposed method.\n3. The paper is well organized and easy to follow.", "weaknesses": "1. The primary reason for the improved quantization performance of the proposed method is the use of multi-bit spikes instead of traditional 0-1 spikes. Specifically, the implementation extends to 4 virtual timesteps, which inevitably reduces the training and inference efficiency. However, the manuscript does not provide a detailed explanation or analysis of this trade-off, which would be beneficial for understanding the overall impact on efficiency.\n2. The empirical comparison can be done in a more thoroughly by comparing with other latest state-of-the-art methods.\n3. Some content in the paper seems unnecessary, such as Appendix A, which does not contribute significantly to the main arguments or findings and could be omitted for conciseness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u0o4gumeUt", "reviewer": "Reviewer_pkTr", "summary": "This paper presents QSD-Transformer, a quantized spike-driven transformer that addresses the challenge of implementing efficient spiking neural networks (SNNs) while maintaining performance. The work shows three key points: (1) a lightweight quantized baseline for spike-driven transformers, (2) a bi-level optimization strategy to address spike information distortion (SID), and (3) information-enhanced LIF (IE-LIF) neurons with fine-grained distillation for improved performance.", "questions": "1. To make the ImageNet results table solid, authors can add additional results such as QKFormer [1] and previous QAT-ViT models [2].\n2. I just wondered why only small sizes of architecture (1.8M, 3.9M, 6.8M) are used for training the ImageNet dataset. Is there any scalability issue with this method?\n3. This work uses multi-bit spikes during training and knowledge distillation with ANN architecture, which causes huge training overhead in training time and memory. Can you present any analysis of this overhead?\n4. In the transfer learning section, the overall information is insufficient. Which bit-width did you use? and could you provide us the accuracy of CIFAR10/100, and CIFAR10-DVS without transfer learning?\n5. Can the authors provide the firing rate information? Compared to the original Spike-driven Transformer-V2, how has the firing rate changed in the self-attention part?\n\n[1] Zhou, Chenlin, et al. \"QKFormer: Hierarchical Spiking Transformer using QK Attention.\" arXiv preprint arXiv:2403.16552 (2024).\n[2] Li, Yanjing, et al. \"Q-vit: Accurate and fully quantized low-bit vision transformer.\" Advances in neural information processing systems 35 (2022): 34451-34463.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- First approach to quantizing spike-based transformers with SID analysis\n- Solid theoretical analysis and proofs for the proposed methods\n- Competitive accuracy on ImageNet (80.3%) with low energy\n- Extensive experiments on multiple tasks (classification, object detection, segmentation)", "weaknesses": "- Lack of comparison with the previous SNN and quantized ANN transformer models (Connected to question #1)\n- Limited scalability in ImageNet experiments (Connected to question #2)\n- Huge training overhead compared to the conventional spike-based transformer due to multi-bit spike and knowledge distillation (Connected to question #3)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hGPlXH7pEJ", "reviewer": "Reviewer_iFNE", "summary": "To tackle the issues of high-parameter spike-based Transformer in resource-constrained applications and the low performance of directly quantized spike-based Transformers, this paper introduces a Quantized Spike-driven Transformer. It uses a bi-level optimization strategy, including an Information-Enhanced LIF neuron and fine-grained distillation, to counteract quantization-induced performance degradation. The comparative and ablation experiments demonstrate the effectiveness of the proposed methods.", "questions": "See Weakness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Based on Information entropy, this paper proposes a bi-level optimization strategy, which mitigates quantization-induced performance drops in baseline quantized spiking Transformers. \n\n2. The proposed IE-LIF spike neuron is hardware-friendly and converts to a binary-spiking LIF neuron during inference to maintain spike-driven nature.\n\n3. Experimental results on ImageNet and a large number of vision tasks (detection, segmentation, and transfer learning) show that the method is effective and energy-efficient on various spike-based transformers.", "weaknesses": "1. The authors describe the implementation method of surrogate gradients for binary spikes in the appendix. However, the proposed IE-LIF in the main text is multi-bit. Could the authors explain how the aforementioned surrogate gradients are deployed in the proposed neurons?\n\n2. Fast-SNN [1] converts quantized ANNs to SNNs and is a competitive ANN-to-SNN method. Like this paper, it aims to reduce quantization error and employs quantization techniques to enhance energy efficiency. The basic idea of Fast-SNN is to reduce both quantization error and accumulating error, achieving excellent performance on many vision tasks (classification, detection, and segmentation). Could you include comparative experimental results with Fast-SNN?\n\n3. There are some typos in the text, such as the equation (1) on line 164  and on line 292, it seems you intended to reference Fig 1(b). \n\n[1] Hu, Yangfan, et al. \"Fast-SNN: fast spiking neural network by converting quantized ANN.\" IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 12, pp. 14546-14562, Dec. 2023, doi: 10.1109/TPAMI.2023.3275769.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l7dU8RBRXA", "reviewer": "Reviewer_Uosk", "summary": "This paper proposes a quantized spike-driven transformer and increases the accuracy of the SNN by proposing an information-enhanced LIF model and fine-grained distillation from the lower level and upper level respectively. Experiments show that these technologies reduce energy consumption significantly while increasing the accuracy of SNN.", "questions": "1. Why do the IE-LIF neuron and fine-grained distillation that reduce the energy consumption? Do these technologies reduce the number of synaptic operations?\n2. Why the energy reduction in the COCO2017 dataset is not significant?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The IE-LIF neuron combines the conversion algorithm and training algorithm, which is novel.\n3. The experimental results are significant.", "weaknesses": "1. The training of IE-LIF neurons does not utilize temporal information, which is not suitable for temporal benchmarks.\n2. The reason why the IE-LIF neuron and fine-grained distillation that reduces the energy consumption is not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Znr7YU5mHb", "reviewer": "Reviewer_ynnK", "summary": "The author proposed Quantized Spike-Driven Transformer(QSD-Transformer) to tackle with the spike information distortion (SID) challenge resulted from quantized spike-driven self-attention (Q-SDSA). The author addressed the problems through two levels: 1) Information-Enhanced LIF(IE-LIF) neuron to rectify the information distribution in Q-SDSA at the lower level. 2) A fine- grained distillation scheme for the QSD-Transformer to align the distribution in Q-SDSA with that in the counterpart ANN at the upper level. QSD-Transformer achieved promising results on multiple computer vision tasks.", "questions": "1. I wonder whether you quantized the membrane potential or not. If you didn't quantize the membrane potential, it seems hard to implement your method on hardware.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extensive experiments and ablation studies on Image Classification, Object Detection and Semantic Segmentation.\n2. The proposed Information-Enhanced LIF(IE-LIF) neuron is effective to rectify the information distribution in Q-SDSA through Information Theory.\n3. Clear writing and methodology.", "weaknesses": "1. The comparison between ANN2SNN and Direct Training methods is limited. Currently, MST is no longer the SOTA method on ANN2SNN method. SpikeZIP-TF (ICML 2024) [1] and ECMT (ACM MM 2024) achieve better performance on Image Classification tasks. The performance of  SpikeZIP-TF and ECMT on ImageNet surpass QSD-Transformer by a large margin. In addition, ANN2SNN methods has advantage on saving computational resources compared with Direct Training methods. It is recommended that the author should conduct a more comprehensive comparison between those two methods.\n2. The method proposed by the author is somewhat cumbersome, did the training time provided by the authors in appendix include the training time of FGD?\n3.  It is recommended that the author should extend the method to NLP tasks to verify the transferability of QSD-Transformer.\n\n[1] Kang You*, Zekai Xu* et al. SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN. International Conference on Machine Learning 2024\n[2] Zihan Huang, Xinyu Shi et al. Towards High-performance Spiking Transformers from ANN to SNN Conversion. ACM Multimedia 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)