QKFormer: Hierarchical Spiking Transformer using Q-K Attention
Authors: Chenlin Zhou
Han Zhang
Zhaokun Zhou
Liutao Yu
Liwei Huang
Xiaopeng Fan
Li Yuan
Zhengyu Ma
Huihui Zhou
Yonghong Tian
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Spiking Neural Network; Neuromorphic Computing; Event-driven; Transformer; Spatio-temporal
Primary area: neuroscience_and_cognitive_science

Abstract: Spiking Transformers, which integrate Spiking Neural Networks (SNNs) with Transformer architectures, have attracted significant attention due to their potential for low energy consumption and high performance. However, there remains a substantial gap in performance between SNNs and Artificial Neural Networks (ANNs). To narrow this gap, we have developed QKFormer, a direct training spiking transformer with the following features: i) _Linear complexity and high energy efficiency_, the novel spike-form Q-K attention module efficiently models the token or channel attention through binary vectors and enables the construction of larger models. ii) _Multi-scale spiking representation_, achieved by a hierarchical structure with the different numbers of tokens across blocks. iii) _Spiking Patch Embedding with Deformed Shortcut (SPEDS)_, enhances spiking information transmission and integration, thus improving overall performance. It is shown that QKFormer achieves significantly superior performance over existing state-of-the-art SNN models on various mainstream datasets. Notably, with comparable size to Spikformer (66.34 M, 74.81\%), QKFormer (64.96 M) achieves a groundbreaking top-1 accuracy of **85.65\%** on ImageNet-1k, substantially outperforming Spikformer by **10.84\%**. To our best knowledge, this is the first time that directly training SNNs have exceeded 85\% accuracy on ImageNet-1K.

QFFomeer: Hierarchical Spiking Transformer using Q-K Attention
Chenlin Zhou 1x
Han Zhang 12+
Zhaokun Zhou 1.34
Liutao Yu!,
Liwei Huang 1.3
Xiaopeng Fan
Li Yuan 1.3
Zhengyu Ma Huihui Zhou
Yonghong Tian 1.3
Pengcheng Laboratory
Harbin Institute of Technology
Speking University
Abstract
Spiking Transformers, which integrate Spiking Neural Networks (SNNs) with Transformer architectures, have attracted significant attention due to their potential for low energy consumption and high performance. However, there remains a substantial gap in performance between SNNs and Artificial Neural Networks (ANNs). To narrow this gap, we have developed QKFormer, direct training spik- ing transformer with the following features: i) Linear complexity and high energy efficiincyy. the novel spike-form Q-K attention module efficiently models the token channel attention through binary vectors and enables the construction of larger models. ii) Multi-scale spiking representation, achieved by hierarchical structure with the different number of tokens across blocks. iii) Spiking Patch Embedding with Deformed Shortcut (SPEDS), enhances spiking information transmission and integration, thus improving overall performance. It is shown that QKFormer achieves significantly superior performance over existing state-of-the-art SNN models various mainstream datasets. Notably, with comparable size to Spik- former (66.34 M, 74.81 1%). QKFormer (64.96 M) achieves a groundbreaking top-1 accuracy of 85.65% on ImageNet-Ik, substantially outperforming Spikformer by 10.84 %. To our best knowledge, this is the first time that directly training SNNs have exceeded 85% accuracy ImageNet- The code and models are available at http:///tthbbbcoo///ccccccnlllld
Introduction
Regarded as the third generation of neural networks [1] the brain- brain-inspired Spiking Neural Networks (SNNs) are potential competitors to Artificial Neural Networks (ANNs) due their high biological plausibility and high energy efficiency attributed to their event-driven properties (2). Transformer, originally designed for natural language processing 3 has flourished various computer vision tasks, including image classification 5 object detection 6
7
g and semantic segmentation
15), which integrate spiking neural networks with transformer architecture, have attracted significant attention. This innovative combination provides great potential develop advanced AI algorithms with high performance and low energy consumption.
I0 Spiking Transformers (Transformer-be based SNNs) 1I 12
As the architecture of the transformers essential the model's performance designing new architectures for transformer-basee SNNs is quite challenging in terms of space requirements for the following reasons [3] 15). i). Spiking Self Attention (SSA) [TT), the core module of spiking transformers, encodes Query, Key, and Value with sparse spikes. However, the
16 g 10],
Egqual
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Newinnn
NEW sareration
Colamm survevaiin
Ani Tolan Allartion Vedby
Aua
Splice
Xew
Chural::
Falko (ulgat 1'
(a) Q-K Token Attention (QKTA)
(b) Q-K Channel Attention (QKCA)
Figure Illustration of Q-K attention with the two versions of Q-K token attention (QKTA) and Q-K channel attention (QKCA). The inputs are binary spikes and there are only sparse additions and mask operations in Q-K attention. As a spike-driven module, Q-K attention efficiently models the token or channel attention through spike-form binary vectors, performing linear complexity to #tokens (or #channels) and high energy efficiency. Spiking Neuron (SN) in this work adopts the Leaky-Integrate-and and--i--nd-ree- (LIF) model, which is shown in Appendix. 7.1
computational complexity (especially space complexity) of SSA scales quadratically to the number of tokens (#tokens), and is the main obstacle to explore architecture that incorporate multi-level features. ii). SNNs process data across the time domain, necessitating a high level of computational and memory resources. This combination leads to considerable consumption of computational resources, making the training process highly demanding in terms of both memory and processing power.
To address these issues, we propose QKFormer with three innovations. i) Q-K attention with linear complexity and high energy efficiency. ii) A hierarchical architecture with decreasing number of tokens across blocks. iii) A novel patch embedding with deformed shorteut module. The linear complexity of Q-K attention is originated from the binary spike- form vector attention. This design lower the energy consumption and the space requirement. The hierarchical architecture starts from small patches and gradually merges neighboring patches in deeper spiking transformer layers with gradually decreasing #tokens, which enables multi-level spiking feature representation and benefits the model performance. The patch embedding with deformed shortcut facilitates spiking information transmission and integration. These merits make QKFormer achieve kate-of-thhcart performance in the SNN domain, in contrast to the previous transformer-baat based SNNs with spiking feature maps of a single resolution. Our main contributions are follows:
1) We develop a novel spike-form Q-K attention mechanism, tailor-made for the spatio-temporal spiking patterns of SNNs, which can easily model the importance of token or channel dimensions with binary values. The Q-K attention has linear complexity #tokens (or #channels) and only adopts two spike-form components: Query (Q) and Key (K).
2) We design a versatile and powerful Spiking Patch Embedding with Deformed Shorteut (SPEDS) module, which enhances spiking information transmission and integration thus improving the perfor- mance of spiking transformers significantly.
3) We build a direct-training hierarchical spiking transformer with different number of tokens across blocks, incorporating Q-K attention and SPEDS, named QKFormer. This marks the effective exploration of hierarchical spiking representation in Transformer based SNNs.
4) Extensive experiments show that the proposed model outperforms the state-of of-the-ar art (SOTA) SNNs on various static and neuromorphic datasets. Notably, QKFormer has achieved a significant milestone, surpassing 85% top-1 accuracy on ImageNet with 4 time steps using the direct training approach for the first time.
2 Related Work
Learning Methods of Spiking Neural Networks. At present, there are mainly two ways to obtain trained SNNs. One involves converting pre-trained ANNs to SNNs (ANN2SNN) [17 18 19 20
21 22 23), replacing the ReLU activation function ANN with spiking neurons. However, This converted SNN suffers from long converting time steps and constraints on the original ANN design.

---

Another method is to directly train SNNs[24) using surrogate gradient [25] 26 27| 28 to address the non-differentiabiliyy of spike excitation function during backpropagation. The direct training method has received more attention due to its low latency and supporting flexible architectural exploration.
Direct Trained SNN Models. 28 proposed the Spike- Brike--lleeettWeee Wise block, which further addressed gradient explosion and gradient vanishing problems, and prolonged the directly trained SNNs beyond a depth of 100 layers with 69.26% accuracy on ImageNet-Ik, Spikformer [ designed a novel spike- form self-attention named Spiking Self Attention (SSA), using sparse spike-form Query, Key, and Value without softmax operation, which was used to construct the Spikformer. Spikformer achieved 74.81% 1% accuracy on ImageNet- Ik with 4 time steps, showing the great potential of Transformer-basee SNNs for the first time. Spikingformer [12] modified Spikformer with pre-activation shortcut, which can avoid the floating-poit point multiplications in synaptic computing and has a lower firing rate. Spikingformer achieved 75.85% accuracy on ImageNet-Ik Ik. 13 designed a novel Spike-Driven Self- Attention (SDSA), which used only masks and addition operations without any multiplication, thus significantly reducing the computation energy compared to the vanilla self-attention. In addition, the proposed Spike-driven Transformer based on SDSA has achieved 77.07% on ImageNet-IK Ik. However, all of these SNN models above remain large performance gap compared with ANN.
3 Method
3.1 Preliminary
Vanilla Self Attention. Vanilla self-attention (VSA) [ in transformers has three Doating-point key components: query (Q,)), key (KF), value (Vx) which are calculated by learnable linear matrics
and input X. The calculation of VSA can be formulated as follows:
Q,, KF, V = X(Wo: WK, Wv),
Q-KF
VSA (Qr, F V>) = Softmax
VF.
(2)
Vd
where F denotes the floating-poit form. Both floating-point matrix multiplication and softmax operation which contains exponent calculation and division, do not align with the properties of SNNs.
Spiking Self Attention. Spikformer [1I] demonstrated a novel apike-form self-attention named Spiking Self Attention (SSA), using sparse spike-form Q, K. V without softmax operation and floating-poin -oint matrix multiplication. The calculation process of SSA is formulated follows:
I= SSNN, (BN, (X(Wr))), I (Q, V),
SSA' K, V) = SN (QK"V**)) *s),
where Q,K, V E RTxNxD the spike-form Q, K. V are computed by learnable linear layers. $ is scaling factor. SN means spiking neuron layer. The calculation of SSA avoids floating-point multiplication, meeting the property of SNNs.
3.2 Q-K Attention
An overview of Q-K attention is shown in Figure I Both VSA and SSA use three key components and have ( or (NN computational complexity, while our proposed Q-K Attention which has linear complexity and only uses two spike-form components: Q and K, which are produced through learnable linear matries.
(5) where X is the input spiking map. According to the detailed calculation mechanism of Q, K. Q-K Attention can be divided into Q-K Token Attention (QKTA) and Q-K Channel Attention (QKCA).
Q = SNQ (BN (XW Q)) K SNK (BN (XWx));
Q-K Token Attention. We here assume = 1 and single head attention for mathematical description. After obtaining spike-form Q, K E RxXNx both Q and K can be formed as a spike matrix N x D (N is the token number, D is the channel number). QKTA process can be formulated as follows:
A. = SN Σ Qui):
X' A K,
(6)
1=0
3

---

Table 1: Computational complexity comparison. N is the token number, D is the channel number.
Methods
VSA
SSA II
SDSA 13
QKTA
QKCA
Time complexity
O(N2D) O(N + ND)
O(N2D) O(N + ND)
O(N O(ND)
O(D)
O(N) O(D)
Space complexity
O(N)
where A, is the N + 1 token attention vector, which models the binary importance of different tokens. A, is a spike-form vector. which is obtained by addition operations (row summation) of Q spike matrix and a following spiking neuron. @ is the Hadamard product between spike tensors, which is equivalent to the mask operation. We apply the spike- forme-form token attention vector A, to the K spike matrix through the column mask operation (token mask), to obtain the output X' of QKTA.
Q-K Channel Attention. The calculation process of Q-K channel attention is similar to the previous Q-K token attention, and can be formulated as : A. SN(E)- Quy). X' A. & K, where A is the 1 * D channel attention vector, which models the binary importance of different channels. A, is spike-form vector, which is obtained by addition operations (column summation) of Q spike matrix and following spiking neuron. Then, the output X' of Q-K Channel Attention is obtained by the row mask operation (channel mask) between A, and K.
X" = SN (BN (Linear (X')).
As shown in 7ormula post-linear layer is also required after obtaining X' of Q-K Token or Channel Attention. In addition, the channel dimension is D/h in the multi-head Q-K attention, where h is the head number. In this work, the spiking neuron uses the LIF model (28). Same with [T]. time step T' an independent dimension for the spiking neuron layer. In other layers, is merged with the batch size. We exploit QKTA in our experiments by default.
Linear Computational Complexity of Q-K Attention. As shown in Table the time complexity of Q-K attention varies based on the implementation approach. Specifically, when utilizing spike-form broadcasted element-wise multiplication, 8, the time complexity can reach up to O(N D). When applying mask operation, the time complexity of Q-K attention is only O(N) or O(D). The space complexity of Q-K attention with the whole process is O(N * D) at most, which is caused by the self-storage consumption Q and K matrix. In terms of the space complexity of attention operation, Q-K attention only requires an extra 1 * D or N * space to store the attention vector with the space complexity of O(N) or ODD). The linear complexity of Q-K attention makes it possible to successfully explore the harge-scale hierarchical architecture SNN model.
High Energy Efficiency of Q-K Attention. As a driven attention module, the linear multiplica- tion is transformed into sparse addition. Mask operation can be implemented on neuromorphic chips through addressing algorithms (29 or AND logic operations (30 with negligible power consumption. Compared with SSA, Q-K attention is much more
which comes from the following reasons: i) Q-K attention only adopts two spike-form components for spike [0, 1] operation without the input and thus has less synaptic computing. ii) Q-K attention has much fewer spiking matrix operations due to its linear complexity of O(N) O(D). iii) Q-K attention discards the scale operation of SSA, which leads to reduced power consumption further.
3.3 No Scaling Factors in Q-K Attention
In VSA 33, assume that q: (q: E R_ Q E Rmxxd) and k, (k, E R1xdd K E Rmmxd) are indepen- dent random variables with mean of O and variance of I, then each element in the product of QK' has mean 0 and variance d. The variance magnitude of QK" grows with the embedding dimension d, which can result in gradient vanishing issues after softmax operation. Therefore. The product of matrices Q and K in VSA [3] is scaled by a factor 1 in Eq. 2 to normalize the product to variance
va
1. Though the softmax function is not adopted due to its non-spike operations (division, exponential operation) in SNNs, SSA-based [II SNNs will suffer obvious performance degradation even cannot converge without scaling because the variance of QKTV output is too large (Assuming that all the spiking elements are independent random variables and subject to Bernoulli Distribution). However, Q-K attention can discard scaling operations thus reducing power consumption because the variance of Q-K attention is much smaller than SSA (e-g. the max theoretical variance of Q-K token attention is only about 1 / 200 of SSA). The detailed analysis can be found in the Appendix 7.2 and Section 4.3

---

3.4 QKFormer
As the computational complexity (especially space complexity) of SSA quadratic to #tokens, previous direct training spiking transformers are all limited to straight-through structures. Combining SSA with hierarchical architecture directly will lead to memory explosion easily when training spiking transformers. To overcome these issues, proposed hierarchical spiking transformer based on Q-K attention, named QKFormer, which constructs hicrarchical spiking feature maps with linear computational complexity to #tokens #channels.
Overall Hierarchical Architecture. The overview of QKFormer is presented in Fig- ure 2
The input form can be formulated as (To x H W x n). In static RGB image datasets, To and n = 3. In temporal neuromorphic datasets, the input To = T, while = 2. In our implementation, we use a patch size of 4 x and thus the input feature dimension (4 4 x n) of each patch is projected into a spike-form arbi- trary dimension (denoted as C') in Spiking Patch Embedding with Deformed Shortcut 1 (SPEDS-1) 1), which together with the following QKFormer blocks are referred as "Stage 1". The number of tokens in Satge 1 is x ). To produce hierarchical spiking representation, the number of tokens is reduced in SPEDS-2 and SPEDS-3 as the network goes deeper. Both SPEDS-2 and SPEDS-3 reduce the number of tokens by a patch size of 2 x 2 (2x downsampling of resolution), and the number of channels is transformed into 2C and 4C, respectively. We denote the SPEDS-2 and the following QKFormer blocks as "Stage 2", which reduces the number of tokens ( HH W While SPEDS-3 and the following Spikformer or QKormer 8 x 8
SPEDS-
ON Ansemin
1: 2
Figure 2: The overview of QKFormer, a hierarchi- cal spiking transformer with Q-K attention.
TE tokens. The number of spiking transformer blocks (QKFormer or Spikformer) in each stage are N1, N2, and N3, respectively. These stages jointly produce hierarchical spike-form representation.
H blocks are referred to "Stage 3" with 16
W
Mixed Spiking Attention Integration. In the former stage of a hierarchical architecture model, the number of channels is small while the number of tokens is large. In the latter stage, the channel number is large while the token number small. Thus leads to suboptimal performance when we only use single type of Q-K attention in hierarchical architecture model. Therefore, we use mixed spiking attention integration in QKFormer. QKTA is conducted the former stage in hierarchical architecture, and we could choose QKCA or SSA in the latter stage. In the subsequent experiments, we use SSA in the last stage of QKFormer and QKTA in the former stages by default.
QKFormer Blocks. Similar to the standard transformer encoder block, QKFormer block contains Q-K Attention module (QKTA or QKCA) and a Spiking MLP (SMLP) block, which can be formulated as follows:
X QKTA (X--1) + X,-I, X/ RTXNND
(8)
X; SMLP (X)) X, RTNNDD
At last, a fully connected layer is used the classifier behind the last block.
3.5 Spiking Patch Embedding with Deformed Shortcut.
Residual shorteuts in SNNs (8 can implement identity mapping, which reduces information loss (facilitates information transmission and integration) in spike communication, thus ensuring the network can be well-behaved in a depth-insensitive way. Previous spiking transformers [2 [3]
use the residual shortcuts to achieve identity mapping, mainly focusing on the spiking attention block and spiking MLP block, and lacking identity mapping in patch embedding across the downsampling block. The input and output of spiking patch embedding block in QKFormer have different channel and token numbers. To realize residual learning in spiking patch embedding, we can perform lightweight linear projection W in the shortcut connections to match the channel and token numbers, thus realizing the identity mapping cross downsampling blocks in spiking patch embedding. Given the input spiking map X, the process of patch embedding can be formulated as follows:
Y = FX, (W.)) + SN(W.X).
(10)
5

---

Table 2: Results on ImageNet-IK. Power calculated as the average theoretical energy consumption when predicting an image from ImageNet test set. The power data for QKFormer and ANNs is evaluated according to Appendix [7.6] and the power data for other works were obtained from related papers. "A2S" denotes "ANN-to-SNN" "HST-I-D" denotes "Hierarchical Spiking Transformer" with I. encoder blocks and channels. HST-10-768" and HST-10-768 means HST-10-766 with 288 and 384 input size for inference. The top-5 accuracy of QKFormer (HST-10-768" is 97.74%.
Methods
Type
Architecture
Input Size
Param
Power
Time Top-1
(M)
(mJ)
Step
Acc (%)
RMP 21
A2S
VGG-16
224
39.90
2048
73.09
QCFS 22
A2S
ResNet-18
224
11.70
1024
74.32
23
A2S
Swin Transforme--
224
28.50
512
78.51
SNN
SEW-ResNet-34
2242
21.79
4.89
67.04
SEW ResNet
SNN SEW-ResNet-101
224
44.55
8.91
4
68.76 76
SNN SEW-ResNet-152
224
60.19
12.89
4
69.26
SNN Spikformer-8-366
224
16.81
7.73
4
70.24
Spikformer 11
SNN Spikformer-8-512
224
29.68
11.58
4
73.38
SNN Spikformer-8-768
224
66.34
21.48
74.81
SNN Spikingfformer-8-888
224
16.81
4.69
72.45
Spikingformer 12
SNN Spikingformer--85517
224
29.68
7.46
74.79
SNN Spikingformer-8-7000
224
66.34
13.68
4
75.85
SNN S S-Trnsformer- 384
224
16.81
3.90
72.28
SNN S Transformer-8-55
224
29.68
1.13
1
71.68
S- S-Transformer
SNN S- Transformer-8--1
224
29.68
4.50
74.57
SNN $ Transfoorer-8-7766
2882
66.34
6.09
77.07
ViT
ANN ViT-B/16
3842
86.59
254.84
77.90
ANN DeiT-B
224
86.59
80.50
81.80
32
ANN
DeiT-B
3842
86.59
254.84
83.10
ANN Swin Transformer-B
224
87.77
70.84
83.50
Swin g
ANN
Swin Transformer-B
3842
87.77
216.20
84.50
SNN
HST-10-384
224
16.47
15.13
4
78.80
SNN HST-10-512
224
29.08
21.99
82.04
SNN HST-10-768
224
64.96
8.52
81.69
QKFormer
SNN HST-10-768
224
64.96
38.91
4
84.22
SNN HST-10-768*
288
64.96
64.27
85.25
SNN
HST-10-768**
3842
64.96
113.64
85.65
In this work, the deformed linear projection Wa is set lightweight convolutional layer with 1 1x kernel and stride > 1, to meet the channel and token numbers of the patch embedding block. MwNN-MaaPollinnSS--CCC
or The function F involved in this work is set as while more layers or more variants are possible. "Conv2D-BN-SN-Com
BN-MaxxPooling-SS1
There are mainly two types of residual shortcuts in deep SNNs. Formula I0] shows the patch embedding the way of activation sediiattionbbforeeeee before-addiiion
28 I1]. The other way of the patch embedding can be formulated as follows:
with the pre-activation residual shorteut [31 12 13
SN(G (X, (W₂)) W.X),
(11)
where the function & correspondingly could be set as Bony2D-BN-MaxPooligg SN--hi------onO-BBNN The intuitive representation of SPEDS is shown in
Appendix 7.4
In this work, the spiking patch embedding of stage 2 or stage 3 in QKFormer can be formulated as
Formula TO The spiking patch embedding stage uses an extra [Conv2D-BN-SN) for spiking encoding front of the block (Formula 10 to transform the non-spike input data into spikes.

---

Table 3; Comparision on CIFARIO, CIFARIOO, DVSI28, CIFARIO-DVS, "Param" denotes "Parame-
ter (M)", "Acc" denotes "Top-1 Aceuracy (%)", "T" denotes "Time Step".
CIFARIO
CIFARI00
DVS128
CIFARIO-DVS
Method
Param
T
Acc
Param
T
Acc
Param
T
Acc
Param T Acc
Spikformer [1]
9.32
95.51
9.32
4
78.21
2.57
16
98.3
2.57
16
80.9
Spikingformer 12
9.32
95.81
9.32
4
78.21
2.57
16
98.3
2.57
16
81.3
CML [1]
9.32
96.04
9.32
4
80.02
2.57
16
98.6
2.57
16 80.9
S-Transformer
13
10.28
95.60
10.28
4
78.4
2.57
16
99.3
2.57
16
80.0
ST5
1.99
16 98.7
1.99
16 79.93
ResNet- (ANN)
12.63
94.97
12.63
1
75.35
Trasnformer (ANN)
9.32
96.73
9.32
1
81.02
QKFormer
6.74
96.18
6.74
4
81.15
1.50
16 98.6
1.50
16 84.0
Experiments
4.1 Results on ImageNet-Ik Classification
Experimental Setup on ImageNet. In this experiment, we use A as the optimizer, which adopted with base learning rate of x 10-1, The actual learning rate was calculated as BatchSize/256 multiplied by the base learning rate. The batch size is set to 512, which is realized by accumulated gradient iterations (33] and distributed across 8 Nvidia V100 GPUs. We trained QKFormer for 200 epochs. In addition, following DeÎT (32), data augmentation techniques including RandAugment
random erasing (35). and stochastic depth [6) are employed in this study. The number of blocks in the three stages is set as {1, 2, 7) respectively.
Main Results on ImageNet. The experimental results demonstrate the superior performance of our proposed QK Former, surpassing previous works' performance by a large margin (Table[2). QKFormer (64.96 M) achieves 85.65% top-1 accuracy and 97.74% top-5 accuracy ImageNet. To begin with, we compare our model with the baseline spiking transformer (i.e., Spikformer [1].. Our QKFomeer models have slightly fewer parameters but much higher performance. For example, our QKFormer (64.96 M, 85.65% 65%) significantly outperforms Spikformer (66.34 M. 74.81 %) by 10.84%. In addition, compared with SDSA, our Q-K attention has lower computational complexity (shown in Table and our FK has much higher performance than S- Trannformmr (built by SDSA) [13]. In detail, OKFormer outperforms T-Tansffommer by 7.55%, 7.47%, and 8.58% respectively on three models with comparable #parameters. Finally, Our QKFormer outperforms the SOTA ANN-to-SNN model MST 23 by 14% and has much fewer time steps meanwhile. To our best knowledge, this the first time that a direct training SNN model has achieved an accuracy of over 85% on ImageNet-1k.
Comparing with ANN Models ImageNet. Our QKFormer is event-driven SNN model, whose output is in binary form (either 0 or 1), the multiplications of activations and weights can be transformed into sparse addition, thus enjoying high energy efficiency. It should be noted that hierarchical architecture will lead to the power increment of QKFormer. This is still very cost-effective compared with ANN models. For instance. QKFormer (64.96M, 85.65 %, SNN, 113.64mJ) Vs. Swin Transformer (88M, 84.5 %, ANN, 216.20mJ) 8 Vs. DeiT-B (86M, 83. 1 %,, ANN, 254.84mJ) [32 Vs. ViT (85.59M. 77.9%, ANN, 754.84mJ). 4. Under the same experiment conditions without pre- training extra training data, our QKFormer has surpassed the most well-known Transformer-based ANNs performance while maintaining high energy efficiency.
4.2 Results on CIFAR and Neuromorphie Datasets
CIFAR Classification. In this experiment, the OKFormer is trained for 400 epochs with batch size of 64 following previous works: Spikformer [II]. Spikingformer [12]. Following Spikformer, we use 4 blocks in QKFormer in total, which are distributed (1, 1, 2) three stages. Due to the hierarchical architecture design, our QKFormer model has only 6.74 M parameters in this case. The results on CIFAR datasets are shown Table 3 For CIFAR1O, our model achieved 96.18% 18% accuracy with 6.74 M parameters. Our proposed QKFormer outperforms Spikformer by 0.67% and reduces 2.58 M
7

---

Stage
A
20
40
60
00
100
20
55A
K
20
40
00
00
100
20 90
83
302
1000
X
150
323
(b)
Figure 3: The visualization and memory consumption of QKTA. is the visualization of Q-K token attention. The white dot means value I, while the black one means value [b] shows the comparison of memory costs between QKTA and SSA under different token numbers. N the token number.
Vertence S5A
Expectaaioo of
Expectitir
1109
130
(b)
Figure 4: (a) shows the variance and expectation of SSA, (b) shows the variance and expectation of QKTA. Assume that the spike elements (either 0 or 1) SSA and QKTA are independent random variables and subject to Bernoulli distribution.
parameters meanwhile. For CIFAR 100, our model achieved 81.15% with 6.74 M parameters. Our proposed QKFormer outperforms Spikformer by 2.94% and reduces 2.58 M parameters meanwhile.
Neuromorphic Classification. We compare our method with SOTA methods on both CIFARIO-DVS and DVS- datasets. In this experiment, We utilize mini QKFormer model with 1.50 M parameter, which has (0, 1, 1) blocks in three stages. The max patch embedding dimension is 256. The training process involves 200 epochs for DVS128 Gesture and 106 epochs for CIFAR 10-DVS. The number of time steps of the spiking neuron is 10 or 16. The experimental results of temporal neuromorphic classification are presented in Table For DVS128-Gesttre dataset, our model with 1.50 M parameters achieves 98.6% accuracy using 16 time steps and 98.3% accuracy using 10 time steps. For CIFARIO-DVS dataset, our model achieves 84.0% accuracy with only 1.50 M parameters using 16 time steps. Our proposed QKFormer significantly outperforms Spikformer by 3.1 % while reducing 1.07 M parameters. In addition, our model with 10 time steps achieves 83.8% accuracy, which outperforms Spikformer by 4.9 % and outperforms the SOTA model (Spikingformer) by 3.9%.
4.3 Analyses on Q-K Attention
Attention Visualization. In this part, we visualize the Q-K token attention (Stage and Stage of the QKFormer model) on ImageNet. As shown in Figure 3(a) A, is the N token attention vector, and X' the output of the attention process, which is obtained by the mask operation between matrix K and attention vector A₁. Specifically, the longitudinal axis denotes the channel index of one head, while the horizontal axis denotes the token index. The #tokens in stage and stage are 562 and 282, respectively. To facilitate visualization, we choose a continuous segment with length of 100 extracted from the whole token vector. The visualization shows Q-K attention can lead to high sparsity of spikes.

---

Memory Consumption. In this experiment, we compare the memory consumption between QKTA (Formula 6 and SSA 4 under different token numbers. We calculate the memory consump- tion of a QKTA and an SSA on GPU by forwarding the input tensor (T. B, C, .). To facilitate the statistics of the impact #tokens N on memory consumption, the #channels C' is set to 256, and the time step 7 and batch size B are set to 1. The experiment result is shown in Figure ((b With the increment of #Tokens, SSA consumes much more GPU memory than QKTA, of which the complexity is linear to #Tokens, For example, SSA consumes about x0x GPU memory than QKTA
when VN= 50.
Spike Firing Rates in QKFormer Blocks, In this experiment, we calculate the spike firing rates of QKFormer blocks of the trained QK- Former (64.9M) on the ImageNet-1K K test set with the 224 x 224 input resolution. The aver- age spike firing rates of the QKFormer blocks in Stagel and Stage2 are shown in Table Note that the spike-form X' is obtained by column mask operation (token mask) between A, and K. In fact, the summation operation in the Q- K attention causes Q to become significantly sparser compared to K when the network converges. Specifically, Q in stage 1 has fire rate of 0.0432, while K has 0.1784. After the accumulation operation along D/h of the multi-head QKTA version, the LIF neuron (A,) exhibits typical averaged fire rate of 0.3477.
Table 4: Spike firing rates in QKFormer blocks.
QKFormer Block
Stagel (fr)
Stage2 (fr)
0.0432
0.0231
K
0.1784
0.0847
QKTA
A
0.3477
0.2655
X
0.0832
0.0350
X"
0.1478
0.0577
Layer1 Layer2 0.2733
0.0518
0.0246
SMLP
0.1869
The Variance and Expectation QKTA. Figure visualize the variance and expectation of QKTA (Formula 15 and [6 in Appendix 7.2) and SSA (rormula[T9]and 20 Appendix (7.2). N is set as 196 and is set as 64, respectively. We can find SSA has much larger variance and expectation than QKTA on the whole. For example, the maximum theoretical variance of QKTA is 16, but the maximum theoretical variance of SSA is over 3000. This is the main reason that Q-K attention can discard scaling operations thus reducing power consumption, but SSA can not.
4.4 Ablation Study
SPEDS Module, this experiment, We replaced the Spiking Patch Splitting (SPS) module in Spikformer with Spiking Patch Embedding with Deformed Shorteut (SPEDS) module, while other conditions remain unchanged. The results (Table 5 show that the SPEDS module is essential QKFormer both static and neuromorphic datasets. In addition, the addition of SPEDS to Spikformer leads to great gains in CIFAR100 (+2.05%) and CIEARIO-DVS (+1.30%), which further verified the effectiveness of SPEDS.
Table 5: Ablation studies of SPEDS module.
Model
CIFARI00 100 (Acc)
CIFARIO-DVS (Acc)
OKFormer (QKTA+ + SSA, baseline)
81.15%
84.00%
QKFormer (QKTA + SSA, w/o SPEDS)
80.08%
83,40%
Spikformer (SSA, w/o scaling)
76.95%
79.30%
Spikformer (SSA)
78.21%
80.90%
Spikformer (SSA) + SPEDS
80.26%
82.20%
Table 6: Ablation studies of Q-K Attention.
Model
CIFAR100 (Acc, Param)
CIFAR10-DVS (Acc, Param)
OKFormer (QKTA SSA, baseline)
81.15%, 6.74M
84.00%, 1.50M
OKromrer (QKCA + SSA)
81.07%, 6.74M
84.30%, 1.50M
Formmr (QKTA QKCA)
81.04%, 6.44M
83.10%, 1.44M
Former (SSA)
81.23%, 6.79M
84.10%. 1.52M
Formrr (QKCA)
81.00%, 6.44M
80.70%, 1,44M
Former (QKTA)
79.09%, 6.44M
80.70%, 1.44M

---

Mixed Spiking Attention Integration with Q-K Attention. In this part, we show different in- tegration strategies of QKCA, QKTA, and SSA. The baseline is our QKFormer (QKTA + SSA, 6.70M). The experimental results (Table show that using single type of Q-K attention (QKTA or QKCA only) in hierarchical architecture model leads to suboptimal performance. particular, the performance decline in QKTA is more obvious. While the mixed spiking attention solutions, such as QKFormer(QKTA + QKCA), QKFormer(QKTA + SSA), and QKFormer(QKCA + SSA) can achieve comparable performance to QKFormer(SSA) while requiring fewer parameters and much fewer memory resources (Figure ( Consequently, the mixed spiking attention solutions are well- suited for larger architectures and more challenging scenarios when considering both computational efficiency and performance.
Residual Connection (RC) & Spiking Neuron
Table 7: Ablation studies of RC, SN, TS.
(SN) & Time Step (TS). The experimental re-
Model
CIFARI00 (Acc)
sults are shown in Table 7 In this block, we replaced the Activation- -eee---efoorreeeeeee Addition (ABA)
QKFormer (baseline)
81.15%
[28 I1 residual connection of QKFormer with
OKFormer (ABA->PA)
81.18%
the Pre-activation (PA) [31 37 way, and the performance slightly improved. In addition, we replaced the LIF spiking neuron with Integrate- and-Fire (IF) and Parametric-Lest Leaky-Integrate- and-Fire (PLIF) [38), which led to slight per- formance degradation. The accuracy regarding different simulation time steps of QKFormer is shown in the last column. When increase the simulation time steps, the performance of QKFormer can be further improved. Specifically, OKFormer achieves 81.30 % accuracy on CIFAR 100 when T=6.
QKFormer (LIF->IF)
80.95%
QKromeer (LIF->PLIF)
81.12%
QK Former (T=1)
78.51 %
QKFormer (T=2)
80.08%
QKForeer (T=4)
81.15%
QKFormer (T=6)
81.30%
5 Conclusion
In this work, we design a novel spike-form Q-K attention considering the properties of SNNs. which can easily model the importance of token or channel dimensions through binary vectors. Q-K attention has linear complexity to #tokens (or #channels) and only adopts two spike-form components: Query (Q) and Key (K). We design a versatile and powerful Spiking Patch Embedding with Deformed Shortcut (SPEDS), which enhances spiking information transmission and integration, thus improving the performance spiking transformers. In addition, we develop a hierarchical spiking transformer based on the proposed Q-K attention and SPEDS in direct training way. named QKFormer, which marks the effective exploration of hierarchical spiking representation in Transformer-baset SNNs. Extensive experiments show that the proposed model achieves SOTA performance neuromorphic datasets. Note that QKFormer achieved top-1 accuracy beyond 85% on ImageNet- Ik with 4 time steps using the direct training way for the first time. With its powerful performance, aim for our investigations to instill optimism in the application of SNNs.
both static and
Limitation. Currently, our model is limited to image / DVS classification tasks. We will extend this work to more tasks, such as segmentation, detection, and language tasks, to test the generalizability in the further. In addition, we will explore efficient and high- gigh-performanc network architectures with fewer time steps based on Q-K attention and other efficient modules, to further reduce the training consumption.
Acknowledgments and Disclosure of Funding
This work is supported by grants from the National Natural Science Foundation of China (62236009, 62206141, 62027804, and 62425101), and the major key project of the Pengcheng Laboratory (PCL2021A13). Computing support was provided by Pengcheng Cloudbrain.
References
[1] Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 1659)1155 1997.
10

---

[2] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike- machine intelligence with neuromorphic computing. Nature, 575(77844:607-617, 2019.
[3] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings the Interna- tional Conference on Neural Information Processing Systems (NeurIPS), volume 30, 2017.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representa- tions (ICLR), 2020.
[5] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 558-567, 2021.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision (ECCV), pages 213-229. Springer, 2020.
[7] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
[8] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10012-10022, 2021.
[9] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 568-578, 2021.
[10] Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan. Volo: Vision outlooker for visual recognition. arXiv preprint arXiv:2 13112, 2021.
[11] Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng YAN, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. In The Eleventh International Conference on Learning Representations, 2023.
[12] Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Han Zhang, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual lcarning for transformer- spiking neural network, 2023.
[13] Man Yao, Jiakui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike- driven transformer, 2023.
[14] Chenlin Zhou, Han Zhang, Zhaokun Zhou, Liutao Yu, Zhengyu Ma, Huihui Zhou, Xiaopeng Fan, and Yonghong Tian. Enhancing the performance of transformer-bast based spiking neural networks by improved downsampling with precise gradient backpropagation, 2023.
[15] Yuchen Wang, Kexin Shi, Chengzhuo Lu, Yuguo Liu, Malu Zhang, and Hong Qu Spatial- temporal self-attention for asynchronous spiking neural networks. In Edith Elkind, editor, Proceedings of the Thirly-Scoond International Joini Conference on Artificial Intelligence, IJCAI-23, pages 3085-3093. International Joint Conferences on Artificial Intelligence Organi- zation, 8 2023. Main Track.
[16) Rong Wang, Mianxin Liu, Xinhong Cheng, Ying Wu, Andrea Hildebrandr, and Changsong Zhou. Segregation, integration, and balance of large-scale resting brain networks configure different cognitive abilities. Proceedings of the National Academy of Sciences, (18(23): 18, 2021.
11

---

[17] Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(():54-66. 66, 2015.
[18] Eric Hunsberger and Chris Eliasmith. Spiking deep networks with lif neurons. arXiv preprint arXiv: 2015.
[19] Tong Bu, Wei Fang. Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann- snn conversion for high-accuracy and
spiking neural networks. In International Conference Learning Representations (ICLR), 2021.
[20] Yuhang Li, Shi-Wee Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. ArXiv, awss/100..6eee 06984, 2021.
[21] Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13558-13567, 2020.
[22] Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann-snn conversion for high-accuracy and ultra-low-latency spiking neural networks, arXiv preprint 2rX1v:2303.04347 2023.
[23] Ziqing Wang. Yuetong Fang, Jiahang Cao, Qiang Zhang, Zhongrui Wang, and Renjing Xu. Masked spiking transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1761-1771, 2023.
[24] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training
spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
[25] Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 63, 2019.
[26] Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Yisen Wang, and Zhouchen Lin. Training feedback spiking neural networks by implicit differentiation on the equilibrium state. In Pro- ceedings of the International Conference on Neural Information Processing Systems (NeurIPS), volume 34, pages 14516-14528, 2021.
[27] Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. In Pro- ceedings of the International Conference on Neural Information Processing Systems (NeurIPS), volume 31, 2018
[28] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep Residual Learning in Spiking Neural Networks. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), volume 34, pages 21056- 21069, 2021.
[29] Ole Juri Richter, QIAO Ning, Qian Liu, and Sadique UI Ameen Sheik. Event-driven spiking convolutional neural network, June 16 2022. US Patent App. 17/601,939.
[30] Jing Pei, Lei Deng, Sen Song, Mingguo Zhao Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He. et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):006-111 2019.
[31] Yifan Hu, Yujie Wu, Lei Deng, and Guoqi Li. Advancing residual learning towards powerful deep spiking neural networks. arXiv preprint arXiv:21 2021.
[32] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference machine learning, pages 10357-10357. PMLR, 2021.
[33] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
12

---

[34] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020.
[35] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001-13008, 2020.
[36] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th Europeun Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 646-661. Springer, 2016.
[37] Chenlin Zhou, Han Zhang. Liutao Yu, Yumin Ye, Zhaokun Zhou, Liwei Huang, Zhengyu Ma, Xiaopeng Fan, Huihui Zhou, and Yonghong Tian. Direct training high-performance deep spiking neural networks: A review of theories and methods. arXiv preprint arXiv:2405.0428)) 2024.
[38] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2661-2671, 2021.
[39] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
[40] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[41]
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. 0--vs: an event- stream dataset for object classification. Frontiers neuroscience, 11:309, 2017.
[42] Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, Jeff Kusnitz, Michael Debole, Steve Esser, Tobi Delbruck, Myron Flickner, and Dharmendra Modha. A low power, fully event-based gesture recognition system. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2017.
[43] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going Deeper With Directly- Trained Larger Spiking Neural Networks. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 1 11062-11070, 2021.
[44] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting. In International Conference on Learning Representations (ICLR), 2021.
[45] Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kernel skeletons for powerful enn via asymmetric convolution blocks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1911-1920, 2019.
[46] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repygg: Making vtg-style convnets great again. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13733-13742. 2021.
[47] Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[48] Guangyao Chen, Peixi Peng, Guoqi Li, and Yonghong Tian. Training full spike neural networks via auxiliary accumulation pathway. arXiv preprint arXiv:2301. 1929,, 2023.
[49] Souvik Kundu, Massoud Pedram, and Peter A Beerel. Hire-snn: Harnessing the inherent robustness of energy merry-efffcieent deep spiking neural networks by training with crafted input noise. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pages 5209-5218, 1.21.
13

---

[50] Mark Horowitz. 1.1 computing's s energy problem (and what we can do about it). In 2014 IEEE International Sotid-State Circuits Conference Digest of Technical Papers (ISSCC), pages 10-14. IEFF, 2014.
[51] Priyadarshini Panda, Sai Aparna Aketi, and Kaushik Roy. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience, 14:653, 2020.
[52] Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[53] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perfforrrree deep learning library. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[54] Wei Fang, Yanqi Chen, Jianhao Ding, Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. Spikingjelly; An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, 940)) 1480, 2023.
[55] Ross Wightman. Pytorch image models. pytorch- moade 2019.
https://github.omm com rww.//rwiia http:/thhw..ccmm//.hhttana/////
14

---

7 Appendix
7.1 Spiking Neuron Model
Spiking neuron is the fundamental unit of SNNS, we choose the Leaky Integrate-ant ann-tee-andFFree (LIF) model as the spiking neuron in our work. The dynamics of LIF neuron can be formulated as follows:
H14 VI 1] (x(4) (V[t 1]
Vreset })
(12)
S[t] O (H V(h): Vid = (1 - S(1) + Vresei S(t),
(13)
(14)
where 7 is the membrane time constant, and [(] is the input current at time step t. When the membrane potential [t) exceeds the firing threshold Vaa. the spiking neuron will trigger spike S[t]. (v) is the Heaviside step function, which equals to 1 when v 2 0 and O otherwise. V[[] represents the membrane potential after the triggered event, which equals to H[[ if no spike is generated and otherwise equals to the reset potential Vreset-
7.2 Q-K Attention Vs. SSA in Scaling Factors
Mathematical Characteristics of Q-K Attention. All the elements in Q-K attention are spike- form, thus we assume that each Qi, [t] are independent random variables and subject to Bernoulli distribution B(fq)- fq is the average firing rate of Q. The expectation and variance of Q-K attention can be formulated as:
d
d Σ ==
E (QKTA)) =
Σ
2igl4
=
E(Qu)(()) dfQ.
(15)
i=u
d Σ
d = Σ
Var (QKTA) = Var
Piold
Var (Qizlt)) = dfa (1 - fQ)),
(16)
1=0
==
N Σ
(QQKCA)
Σ 2ull
E (Qiu(()) = NSe.
(17)
1=
N
Var (QKCA)= Var
Σ
Var (Qiz NSQ Sq),
(18)
where d D/h is the feature dimension of a head in the multi-head Q-K attention and N is the token number.
Mathematical Characteristics of SSA. Similar to the above analysis process, assume that all elements [t). [t) [t, in SSA are independent random variables and subject to Bernoulli distribution ((f2), Bffk), B(fv). respectively. fo. fK and fv are the average firing rate of Q, K and V, respectively. We can calculate the expectation and variance of SSA as follows.
N l,
E(SSA) = E
=
E (Qa, (V,, = Ndfafkfv.
(19)
Var(SSA) = Var
Qi, [K (t,,
ΣΣ Var (1,
i=
=
fv)
+folK
fx) 1- (1- -v)
+ (1- (1- (1 (v))
(20)
15

---

Figure visualize the variance and expectation of QKTA (Formula 15 and 16) and SSA (Formula 19 and 20 N set as 196 and d is set as 64, respectively. We can find SSA has much larger variance and expectation than QKTA on the whole. For example, the maximum theoretical variance of QKTA 16, but the maximum theoretical variance SSA over 3000. This is the main reason that Q-K attention can discard scaling operations thus reducing power consumption, but SSA can not.
7.3 Futher Discussion on Q-K Attention
The Complexity of Attention Mechanisms.
The computational complexity of SSA: Q, K 10, 1J The attention map (Q x KT E ZNNN) ) is obtained by matrix multiplication of matrix [0, I and matrix [0, 1 DXNN which thus need O(N D) computation. The computational com- plexity of SDSA: Q, K E [0, 1] [Nx The attention map (Q & K E [0, 1} is obtained by the Hadamard product of matrix (0, 1 N and matrix [0, 1] N which thus need (NND) computation.
The computational complexity of Q-K Attention: Our attention vector (A, E [0, 1] [XX is computed
by A, SN Qi,)), which depends on the row or column accumulation of the matrix (Q e [0, P) thus only needs ((N) O(D) computation.
PLIF for Scaling, Q-K attention can discard scaling operations to reduce power consumption On these datasets used in this article's experiments because the variance of Q-K attention is much smaller than SSA (e.g. the max theoretical variance of Q-K token attention only about 200 of SSA). For generality, we can also replace the LIF after attention calculation with PLIF (LIF with trainable parameters) allowing for adaptively controlling the fire rate of that spiking neuron, which can be seen as learnable scaling. It can be expressed as A, PLIF == Qi.). The results show that this modification brings a 0.2% performance improvement on CIFAR 100 (Ace = 81.17%, the firing rate of A, is 0.2952 in stagel and 0.4008 in stage 2), while increasing the training time to 1.3 times.
7.4
Supplementary for Method 3.5
(4)
(b)
Minholkg
StilSSeg EIF
MuniSepp
MultStop
MacPosling
Cono HIN
Comy II
MasPodling
MaxPooling
BN
Mulhhiep LIF
MiltSker
TV
MasPooling
MultStep LIF
MiltStop LIF
MuliSSep LIF
MacPooling
Cono HNNi
Cony
MaliSigg LIF
MiltStte
Cosy BN
Ciarn - BN
Mutteitep LIII
SPEDS-2 & SPEDS-3 in Pre-activation
SPS in Spikformer
SPEDS-1
SPEDS-2 & SPEDS-3
Figure 5: (a) Spiking Patch Splitting (SPS) module in Spikformer. (b) Spiking Patch Embedding with Deformed Shorteut (SPEDS) module in QKFormer.
Experimental Details
Datasets. We evaluate FKFormer static image classification and neuromorphic classification. The former includes ImageNet-11 339, CIFAR10/100 40 The latter contains CIFAR10-DVS 41 and DVS128 Gesture [42].
ImageNet-IK: the most typical static image dataset for classification. contains 1.28 million images for training and 50k images for validation, with total of 1,000 categories. CIFAR 10/CIFAR 100 provides 50, 000 train and 10, 000 test images with 32 x 32 resolution. The difference is that CIFAR10 contains 10 categories for classification. While CIFARION contains 100 categories, owning better distinguishing ability for the classification algorithm.
CIFAR 10-VR is event-based neuromorphic dataset converted from the static image dataset by capturing shifting image samples through Dynamic Vision Sensor (DVS) camera, which provides 9,000 training samples and 1,000 test samples. DVS128 Gesture is event-based bastd gesture
16

---

recognition dataset that contains 1342 samples of 11 hand gesture categories from 29 individuals under illumination conditions, each gesture has an average duration of 6 seconds.
Training Details. In our experiments, we use NVIDIA Tesla V100 SXM2 32GB GPUs when training models ImageNet, while GPU is used train other datasets (CIFARIO, CIFAR100, DVS128 Gesture, CIFARIO-DVS), In direct training SNN models with surrogate function,
(21)
+ exp(-axr)
we select the Sigmoid function (x) as the surrogate function with a 4 during the backpropagation of direct training in all experiments.
Experimental Details on CIFAR and Neuromorphic Classification. We evaluate our QKFormer on small-scale datasets. including CIFAR10, 10, CIFAR 100 [40] and temporal neuromorphic datasets (CIFAR 10-DVS and DVS128 Gesture
42
The detailed results on the four small-scale datasets are
presented in Table B
Table 8: Comparision on CIFAR10, CIFARI00, DVS128, CIFARIO-DVS.
Dataset
Methods
Architecture
Param (M)
Time
Top-1 Acc
Step
(%)
STBP-tdBN
ResNet-19
12.63
4
92.92
TET
ResNet-19
12.63
4
94.44
Spikformer
Spikformer-43384
9.32
4
95.51
CIFARIO
Spikingformer 12
Spikingformer-4-384
9.32
4
95.81
CML 14
Spikformer-43384
9.32
96.04 4
S-Transformer 13
S- TrTransformer225522
10.28
95.60 4
QKFormer
HST-4-384
6.74
4
96.18
STBP-tdNNER 43
ResNet-19
12.63
4
70.86
TET
Re
12.63
4
74.47
Spikformer
Spikformer-43384
9.32
4
78.21
CIFARIO0
Spikingformer 12
Spikingffrmerr4-384
9.32
78.21 4
CML
Spikformer-43384
9.32
4
80.02
S- S-Transfommer
13
S- Trannsformer225512
10.28
4
78.4
QKFormer
HST-4-384
6.74
4
81.15
Spikformer
Spikformer-2-256
2.57
10, 16
96.9. 98.3
Spikingformer
Spikingformer-222566
2.57
10, 16
96.2, 98.3
CML[14]
Spikformer-2-256
2.57
10, 16
97.6. 98.6
DVS128
S-Transfomer 13
2-Transformer222256
2.57
16
99.3
STSA[15]
STSFormer-2-256
1.99
10, 16
97.3 98.7
QKFormer
HST-2-256
1.50
10, 16
98.3 98.6
Spikformer
Spikformer-2-256
2.57
10, 16
78.9, 80.9
Spikingformer
Spikingformer-222566
2.57
10. 16
,9.9,81 81.3
CML 14
Spikformer-22256
2.57
10, 16
79.2, 80.9
CIFARIO-DVS
S- &Transformer
1
S- 2Transfommer-22266
2.57
16
80.0
STSA
STSFormer-2-256
1.99
10, 16
OKFormer
HST-2-256
1.50
10, 16
83.8 84.0

---

Training and Testing Curve on ImageNet. We show the training loss, test loss, top-1, and top-5 test accuracy of QKFormer (64.96M, 29.08M, 16.47M) on ImageNet- in Figure
(a) Training loss
(b) Testing loss
(c) Top-1 accuracy
(d) Top-5 accuracy
Figure 6: Training loss, test loss, top-1 and top-5 test accuracy of QKFormer on ImageNet-IK, The input resolution of training and testing are 224 224.
7.6 Energy Consumption Calculation of QKFormer and ANNS
The homogeneity of convolution allows the following BN and linear scaling transformation to be cquivalently fused into the convolutional layer with an added bias when deployment [45] 46 47 48) Therefore, when calculating the theoretical energy consumption, the consumption of BN layers could be ignored. We calculate the number of Synaptic Operations (SOPs) of spike before calculating theoretical energy consumption for QKFormer.
SOP/ = fr X FLOPS!
(22)
where is block/layer in QKFormer, fr is the firing rate of the block/layer and 7 is the simulation time step of spiking neuron. FLOPs refers to floating point operations of block/layer 1, which is the number of aiiiilly-and-cccumlla (MAC) operations. And SOP' is the number of spike- assume that the
accumulate (AC) operations. Refer to previous works 49
50
12 51 52
MAC and AC operations are implemented on the 45nm hardware 50 where FAMAC = 4.6pJ and FAC 0.9p.. The theoretical energy consumption of QKFormer be calculated follows:
EqgFormer EAC
Σ SOP ConY +
Σ SOP!
SOP
QKTA
Σ
SOP
SSA
+ EMAC x (FLOP) Conv)
(23)
23 shows the energy consumption of QKFormer. FLOP Cone is the first layer encoding the non-spike input into spike-form. Then the SOPs of N SNN Conv layers, M QKTA layers, and Z SSA layers are added together and multiplied by EAC- For ANNs, the theoretical energy consumption can be calculated:
FANN = EMAC x FLOPs
(24)
7.7 Supplementary for Memory Consumption in Experiment 4.3
Table 9 shows the detailed values of Figure 4 in the main body of this paper (Experiment 4.3). We compare the memory consumption between QKTA (Formula(6) and SSA (Formula (4) under different token numbers, which is calculated
GPU by forwarding the input tensor B, To facilitate the statistics of the impact of #tokens N on memory consumption, the #channels C' are set 1. The experiment result is shown in Figure 3(b) With the increment of #Tokens, SSA consumes much more GPU memory than QKTA, of which the complexity is linear to #Tokens. For example, SSA consumes about 10x GPU memory than QKTA when / 50.
256, and the time step T and batch size B are
7.8 The Training and Inference Time Comparison.
We organized the experiment to test the training and inference time QKFormer compared with Spikformer. We carried out this experiment on ImageNet with input size of 224*224. This
18

---

Table 9: Detailed values of memory consumption of Figure 4.
VN
QKTA (M)
SSA (M)
SSA / QKTA
10
0.10
0.14
1.37
20
0.40
1.00
2.53
30
0.89
3,97
4.46
40
1.58
12.19
7.71
50
2.47
26.44
10.70
60
3.56
53.52
15.04
70
4.84
97.64
20.17
80
6.32
162.50
25.70
90
8.18
258.19
31.55
100
10.35
391.77
37.85
110
12.14
570.69
47.01
120
14.23
806.06
56.66
130
16.70
1107.50
66,33
140
20.22
1485.14
73.43
150
22.26
1954.03
87.79
160
26.29
2525.00
96.03
170
28.55
3214.30
112.57
180
32.37
4036.88
124.71
190
36.41
5007.25
137.51
200
40.46
6143.06
151.84
experiment is carried out Ubuntu 18.04.6 LTS server with the Intel(R) Xeon(R) W-2125 CPU @ 4.00GHz, and the GeForce RTX 2080 (8G) GPU. "BS" means Batch Size. The experimental results follows:
Table 10: The training and inference time comparison between QKFormer and Spikformer.
Model
Inference time (1 batch)
Training time (1 batch)
Spikformer(29 .68M, T=4). BS=6
1.63s
2.65s
QKFormer(299088M, BS=6
1.82s
3.62s
Spikformer(29.66) 66M, T=4), BS=1
1.46s
2.08s
QKFormer(29.08M. T=4). BS=1
1.33s
2.72s
In terms of inference time, QKFormer and Spikformer are very close. In terms of training time, QKFormer about 1.35 times the training time of Spikformer in one batch, caused by hierarchical architecture. The training epochs of QKFormer ImageNet are 200, while the training epochs of Spikformer are 300 [11, thus, the total training time cost of QKFormer on ImageNet is close to Spikformer's.
7.9 Discussion
Prospect. The human brain has powerful intelligence that runs with low power consumption, so how to develop novel artificial intelligence algorithms to achieve high performance with the low power consumption of the human brain level is one of the Al's ultimate goals. SNN is an attractive potential way to achieve it. QKFrrmer directly trained on ImageNet-1 has groundbreaking leap forward with 10.84% accuracy improvement compared to the previous SNN model while maintaining the energy-efficient feature, which marks important step towards this goal. Combined with pre-training the future, the performance of QKFormer is expected to be further improved.
Reproducibility. The experimental results in this paper are reproducible. All experiments are implemented based Pytorch (53), SpikingJelly (54) and Timm (55). We explain the details of model training and configuration in the main text and Appendix. Our codes and models of QKFormer will be available on GitHub after review.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See the Abstract and Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See the Appendix. 7.9
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
See Section.
3 and Appendix. 7.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section. 4 and Appendix.
7.5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer:
[No]
Justification:
Our codes and models of QKFormer will be available on GitHub after review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips. cc/
publ icl gu des Codeeuumees ss aisionoolii icy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. cc/
wwwbliccguideee / //odideee/Cdeeeeeeee Cooeeeooeeeeeeeeee SudeSuumiiee wwwmisssiiii ssssnoooiccyyyycco for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section. 4 and Appendix.
7.5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix. 7.5
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://pppp...aaa.p.....co/oo/ap
Question:
Answer: [Yes]
Justification:
This research in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[NA]
Justification: This research belongs to foundational research and is not tied to particular applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Appendix. 7.5
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper thcode com dataaetteett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "Bv8zIlDuMG", "reviewer": "Reviewer_m2NM", "summary": "The authors proposed QKFormer, which pushes SNN to 85% accuracy in ImageNet, becoming a new sota and contributing to the SNN community.", "questions": "1. In Table2 the authors list a comparison of multiple metrics for different models, but there is a large time step for SNN, how does QKFormer's time during training or inference compare to other models?\n2. The authors could add an explanation of Table1 to the supplemental material to visually analyze the complexity of other attention mechanisms.\n3. Q-K token attention Sum each column and then use LIF on the column vectors, as shown in Fig. 1(a), 4 to 1,2 to 0. How is the threshold set?", "limitations": "See Questions and Weaknesses.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The authors propose a model accuracy of Sota, which is 10% higher than the previous spikformer, and the number of parameters is lower than spikformer, which is undoubtedly the leader of the SNN-Transformer series.\n2. The description is clear and the comparison is complete. The working mechanism of Q-K Attention is clearly expressed. The visualization results are rich. Comparison experiments are complete, comparing the time complexity and space complexity of different attentions in SNN domain in recent years, comparing with the Transformer architecture in ANN domain, covering the commonly used static datasets CIFAR10, CIFAR100, ImageNet, and DVS datasets DVS128, CIFAR10-DVS.", "weaknesses": "1. The current accuracy is high, but it still requires a large time step and a large amount of computation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We have carefully studied your comments and have made every effort to address your concerns. We will include the relevant analysis in the revised manuscript accordingly.\n\n\n\n### **Weaknesses**\n> ***Weaknesses 1**: The current accuracy is high, but it still requires a large time step and a large amount of computation.*\n\n**WA1**: QKFormer achieves SOTA performance by keeping the same time step settings as the previous direct trained spiking transformers [1,2,3] (such as $T=4$ on ImageNet and CIFAR dataset). Further reducing the time step is worth exploring in SNN and your feedback will inspire our future work.\n\n\n### **Questions** \n> ***Questions 1**: In Table 2 the authors list a comparison of multiple metrics for different models, but there is a large time step for SNN, how does QKFormer's time during training or inference compare to other models?*\n\n**QA1**: Thank you for your insightful question! To address your concern, we have tested the training and inference time of QKFormer and Spikformer for comparison [1]. We carry out this experiment on ImageNet with an input size of 224*224, on an Ubuntu 18.04.6 LTS server with an Intel(R) Xeon(R) W-2125 CPU @ 4.00GHz, and a GeForce RTX 2080 (8G) GPU. \"BS\" means Batch Size. The experimental results are as follows:\n\n\n**Table R4: The training and inference time comparison**\n\n| Model | Inference time (1 batch) | Training time (1 batch)|\n| -------- | -------- | -------- |\n| Spikformer(29.68M, T=4) , BS=6  | 1.63s     | 2.65s    |\n| QKFormer(29.08M,T=4) , BS=6  | 1.82s      | 3.62s    |\n| Spikformer(29.68M, T=4) , BS=1 | 1.46s      | 2.08s   |\n| QKFormer(29.08M, T=4) , BS=1 | 1.33s      | 2.72s    |\n\n**In terms of inference time**, QKFormer and Spikformer perform comparably. **In terms of training time**, QKFormer requires approximately 1.35 times the duration per batch compared to Spikformer, due to its hierarchical architecture. **In terms of the training epochs**, QKFormer is trained on ImageNet for 200 epochs, while Spikformer requires 300 epochs [1]. Consequently, **the total training time cost of QKFormer on ImageNet is close to Spikformer's**.\n\n\n\n> ***Questions 2**: The authors could add an explanation of Table1 to the supplemental material to visually analyze the complexity of other attention mechanisms.*\n\n**QA2**: Thank you for your suggestion! We explain Table1 in detail as follows and will add it to the revised manuscript accordingly.\n\nThe computational complexity of **SSA**: $Q, K \\in [0, 1]^{N \\times D}$. The attention map ($Q \\times K^{\\mathrm{T}} \\in Z^{N \\times N}$) is obtained by matrix multiplication of matrix $[0, 1]^{N \\times D}$ and matrix $[0, 1]^{D \\times N}$, which requires $O(N^2 D)$ computation.\n \nThe computational complexity of **SDSA**: $Q, K \\in [0, 1]^{N \\times D}$. The attention map ($Q \\otimes K \\in [0, 1]^{N \\times D}$ ) is obtained by the Hadamard product of matrix $[0, 1]^{N \\times D}$ and matrix $[0, 1]^{N \\times D}$, which requires $O(ND)$ computation.\n\nThe computational complexity of **Q-K Attention**: Our attention vector (${A}_t \\in [0, 1]^{N \\times 1}$) is computed by\n\n${A}\\_t = SN(\\sum_{i=0}^D {Q}_{i, j})$,\n\nwhich depends on the row or column accumulation of the $Q$ matrix ($Q \\in [0, 1]^{N \\times D}$), thus only requires $O(N)$ or $O(D)$ computation. \n\n\n> ***Questions 3**: Q-K token attention Sum each column and then use LIF on the column vectors, as shown in Fig. 1(a), 4 to 1,2 to 0. How is the threshold set?*\n\n**QA3**: The LIF neuron is implemented by Spikingjelly [4]. The time constant is set as 2, the threshold is 0.5 by default. You may worry that the accumulation operation along $D/h$ will lead to the following LIF over-fires. \n\ni) Actually, **the hyperparameter settings of LIF in our work will lead to $Q$ becoming very sparse compared to $K$ when the network converges**. **see Table R3**, **the $Q$ in stage 1 has a fire rate of 0.0432, while $K$ has 0.1784**. After the accumulation operation along $D/h$ of the multi-head QKTA version, **the LIF ($A\\_t$) has a typical fire rate of 0.3477**.\n\nii) In addition, we have carried out an experiment that replaced that LIF with PLIF (LIF with trainable parameters) to tune the fire rate of that spiking neuron adaptively. The result shows that this way only brings a 0.2% performance improvement on CIFAR 100 (Acc = 81.17%, the firing rate of PLIF($A\\_t$) is 0.2952 in stage1 and 0.4008 in stage2), but the training time will become about 1.3 times compared with the LIF version.\n\nWe will include the details in the revised manuscript accordingly.\n\n\n[1] Spikformer: When spiking neural network meets transformer. In ICLR, 2023.\n\n[2] Spikingformer: Spike-driven residual learning for transformer-based spiking neural network. In Arxiv, 2023.\n\n[3] Spike-driven transformer. In NeurIPS, 2023.\n\n[4] SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence. In Science Advance, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eA2b0ONFGd", "reviewer": "Reviewer_ukE8", "summary": "The author has thoughtfully considered the attention structure of the existing Spiking Transformer (SSA) as well as issues present in other parts. Spiking neural networks are characterized by their high efficiency and energy-saving features. For this purpose, the author has proposed a more efficient attention mechanism. To better enhance performance, the author has also made modifications to the embedding part. From the experimental results, the new model achieved better performance on several typical datasets with lower energy consumption and fewer parameters. Additionally, the ablation experiments designed by the author demonstrated the rationality of the attention mechanism and embedding modifications. Overall, the method proposed in the article is highly feasible and logically coherent.", "questions": "I ampleased to see that such a simplified attention mechanism can improve performance while saving energy. But I still have the following questions:\n1. The capability of SPEDS on event datasets needs to be demonstrated.\n2. As shown Eq.6, row and column-wise summation is performed when computing Q-K attention. Attention mechanisms are designed to integrate global attention. Would Q-K attention cause the loss of features that do not align with the row and column directions?\n3. After summation in Eq.6,, Q undergoes neuron processing, is this to ensure the model is spike-driven? Representations in SNNs are inherently sparse; would this lead to feature loss?", "limitations": "See weaknesses and Questions", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The article analyzes the issue of computational redundancy in existing Spiking Transformer attention mechanisms.\n2. The Q-K Attention offers an almost perfect solution, being simple in design (not a bad thing), yet reducing computational complexity while enhancing performance.\n3. A better embedding method has been designed.\n4. Ablation experiments have demonstrated the rationality of these approaches.", "weaknesses": "1. Lack of an overview of pipeline. Some charts, such as Figure 3, seem to be of little significance. In contrast, SPEDS might require a more intuitive representation, given that you have made significant modifications to the previous embedding.\n2. Could the ablation study include additional experiments? We have seen experiments on SPEDS and Q-K Attention using CIFAR-100. Could we conduct a similar set of experiments on neuromorphic datasets? Given the unique nature of events, we hope to see SPEDS achieve performance improvements on neuromorphic datasets as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments and your time in reading our paper.  We have carefully studied your comments and have made every effort to address your concerns. We will include the relevant analysis in the revised manuscript accordingly.\n\n### **Weaknesses**\n> ***Weaknesses 1**: Lack of an overview of pipeline. Some charts, such as Figure 3, seem to be of little significance. In contrast, SPEDS might require a more intuitive representation, given that you have made significant modifications to the previous embedding.*\n\n**WA1:** Thanks for your constructive advice! Please refer to the PDF in \"Global Rebuttal\", where we have added a more detailed pipeline including the overall architecture and all modules of QKFormer (Figure R1 in the uploaded pdf file). We will continue to improve the figure and include it in the revised manuscript accordingly.\n\n\n> ***Weaknesses 2**: Could the ablation study include additional experiments? We have seen experiments on SPEDS and Q-K Attention using CIFAR-100. Could we conduct a similar set of experiments on neuromorphic datasets? Given the unique nature of events, we hope to see SPEDS achieve performance improvements on neuromorphic datasets as well.*\n> ***Questions 1**: The capability of SPEDS on event datasets needs to be demonstrated.*\n\n**WA2 & WQ1:** Please refer to \"Global Rebuttal\". We add ablation experiments on the neuromorphic dataset: CIFAR10-DVS. In addition, we add another two comparison models:  QKFormer(w/o SPEDS) and QKFormer(SSA). The experimental results are shown in **Table R1** and **Table R2**, with the corresponding discussion provided following these tables.\n\n**SPEDS module on neuromorphic dataset**: The results show that SPEDS leads to general performance improvements on the CIFAR10-DVS dataset. a. The SPEDS module is essential to QKFormer on both static and neuromorphic datasets. b. Adding SPEDS to Spikformer brings great gains in CIFAR100 (+2.05%) and CIFAR10-DVS (+1.30%).\n\n\n\n\n### **Questions**\n> ***Questions 2**: As shown Eq.6, row and column-wise summation is performed when computing Q-K attention. Attention mechanisms are designed to integrate global attention. Would Q-K attention cause the loss of features that do not align with the row and column directions?*\n> ***Questions 3**: After summation in Eq.6, Q undergoes neuron processing, is this to ensure the model is spike-driven? Representations in SNNs are inherently sparse; would this lead to feature loss?*\n\n**QA2 & QA3:** **The reviewer correctly notes that neuron processing after summation ensures the model is spike-driven.** By adding the spiking neuron in Eq. 6, the output in Figure 1 will be a spike-based representation rather than an integer matrix. This inclusion prevents non-spike computation between integer outputs and float weights in the post-linear layer.\n\n\n**Q-K attention models the importance of token or channel dimensions.** In other words, it identifies which token or channel is more important effectively. In contrast, SSA models the importance relationship between each two tokens. As you mentioned, a single type of Q-K attention may cause the loss of features that do not align with the row and column directions. So we proposed **the mixed spiking attention solution of QKFormer**, such as QKFormer(QKTA + QKCA), QKFormer(QKTA + SSA), and QKFormer(QKCA + SSA) with high performance. Please refer to Table R.2 and the corresponding discussion in \"Global Rebuttal\" (2. Ablation Study of Q-K Attention) for details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m29xYsxq7E", "reviewer": "Reviewer_Etfg", "summary": "This model introduces a spike-form Q-K attention mechanism that efficiently models the importance of token or channel dimensions using binary values, significantly reducing computational complexity .The model is evaluated on the ImageNet-1K dataset, achieving an impressive top-1 accuracy of 85.65%, surpassing existing benchmarks for spiking transformers.", "questions": "1.\tThe paper suggests that Q-K attention proposed here is simpler compared to SDSA-3 in Spike-Driven Transformer V2, and ablation experiments show performance degradation with Q-K Attention. The QKFormer architecture closely resembles [1] yet achieves a 5.65% improvement on ImageNet. Could this improvement be attributed to superior data augmentation or training strategies rather than the effectiveness of the attention methods and model architecture?\n2.\tSince the accuracy on ImageNet-1K significantly exceeds those of existing comparable models, it is challenging to evaluate the results without access to the corresponding code. Could you please make the code open source?", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed spike-form Q-K attention mechanism provides an efficient way to handle token or channel dimensions with binary spikes.\n2.\tThe model achieves over 85% accuracy on ImageNet-1K, which is a very impressive result.", "weaknesses": "1、The Q-K attention module discussed in the paper has already been introduced in Spike-Driven Transformer V2[1] (SDSA-2 in Figure 3). Additionally, the overall architecture of the QKFormer model closely resembles that in [2].\n2、 In Section 4.1, the authors incorrectly compare the full-precision results of their work with Binary Neural Networks. Upon reviewing the original texts, it was found that PokeBNN (20.7MB) [2] and MeliusNet59 (17.4MB) [3] report model sizes, not parameter counts (20.7M, 17.4M). Comparisons should use consistent metrics.\n3、The baseline for the Q-K attention ablation study (QKCA + SSA) is not reasonable. There is no information on the model with only SSA or without the attention module.\n4、The effectiveness of the SPEDS module with only Q-K attention and without SPEDS remains unclear. It appears that the architecture of Spikformer differs from QKFormer, with QKFormer’s architecture being more similar to that in Spike-Driven Transformer V2\n\n[1] Man Yao, JiaKui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, XU Bo, and Guoqi Li. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. In The Twelfth International Conference on Learning Representations, 2023.\n[2] Zhang Y, Zhang Z, Lew L. Pokebnn: A binary pursuit of lightweight accuracy[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 12475-12485.\n[3] Bethge J, Bartz C, Yang H, et al. Meliusnet: An improved network architecture for binary neural networks[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021: 1439-1448.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback and your time in reading our manuscript. We hope that the responses below could address your concerns. We will include the relevant analysis in the revised manuscript accordingly.\n\n### **Weaknesses**\n> ***Weaknesses 1**: The Q-K attention module in the paper has already been introduced in Spike-Driven Transformer V2[1] (SDSA-2 in Figure 3). Additionally, the overall architecture of the QKFormer closely resembles that in [2]. **Question 1**: The paper suggests that Q-K attention here is simpler compared to SDSA-3 in Spike-Driven Transformer V2, and ablation experiments show performance degradation with Q-K Attention. The QKFormer architecture closely resembles [1] yet achieves a 5.65% improvement on ImageNet. Could this improvement be attributed to superior data augmentation or training strategies rather than the effectiveness of the attention methods and model architecture?*\n\n**WA1 & WQ1:** \nThanks for your careful reading and comments! There are several major differences in our attention compared to SDSA-2[1], and we have other designs like SPEDS： \n1) **In terms of attention**, we have proposed a **mixed spiking attention framework in hierarchical architecture for integration**, while Spike-Driven Transformer V2 uses single-dimension attention. **QKFormer (QKTA + SSA)** serves as the primary model tested in the main experiments on ImageNet-1K, CIFAR, and neuromorphic datasets. The mixed spiking attention framework (including QKTA + SSA and other mixed attention variants, as detailed in \"2. Ablation Study of Q-K Attention\" in the document \"Global Rebuttal\") could efficiently leverage the importance across multiple dimensions and improve the performance compared with SDSA-2, but with higher computation efficiency compared with only SSA. Therefore, our mixed attention strikes a balance between high performance and computing requirements in QKFormer architecture, enabling model size enlargements and performance improvements on the ImageNet-1K dataset.\n\n2) **In terms of the synaptic computing layer of attention**, Q-K Attention uses a straightforward, deployment-friendly linear layer.  In contrast, SDSA-2 employs re-parameterization convolution sequences (conv->bn->conv->conv->bn layers), which necessitate tedious post-processing conversion to achieve a truly spike-driven SNN model.\n\n3) **Other differences in architecture:** a new patch embedding module (SPEDS in Sec3.5 and 4.4 in the manuscript), hierarchical stages, etc.\n\n**From the results of the ablation study (Table R1 and Table R2) and their following discussions, our performance improvement mainly comes from the mixed attention, SPEDS, and overall hierarchical architecture.** \n\n**Training details:** For training strategy, we adopt a direct training method following Spikformer [2]. For data augmentation, we follow DeiT [3], using RandAugment [4], random erasing [5], and stochastic depth [6]. \n\nIn addition, the earliest version of Spike-Driven Transformer V2 was released on Arxiv on Feb 15, 2024, while ours was released on March 25, 2024. This indicates that both studies were conducted concurrently.\n\n\n**Finally, we would appreciate it if you could further clarify the similarities between QKFormer and PokeBNN [7], as mentioned in Weakness 1. This will help us address your concerns more effectively and provide a more thorough response. Thank you!**\n\n\n> ***Weaknesses 2**: In Sec 4.1, the authors incorrectly compare the full-precision results of their work with BNNs. It was found that PokeBNN (20.7MB) [2] and MeliusNet59 (17.4MB) [3] report model sizes, not parameter counts (20.7M, 17.4M). Comparisons should use consistent metrics.*\n\n**WA2:** We apologize for the oversight and will update the manuscript later.  We found that both PokeBNN [7] and MeliusNet59 [8] haven't reported the model parameters, but instead reported the number of computation operations. The comparison is as follows, and the operation numbers are comparable:\n\nPokeBNN 2.0x (77.2\\%, BNN SOTA)[7]:14.412G binary operations, 0.0145G Int4_MACs, and 0.0107G Int8_MACs.\n\nMeliusNet-59 (71.0\\%, BNN)[8]:18.3G binary operations and 0.245G MACs\n\nQKFormer-10-384(78.80\\%, SNN): 15.12G ACs and 0.26G MACs.\n\n\n> ***Weaknesses 3**: The baseline for the Q-K attention ablation study (QKCA + SSA) is not reasonable. There is no information on the model with only SSA or without the attention module.**Weaknesses 4**: The effectiveness of the SPEDS module with only Q-K attention and without SPEDS remains unclear. It appears that the architecture of Spikformer differs from QKFormer, with QKFormer’s architecture being more similar to that in Spike-Driven Transformer V2.* \n\n\n**WA3 & WA4:**  Thanks for your constructive advice, which has greatly helped improve the quality of our manuscript. **Please refer to \"Global Rebuttal\" for responses to Weaknesses 3 and Weaknesses 4**.  According to your thoughtful comments, we add another two comparison models: QKFormer(w/o SPEDS) and QKFormer(SSA). In addition, we add the ablation study on \n CIFAR10-DVS. The results indicate the effectiveness of mixed attention and SPEDS on both static and DVS datasets.\n\n### **Question**\n\n> ***Question 2**: code open source?* \n\n**WQ2:** We will open the code as soon as the manuscript is accepted.\n\n\n[1] Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. In ICLR 2024.\n\n[2] Spikformer: When spiking neural network meets transformer. In ICLR, 2023.\n\n[3] Training data-efficient image transformers & distillation through attention. In ICML, 2021.\n\n[4] Randaugment: Practical automated data augmentation with a reduced search space. In CVPR workshop, 2020.\n\n[5] Random erasing data augmentation. In AAAI, 2020.\n\n[6] Deep networks with stochastic depth. In ECCV, 2016.\n\n[7] Pokebnn: A binary pursuit of lightweight accuracy, In CVPR, 2022.\n\n[8] Meliusnet: An improved network architecture for binary neural networks. In WACV，2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z0i6PxnlH8", "reviewer": "Reviewer_7jT8", "summary": "The authors introduce a novel spiking transformer, QKFormer, which incorporates several innovative features: a spike-form Q-K attention mechanism with linear complexity and enhanced energy efficiency, a hierarchical structure that facilitates multi-scale spiking representation, and a spiking patch embedding with a deformed shortcut designed to optimize spiking information transmission and integration. Notably, this model achieves a remarkable top-1 accuracy of 85.65% on the ImageNet-1k dataset, marking the first instance where directly trained spiking neural networks have surpassed the 85% accuracy threshold on this benchmark.", "questions": "1) Could the summation operation in the QK attention mechanism potentially result in an excessively high firing rate within the attention vector? \n2) Could you elaborate on the relationship between Figure 3a and the original input image? \n3) As noted in the weaknesses section.", "limitations": "The authors briefly mention limitations in the Appendix, but discussing them in the main text would enhance clarity. Another limitation is that the model was only evaluated on image classification, limiting its generalizability.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The authors' introduction of a novel spike-form Q-K attention module with linear complexity represents a significant advancement in mitigating the quadratic computational complexity of spiking self-attention, a core component of traditional spiking transformers. This innovation demonstrates a substantial degree of originality and contributes to the optimization of spiking neural network architectures.\n2) The authors' design of a powerful spiking patch embedding with deformed shortcut (SPEDS) is a notable achievement, as it effectively enhances spiking information transmission and integration. This enhancement contributes to the overall performance of the proposed QKFormer model and underscores the authors' creative approach to improving SNN performance.\n3) The proposed QKFormer model, leveraging the innovative Q-K attention and SPEDS, represents a substantial leap forward in SNN performance. Its ability to outperform state-of-the-art SNNs on various static and neuromorphic datasets, particularly the ImageNet-1K dataset, where it achieves over 85% accuracy for the first time with directly trained SNNs, underscores its significance and potential impact on real-world applications of SNNs. This achievement not only demonstrates the model's quality but also highlights its potential to revolutionize the field of SNN research.", "weaknesses": "1) The authors' assertion that SPEDS can facilitate spiking information transmission and integration lacks sufficient elaboration on the underlying mechanisms. \n2) Clarifying the rationale behind the assertion that the low variance of Q-K attention negates the necessity for scaling operations would significantly enhance the paper's rigor and credibility. \n3) To improve the overall writing quality and grammar of the paper, several revisions are recommended. Firstly, in the caption of Figure 1, the phrase \"The input is\" should be revised to \"The inputs are\" to ensure consistency with the plural form of \"inputs.\" Secondly, the expression in line 246 appears unusual and requires refinement to enhance clarity. Lastly, the abbreviation \"LF\" should be corrected to \"IF\" in line 316, assuming it is a typographical error and should stand for \"if.\" Additionally, a comprehensive review and refinement of the entire paper's language, grammar, and structure would further elevate its academic style and readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We have carefully studied your comments and have made every effort to address your concerns. We will include the relevant analysis in the revised manuscript accordingly.\n\n### **Weaknesses**\n> ***Weaknesses 1**: The authors' assertion that SPEDS can facilitate spiking information transmission and integration lacks sufficient elaboration on the underlying mechanisms.*\n\n**WA1:** Residual shortcuts in SNNs [1] can implement identity mapping, which reduces information loss (facilitates information transmission and integration) in spike communication, thus ensuring the network can be well-behaved in a depth-insensitive way. Previous spiking transformers [2, 3] use the residual shortcuts to achieve identity mapping, mainly focusing on the spiking Attention block and spiking MLP block, and lacking identity mapping in patch embedding across the downsampling block. **In SEPDS, we perform a lightweight linear projection $\\mathbf{W}_d$ in the shortcut connections to match the channel (and token) numbers, thus realizing the identity mapping cross downsampling blocks in spiking patch embedding.**\n\n> ***Weaknesses 2**: Clarifying the rationale behind the assertion that the low variance of Q-K attention negates the necessity for scaling operations would significantly enhance the paper's rigor and credibility.*\n\n**WA2:** Self-attention (sec. 3.2.1) [4] shows the variance magnitude of ${Q} {K}^{\\mathrm{T}}$ grows with the embedding dimension $d$,  which can result in gradient vanishing issues after the softmax operation. In other words, if the input magnitude is very large, the gradient of softmax will tend to 0, causing the gradient to disappear. **The larger the variance, the more likely the dot product result is larger in magnitude.** Therefore, the product of matrices ${Q}$ and ${K}$ in self-attention is scaled by a factor $\\frac{1}{\\sqrt{d}}$ to normalize the product to variance 1.  In contrast, SSA-based [3] SNNs are prone to suffer from performance degradation, and even cannot converge without scaling, because the variance of ${Q} {K}^{\\mathrm{T}} {V}$ output is too large for LIF neuron with surrogate gradient method. However, Q-K attention can discard scaling operations and reduce power consumption because the variance of Q-K attention is much smaller than SSA (e.g. the max theoretical variance of Q-K token attention is only about 1 / 200 of SSA, see Sec. 3.3 and 4.3 and Appendix 6.2 in the manuscript).\n\n> ***Weaknesses 3**: To improve the overall writing quality and grammar of the paper, several revisions are recommended. ... a comprehensive review and refinement of the entire paper's language, grammar, and structure would further elevate its academic style and readability.*\n\n**WA3:** Sorry for this and we sincerely thank you for your detailed suggestions, which are very helpful for us to improve the quality of our manuscript. We have made the following revisions:\n1) \"The input are\" in Figure 1 has been changed to \"The inputs are\".\n2) line 246 \"Results on CIFAR and Temporal Neuromorphic Classification\" has been changed to \"Results on CIFAR and Neuromorphic Datasets\".\n3) line 316 \"with LF and PLIF\" has been changed to \"with Integrate-and-Fire (IF)  and Parametric Leaky Integrated-and-Fire (PLIF)\".\n\nIn addition, we have conducted a comprehensive grammar and structure review of the entire manuscript.\n\n### **Question**\n\n> ***Question 1**: Could the summation operation in the QK attention mechanism potentially result in an excessively high firing rate within the attention vector?*\n\n**QA1:** Actually, the summation operation in the Q-K attention will lead to $Q$ becoming very sparse compared to $K$ when the network converges. **As shown in Table R3 in \"Global Rebuttal\"**, the $Q$ in stage 1 has a fire rate of 0.0432, while $K$ has 0.1784. After the accumulation operation along $D/h$ of the multi-head QKTA version, **the LIF ($A\\_t$) has a normal fire rate of 0.3477**. In addition, we could replace that LIF with PLIF to adaptively control the fire rate of that spiking neuron. The result shows that this way only brings a 0.2% performance improvement on CIFAR 100 (Acc = 81.17%, the firing rate of PLIF ($A_{t}$) is 0.2952 in stage1 and 0.4008 in stage2), but the training time will become about 1.3 times compared with the previous.\n\n> ***Question 2**: Could you elaborate on the relationship between Figure 3a and the original input image?*\n\n**QA2:** Figure 3a is mainly to visualize the firing state of Q-K attention in the network. We randomly select an image (224\\*224) from ImageNet test set for inference and visualized ${{A}}_t$, ${K}$ and ${X}^{\\prime}$ in QKTA. Input image: 224\\*224 —> Stage1 feature: 56\\*56 (flatten to 3136 tokens) —> Stage2 feature: 28\\*28 (flatten to 784 tokens). Figure 3a chooses the continuous token segment with a length of 100 ([1:100] from [1:3136] in stage 1 and [1:784] in stage 2) to visualize the firing state in QKTA.\n\n### Limitations\n\n> ***Limitations 1**: limitation discussion*\n\n**LA1:** According to your thoughtful comments, the limitation discussion has been changed to \"The limitation discussion has been changed to “Currently, our model is limited to image/DVS classification tasks. We will extend this work to more tasks, such as segmentation, detection, and language tasks, to test the generalizability in the further. In addition, we will explore efficient and high-performance network architectures with fewer time steps based on Q-K attention and other efficient modules, to further reduce the training consumption.” We will add this discussion to the main text in the revised manuscript.\" We will add this discussion to the main text in the revised manuscript.\n\n[1] Deep Residual Learning in Spiking Neural Networks. In NeurIPS, 2021.\n\n[2] Spikformer: When spiking neural network meets transformer. In ICLR, 2023.\n\n[3] Spike-driven transformer. In NeurIPS, 2023.\n\n[4] Attention is all you need. In NeurIPS, 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear ACs and Reviewers,

We would like to first express our gratitude to all the reviewers for their valuable comments. We are encouraged that reviewers have commended the performance of QKFormer architecture and its components including Q-K Attention (Q-K Token Attention and Q-K Channel Attention) and Spiking Patch Embedding with Deformed Shortcut (SPEDS) modules. Meanwhile, most reviewers are concerned about the ablation study of Q-K attention and SPEDS module, and whether the accumulation mechanism of Q-K attention leads to over-fire of spiking neurons. Our responses to these questions are as follows. We will include the relevant analysis in the revised manuscript accordingly.

### **1. Ablation Study for SPEDS Module：**

Based on the ablation results presented in Table 4 of the manuscript, we have added:

a. another model, QKFormer(w/o SPEDS); b. the ablation study on the neuromorphic dataset, CIFAR10-DVS. The experimental results are as follows:

**Table R1: Ablation studies for SPEDS.**
| Model | CIFAR100 Acc  |CIFAR10-DVS Acc|
| -------- | -------- | -------- |
| QKFormer(QKTA+SSA)| 81.15%  | 84.00%|
| QKFormer(QKTA+SSA, w/o SPEDS) | 80.08% |  83.40%|
| Spikformer(SSA) | 78.21% |  80.90%|
| Spikformer(SSA) + SPEDS | 80.26% |  82.20%|

The results show that the SPEDS module is essential to QKFormer on both static and neuromorphic datasets. In addition, the addition of SPEDS to Spikformer leads to great gains in CIFAR100 (+2.05%) and CIFAR10-DVS (+1.30%), which further verified the effectiveness of SPEDS.

### **2. Ablation Study of Q-K Attention：**

Based on the ablation results presented in Table 4 of the manuscript, we have added:

a. another model, QKFormer(SSA); b. the ablation study on the neuromorphic dataset, CIFAR10-DVS. The experimental results are as follows:


**Table R2: Ablation studies for Q-K Attention.**
| Model | CIFAR100 (Acc, Param) |CIFAR10-DVS (Acc, Param)|
| -------- | -------- | -------- |
| QKFormer(QKTA+SSA, Baseline)| 81.15%, 6.70M  | 84.0%, 1.50M|
| QKFormer (QKCA + SSA) | 81.07%, 6.70M       | 84.30%, 1.50M|
| QKFormer (QKTA + QKCA) | 81.04%, 6.44M      | 83.10%, 1.44M|
| QKFormer (SSA ) | 81.23%, 6.79M             |  84.10%, 1.52M|
| QKFormer (QKCA) | 81.00%, 6.44M             |  81.50%, 1.44M|
| QKFormer (QKTA) | 79.09%, 6.44M             |  80.70%, 1.44M|

**1) QKFormer(SSA):** 

The results in the table above clearly demonstrate **the superiority of our QKFormer architecture: fewer parameters, much stronger performance**. For instance, CIFAR100: QKFormer(SSA, 6.79M, 81.23%) VS. Spikformer(SSA, 9.32M, 78.21%) and CIFAR10-DVS: QKFormer(SSA, 1.52M, 84.10%) VS. Spikformer(SSA, 2.57M, 80.90%). **Moreover, applying the hierarchical structure with SSA, QKFormer(SSA), on large datasets (such as ImageNet-1k with 224 * 224 input size) poses challenges due to its high requirements of computational and memory resources.** For example, due to memory explosion, QKFormer(SSA) cannot be trained on ImageNet even with a batch size of 1 on an NVIDIA Tesla V100 GPU (32G). This limitation of SSA is also a key reason for us to propose the Q-K attention module with reduced computational complexity.


**2) Mixed spiking attention integration：**

**Through our exploration, the mixed spiking attention of QKFormer is the optimal solution when considering both computational efficiency and performance.** Using a single type of Q-K attention leads to reduced performance. Mixed spiking attention solutions, such as QKFormer(QKTA + QKCA), QKFormer(QKTA + SSA), and QKFormer(QKCA + SSA) can achieve comparable performance to QKFormer(SSA) while requiring fewer parameters and much fewer memory resources (Figure 3b and Table 7 in the manuscript). Consequently, the mixed spiking attention solutions are well-suited for larger architectures and more challenging scenarios."


### **3. Spike Firing Rates in Q-K Attention (QKTA)：**
We have calculated the spike firing rates for the QKFormer blocks of the trained QKFormer (64.9M) on ImageNet-1k and included them in Appendix A.5 as Table 6. We apologize for the lack of sufficient discussion. The average spike firing rates of the QKTA across neurons and time steps in Stage1 and Stage2 are also as follows:

**Table R3: Spike firing rates in Q-K Attention (QKTA) on ImageNet. $A_{t}$ denotes the fire rate of the LIF neuron after accumulation operation.**
| QKTA | Stage1(firing rate) |Stage2(firing rate)|
| -------- | -------- | -------- |
| $Q$| 0.0432  | 0.0231|
| $K$ | 0.1784       | 0.0847|
| $A_{t}$ | 0.3477      | 0.2655|
| ${X}^{\prime}$ | 0.0832       | 0.0350|
| ${X}^{\prime\prime}$ | 0.1478       | 0.0577|


In fact, the summation operation in the Q-K attention causes **$Q$ to become significantly sparser compared to $K$ when the network converges**. Specifically, $Q$ in stage 1 has a fire rate of 0.0432, while $K$ has 0.1784. **After the accumulation operation along $D/h$ of the multi-head QKTA version, the LIF neuron ($A_{t}$) exhibits a typical averaged fire rate of 0.3477.** In addition, replacing the LIF with PLIF (LIF with trainable parameters) allows for adaptively controlling the fire rate of that spiking neuron. The results show that this modification only brings a 0.2% performance improvement on CIFAR 100 (Acc = 81.17%, the firing rate of PLIF($A_{t}$) is 0.2952 in stage1 and 0.4008 in stage2), while increasing the training time to 1.3 times. 


Best regards

Authors

Meta-review: Spiking neural networks that more closely resemble real neurons have long been of interest as a more biologically grounded alternative to deep networks, but their performance have long lagged behind. In this paper, a spiking neural network transformer model is presented that achieves a large improvement above the state of the art in spiking neural networks. Specific design decisions also make this model very efficient along with being very accurate (such as a new efficient spike-based attention module that allows the construction of larger models, a multi-scale spiking architecture that models a hierarchy of representations, a new embedding approach). The paper is well written and describes the approach clearly. The authors perform a comprehensive analysis of the time and space complexity of recent spiking neural network architecture on the main datasets typically used in such approaches, effectively comparing their approach in a sound way to existing work. The author also perform informative ablation analysis. The analyses in the paper were appreciated by the reviewers, who also requested additional clarifications and analyses. All these modifications should be added to the paper for the camera ready version.
Decision: Accept (spotlight)