Spike-driven Transformer
Authors: Man Yao
JiaKui Hu
Zhaokun Zhou
Li Yuan
Yonghong Tian
Bo XU
Guoqi Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Spiking Neural Networks; Transformer; Neuromorphic Computing; Event-driven; Linear Attention

Abstract: Spiking Neural Networks (SNNs) provide an energy-efficient deep learning option due to their unique spike-based event-driven (i.e., spike-driven) paradigm. In this paper, we incorporate the spike-driven paradigm into Transformer by the proposed Spike-driven Transformer with four unique properties: (1) Event-driven, no calculation is triggered when the input of Transformer is zero; (2) Binary spike communication, all matrix multiplications associated with the spike matrix can be transformed into sparse additions; (3) Self-attention with linear complexity at both token and channel dimensions; (4) The operations between spike-form Query, Key, and Value are mask and addition. Together, there are only sparse addition operations in the Spike-driven Transformer. To this end, we design a novel Spike-Driven Self-Attention (SDSA), which exploits only mask and addition operations without any multiplication, and thus having up to $87.2\times$ lower computation energy than vanilla self-attention. Especially in SDSA, the matrix multiplication between Query, Key, and Value is designed as the mask operation. In addition, we rearrange all residual connections in the vanilla Transformer before the activation functions to ensure that all neurons transmit binary spike signals. It is shown that the Spike-driven Transformer can achieve 77.1\% top-1 accuracy on ImageNet-1K, which is the state-of-the-art result in the SNN field.

Spike-driven Transformer
Man Yao 12- Jiakui 3u Zhaokun 2hou? Li Yuan Yonghong Tian Bo Xu Guoqi Lift
Institute of Automation, Chinese Academy of Sciences, Beijing, China Peng Cheng Laboratory, Shenzhen, Guangzhou, China $Peking University, Beijing, China
Abstract
Spiking Neural Networks (SNNs) provide an energy-efficient deep learning option due to their unique spike-based event-driven (i.e., spike-driven) paradigm. In this paper, we incorporate the spike-driven paradigm into Transformer by the proposed Spike-driven Transformer with four unique properties: i) Event-driven, no calculation is triggered when the input of Transformer zero; ii) Binary spike communication, all matrix multiplications associated with the spike matrix can transformed into sparse additions; iii) Self-attention with linear complexity both token and channel dimensions; iv) The operations between spike-form Query, Key, and Value are mask and addition. Together, there are only sparse addition operations in the Spike-driven Transformer. To this end, we design novel Spike- Driven Self Attention (SDSA), which exploits only mask and addition operations without any multiplication, and thus having up to 87.2x lower computation energy than vanilla self-attention. Especially in SDSA, the matrix multiplication between Query, Key, and Value is designed the mask operation. In addition, rearrange all residual connections the vanilla Transformer before the activation functions ensure that all neurons transmit binary spike signals. It shown that the Spike- driven Transformer can achieve 77. % top-1 accuracy on ImageNet- K, which is the state-of-the-arr result in the SNN field. All source code and models available at https:://// //gi //githubb.........mme com Btt//BIIILbbbpieeee Spikeaa///pkkeeeee Sttp//ppiie--iieeene Taannn-rrassff ormer
Introduction
One of the most crucial computational characteristics of bio-inspired Spiking Neural Networks (SNNs) [1] is spike-based event-driven (spike-driven): i) When a computation is event-driven, it is triggered sparsely events (spike with address information) occur; ii) If only binary spikes or 1) are employed for communication between spiking neurons, the network's 's operations are synaptic ACcumulate (AC). When implementing SNNs on neuromorphic chips, such as TrueNorth [2] Loihi 3 and Tianjic [4, only small fraction of spiking neurons at any moment being active and the rest being idle. Thus, spike-driven neuromorphic computing that only performs sparse addition operations is regarded as promising low-power alternative to traditional Al 5 6 7
Although SNNs have apparent ad advantages in bio bio-plausibility and energy efficiency, their applications are limited by poor task accuracy. Transformers have shown high performance in various tasks for their self-attention [ 10] Incorporating the effectiveness of Transformer with the high energy efficiency of SNNs is a natural and exciting idea. There has been some research in this direction, but all so far have relied "hybrid computing". Namely, Mattiply-and-ACemmiiil (MAC) operations dominated
"gqual contribution
CCoresponding author, guoqi. li i@ia.ac.cn
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Vanilla Self-Attention (VSA)
(b) Spike-Driven Seff-Attention (SDSA)
Continupus input value X
Binrry salce inpui 5
Version
SDSA
Version
SDSA
Binary spile imput S
Lincar
Lincar
Linear
Linear
K
PC):
K
2
Scale
X
nkd
Softmax
xxd
X
0 (DNV2 D
Linear
(ND +
Linear
1xd
Dot product
Broadcasted element-w multipicctionn
21 x.
( ND)
Spiking neuron
Hadamard product (Element-wiis mask)
Column summation
Figure 1: Comparison Vanilla Self-Attention (VSA) and our Spike- Self- Self-Attention (SDSA). (a) is typical Vanilla Self-Attention (VSA) [8] (b) are two equivalent versions of SDSA. The input of SDSA are binary spikes. In SDSA, there are only mask and sparse additions. Version 1: Spike Q and K first perform element-wise mask, i.e., Hadamard product; then column summation and spike neuron layer are adopted to obtain the binary attention vector; finally, the binary attention vector is applied to the spike V to mask some channels (features). Version 2: An equivalent version of Version 1 (see Section (3.) reveals that SDSA is a unique type of linear attention (spiking neuron layer the kernel function) whose time complexity is linear with both token and channel dimensions. Typically, performing self-attention in VSA and SDSA requires 2N D multiply-and accumulate aaccuulla and 0.02.N accumulate operations respectively, where N is the number of tokens, D is the channel dimensions, 0.02 is the non-zero ratio of the matrix after the mask of Q and K. Thus, the between the spike Q, K, and V has almost no energy consumption.
operator
by vanilla Transformer components and AC operations caused by spiking neurons both exist in the existing spiking Transformers. One popular approach is to replace some of the neurons in Transformer with spiking neurons do various of tasks [11 12 B
14
15
16,
17
18
19
20
21
22 and
keeping the MAC-required operations like dot- dot-product, softmax, scale, etc.
Though hybrid computing helps reduce the accuracy loss brought on by adding spiking neurons to the Transformer, it can be challenging to benefit from SNN's low energy cost, especially given that current spiking Transformers are hardly usable on neuromorphic chips. To address this issue, we propose novel Spike-driven Transformer that achieves the spike-driven nature of SNNs throughout the network while having great task performance. Two core modules of Transformer, Vanilla Self Celf-AAttntion (VSA) and Multi-Layer Perceptron (MLP), are re-designed to have a spike-driven paradigm.
The three input matrices for VSA are Query (Q), Key (K), and Value (V), (Fig, aa)). Q and K first perform similarity calculations to obtain the attention map, which includes three steps of matrix multiplication, scale and softmax. The attention map is then used to weight the V (another matrix multiplication). The typical spiking self-attentions in the current spiking Transformers 20 19 would convert Q, K, V into spike-form before performing two matrix multiplications similar to those in VSA. The distinction is that spike matrix multiplications can be converted into addition, and softmax is not necessary (20). But these methods not only yield large integers in the output (thus requiring an additional scale multiplication for normalization to avoid gradient vanishing), but also fails to exploit the full energy-efficiency potential of the spike-driven paradigm combined with self-attention.
We propose Spike-Driven Self-Attentin (SDSA) address these issues, including two aspects (see SDSA Version 1 in Fig. [b)): i) Hadamard product replaces matrix multiplication; ii) matrix column- wise summation and spiking neuron layer take the role of softmax and scale. The former can be considered as not consuming energy because the Hadamard product between spikes is equivalent to the element- wise mask. The latter also consumes almost no energy since the matrix to be summed column- column-by-colume by-column is very sparse (typically, the ratio of non-zero elements is less than 0.02). We also observe that SDSA a special kind of linear attention 23 24. i.e., Version of Fig. [b). this view, the spiking neuron layer that converts Q, K, and V into spike form is a kernel function.
2

---

Additionally, existing spiking Transformers 220 12 usually follow the SEW-SNN residual design 25], whose shorteut is spike addition and thus outputs multi-bit (integer) spikes. This shortcut can satisfy event-driven, but introduces integer multiplication. We modified the residual connections throughout the Transformer architecture as shortcuts between membrane potentials [26 27 to address this issue (Section 3.2). The proposed Spike-driven Transformer improves task accuracy on both static and neuromorphic event-based datasets. The main contributions of this paper are as follows:
We propose a novel Spike-driven Transformer that only exploits sparse addition. This is the first time that the spike-driven paradigm has been incorporated into Transformer, and the proposed model is hardware-friendly to neuromorphic chips.
We design a Spike-Driven Self-Attention (SDSA). The self-attention operator between spike Query, Key, Value is replaced by mask and sparse addition with essentially no energy consumption. SDSA is computationally linear in both tokens and channels. Overall, the energy cost of SDSA (including Query, Key, Value generation parts) is 87.2x lower than its vanilla self-attention counterpart.
We rearrange the residual connections so that all spiking neurons in Spike-driven Transformer communicate via binary spikes.
Extensive experiments show that the proposed architecture can outperform or comparable to State- Of-The-Art (SOTA) SNNs on both static and neuromorphic datasets. We achieved 77.1 accuracy on ImageNet- K, which is the SOTA result in the SNN field.
2 Related Works
Bio-inspired Spiking Neural Networks can profit from advanced deep learning and neuroscience knowledge concurrently [28 5 7 Many biological mechanisms are leveraged to inspire SNN's neuron modeling [ 29). learning rules 30 31]. etc. Existing studies have shown that SNNs are more
suited for incorporating with brain mechanisms, e.g., long short-term memory B2 33 attention
etc. Moreover, while keeping its own spike-driven benefits, SNNS have greatly improved its task accuracy by integrating deep learning technologies like network architecture 26 25 36),
34 27 35
gradient backpropagation [37 38), normalization [39 40), etc. Our goal is to combine SNN and Transformer architectures. One way is to discretize Transformer into spike form through neuron
equivalence [11 i.e., ANN2SNN, but this requires long simulation timestep and boosts the energy consumption. We employ the direct training method, using the first SNN layer the spike encoding layer and applying surrogate gradient training
Neuromorphic Chips. As opposed to the compute and memory separated processors used in ANNs. neuromorphic chips use non-von Neumann architectures, which are inspired by the structure and function of the brain [5] 7 28). Because of the choice that uses spiking neurons and synapses as basic units, neuromorphic chips
have unique features, such as highly parallel operation, collocated processing and memory. inherent scalability, and spike-driven computing, etc. Typical neuromorphic chips consume tens to hundreds of mWs [48]. Conv and MLP in neuromorphic chips are equivalent to a cheap addressing algorithm (49 since the sparse spike-driven computing, i.e., to find out which synapses and neurons need to be involved in the addition operation. Our Transformer design strictly follows the spike-driven paradigm, thus it is friendly to deploy neuromorphic chips.
Efficient Transformers. Transformer and its variants have been widely used numerous tasks, such as natural language processing [8] (50 [1] and computer vision [52] 53 (4]. However, deploying these models on mobile devices with limited resources remains challenging because of their inherent complexity [24] 55 Typical optimization methods include convolution and self-attention mixing
[55] (6]. Transformer's 's own mechanism (token mechanism [57] 58), self-attention [59 60], multi-
head 61 62] and so on) optimization, etc. An important direction for efficient Transformers is linear attention on tokens since the computation scale of self-attention is quadratic with token number. Removing the softmax in self-attention and re-arranging the computation order of Query. Key, and Value is the main way to achieve linear attention 60
66) For the spiking Transformer, softmax cannot exist, thus spiking Transformer be a kind of linear attention.
3

---

SPS
Spike-driven Encoder Block x
MP
Classifiation
Conv 2D
BN
Conv 20
Uo
SDSA
Limear
BN
Linear
BN
BN
Conv 2D
PSM
RPE
MLP
MP
Max Pooling
Spike- Drivee Self-Attention (SDSA)
Shortcut
Spike Neuron
Linear
Mask & Add
Layer
Element-wise
Linear
BN
K
Linear
BN
Add
Membrane
Linear
BN
Potential
Figure 2: The overview Spike-driven Transformer. We follow the network structure in (20] but make two new key designs. First, we propose Spike-driven Self-Attention (SDSA) module, which consists of only mask and sparse addition operations (Fig. [b)). Second, We redesign the shortcuts in the whole network, involving position embedding, self-attention, and MLP parts. As indicated by the red line, the shortcut is constructed before the spike neuron layer. That is, we establish residual connections between membrane potentials to make sure that the values in the spike matrix are all binary, which allows the multiplication of the spike and weight matrices be converted into addition operations. By contrast, previous works 220 19] build shortcut between spike tensors in different layers, resulting in the output of spike neurons as multi-bit (integer) spikes.
3 Spike-driven Transformer
We propose Spike-driven Transformer, which incorporates Transformer into the spike-driven paradigm with only sparse addition. We first briefly introduce the spike neuron layer, then introduce the overview and components of the drive-driven Transformer one by one.
The spiking neuron model is simplified from the biological neuron model 67 29
Leaky Integrate-
and-Fire (LIF) spiking neuron [1. which have biological neuronal dynamics and are easy to simulate on a computer, is uniformly adopted this work. The dynamics of the LIF layer 37 is governed by
U H| 1|+ X
Sit] Hea (U uan),
(2)
[[t] S(t] (BU(t) S(t)),
where denote the timestep. ((t means the membrane potential which is produced by coupling the spatial input information X and the temporal input H 1), where X can be obtained through operators such as Conv, MLP, and self-attention. When membrane potential exceeds the threshold
Web. neuron will fire a spike, otherwise will not. Thus, the spatial output tensor S[1] contains only 1 or 0. Hea(-) is Heaviside step function that satisfies Hea (x) 1 when x 2 0. otherwise Hea. (a)) = 0. [1] indicates the temporal output, where Vreset denotes the reset potential which
set after activating the output spiking. 3 < is the decay factor, if the spiking neuron does not fire,
the membrane potential U[t] will decay to [t).
3.1 Overall Architecture
Fig. shows the overview Spike-driven Transformer that includes four parts: Spiking Patch Splitting (SPS), SDSA, MLP, and linear classification head. For the SPS part, we follow the design 20). Given 2D image sequence I E XCC*HXX the Patch Splitting Module (PSM), i.e.. first four Conv layers, linearly projects and splits it into sequence of N flattened spike patches with D dimensional channel, where T (images are repeated T times in the static dataset as input), C, H. and W denote timestep, channel, height and width of the 2D image sequence. Another Conv layer

---

is then used to generate Relative Position Embedding (RPE). Together, the SPS part is written as:
v PSM (I),
I TERTXCHHXM , R (4)
SN(U),
E E RPE RPEERTXNXD Uo E RTNND
(5)
RPF. BN(Conv2d(s)),
(6)
Up = + RPE.
(7)
where and Un are the output membrane potential tensor of PSM and SPS respectively, S.(.)) denote
spike neuron layer. Note, in Eq before executing Comv22l()), ), 8 RRxN>D will transposed
into E R RTxCxHxW We pass the Un to the I.-boock Spike-driven Transformer encoder, which consists of a SDSA and a MLP block. Residual connections are applied to membrane potentials efficient approach to model the local-global
in both SDSA and MLP block. SDSA provides information of images utilizing spike Q, K, and V without scale and softmax (see Sec. (.3). A Global PPoliig (GAP) is utilized the processed feature from spike-driven encoder and outputs the D-dimension channel which will sent to the fully-com sulll-coonneted-llad (CH) to output the prediction Y. The three parts SDSA, MLP and CH can be written follows:
Classification Head
So SN., U/ (DIA(S + U1-1.
So E R TxNND
(8)
Uj E TXNxD ,1 1...L
(9)
S/ SN(U)),
S RRNNDD _= 1...
(10)
S = SN(MLP(S)) + U)).
S E RRNXD 1=
(11)
Y CH(GAP(SL)),
(12)
where Uj and S represent membrane potential and spike tensor output by SDSA at /-th layer.
3.2 Membrane Shortcut in Spike-driven Transformer
is a crucial basic operation in Transformer architecture. There are three Vanilla Res-SNN (39). similar vanilla Res-CNN [68), performs a shortcut between membrane potential and spike. Spike-Element Wise (SEW) Res-SNN (25) employs shortcut to connect the output spikes different layers. Membrane Shortcut (MS) Res-SNN [26], creating a shortcut between membrane potential of spiking neurons in various layers. There is no uniformly standard shorteut in the current SNN community, and SEW shortcut is adopted by existing spiking Transformers [20 12]. As shown in Eq. 7 Eq. 9 and Eq. 11 we leverage the membrane shorteut in the proposed Spike- driven Transformer for four reasons:
Residual connection
668
69
shorteut techniques in existing Conv-hased SNNs 27
Spike-driven refers the ability to transform matrix multiplication between weight and spike tensors into sparse additions. Only binary spikes can support the spike-driven function. However, the values in the spike tensors are multi-bit (integer) spikes, as the SEW shortcut builds the addition between binary spikes. By contrast, as shown in Eq. 8 Eq. T0 Eq, I1 SN is followed by the MS shortcut, which ensures that there are always only binary spike signals the spike tensor.
High performance. The task accuracy of OS-RResSSee SNN is higher than that of SEW-Res SWW-Res-SNN [6) [2] 25). also in Transformer- SNN (see Table 5lin in this work).
approach to optimize the membrane potential distribution. This is consistent with other neuroscience-inspiee methods to optimize the internal dynamics of SNNs, such complex spiking neuron design 770], attention mechanism (27). long
Bio-plausibility. MS shortcut can be understood
short-term memory 33 recurrent connection (71), information maximization [72]. etc.
Dynamical isometry, MS-Res-SMM Res-Re--NNNNN has been proven 26 satisfy dynamical isometry theory B
which is theoretical explanation of well-behaved deep neural networks 774 75
3.3 Spike-driven Self-Attention
Vanilla Self-Attention (VSA). Given a float-point input feature sequence X E NND, float-point Query (Q), Key (K), and Value (V) in NDD are calculated by three learnable linear matrices, respectively. Standard scaled dot-product self-attention (Fig. 2 is computed as [52]:
QK
VSA(Q, K, softmax
V.
(13)
Vd
5

---

where d = D/H is the feature dimension of one head and H is the head number, vd is the scale factor. The time complexity of VSA is O D + D).
Spike-Driven Self-Attention (SDSA) Version 1. As shown in Fig. (()) left part, given a spike input feature sequence 5 E x float-point Q, K, and V in R x x are calculated by three learnable linear matrices, respectively. Note, the linear operation here is only addition, because the input S is spike tensor. A spike neuron layer (-)(-) follows, converting Q, K, V into spike tensor Qs. Ks. and Vs. SDSA Version 1 (SDSA-VI) is presented as:
SDSA(Q, K, V) g(Qs, Ks) @ Vs= = SN (SUM. (Qs @ Ks)) @ Vs.
(14)
where @ is the Hadamard product, s(-) is used to compute the attention map, SUMc(-) represents the sum of each column. The outputs of both g()) and (-) are D-dimensional row vectors. The Hadamard product between spike tensors is equivalent to the mask operation.
Discussion on SDSA. Since the Hadamard product among Qs, Ks, and Vs in RN (we here assume T = 1 for mathematical understanding) can be exchanged, Eq. 14 can also be written as:
SDSA(Q, K, v) = Qs @ g(Ks, Vss) = Qs @ SN ES(SSMM, (Ks & Vs)).
(15)
Note, 14 and Eq [5] are equivalent functionally. In this view, Eq is a linear attention 23
63
whose computational complexity is linear in token number N because Kg and Vs can participate in calculate first. This is thanks to the softmax operation in the VSA is dropped here. The function of softmax needs to be replaced by the kernel function. Specific to our SDSA, SN((-) is the kernel function. Further, we can assume special case [62.. H D, i.e., the number of channels per head is one. After the self-attention operation is performed on the H heads respectively, the outputs are concatenated together. Specifically,
SDSA(Q', K*, Vř) SN(Q") V*) SN(Q") (NN(")))) (SN(Q') o SN(V'))
(16)
where Q', K', V in N= are the i-th vectors in Q, K, V respectively, @ is the dot product operation. The output of g(Kt, V1) is a scalar, 0 or 1. Since the operation between SW(Q')) and g(K', V*) is a mask, the whole SDSA only needs to be calculated // D times for g(K Vt). The computational complexity of SDSA is O(0 ND), which is linear with both N and D (see Fig. bb) right part). Vectors K" and V are very sparse, typically less than 0.01 (Table 2p. Together, the whole SDSA only needs about 0.02N times of addition, and its energy consumption is negligible.
Interestingly, Eq. 15 actually converts the soft vanilla self-attention hard self-attention, where the attention scores in soft and hard attention are continuous- and binary-valued, respectively [76]. Thus, the practice of the spike-driven paradigm in this work leverages binary self-attention scores to directly mask unimportant channels in the sparse spike Value tensor. Although this introduces slight loss of accuracy (Table 5) (-) consumes almost no energy.
4 Theoretical Energy Consumption Analysis
Three key computational modules in deep learning are Conv, MLP, and self-attention. In this Section, We discuss how the spike-driven paradigm achieves high energy efficiency on these operators.
Spike-driven in Cony and MLP. Spike-driven combines two properties, event-driven and binary spike-based communication. The former means that computation is triggered when the input is zero. The binary restriction in the latter indicates that there are only additions. In summary, in spike-driven Conv and MLP, matrix multiplication is transformed into sparse addition, which is implemented as addressable addition in neuromorphic chips [49]
Spike-driven in Self-attention. Qs. Ks. Vs in spiking self-attention involve two matrix multipli- cations. One approach is to perform multiplication directly between Qs. Ks. Vs. which is then converted to sparse addition, like spike-driven Conv and MLP. The previous work [20] did just that. We provide a new scheme that performs element-wise multiplication between Qs, Ks. Vs. Since all elements in spike tensors either 0 1, element multiplication is equivalent to a mask operation with no energy consumption. Mask operations can be implemented in neuromorphic chips through addressing algorithms 44) or AND logic operations (4.
Energy Consumption Comparison. The times of paaiing-point operations (FLOPs) is often used to estimate the computational burden in ANNs, where almost all FLOPs are MAC. Under the same

---

Table 1: Energy evaluation. F'Coooo and FLMEP represent the FLOPs of the Conv and MLP models in the ANNs, respectively. Rc. RM, R. R denote the spike firing rates (the proportion of non-zero elements in the spike matrix) various spike matrices. We give the strict definitions and calculation methods of these indicators in the Supplementary due to space constraints.
Vanilla
Spike-driven Transformer
Transformer 52
(This work)
First Conv
EMAC. F.Coooe
EMAC T Ro FL.Covv EAC- -.- -R- FLooo
SPS
Other Conv
EMAC- FCCon
Q K,V.
EMAC 3ND2
EAC - R
(QQ,KW
EMAC 2N2D
EAC -- ..N
Self-attention
Scale
Softmax
EMAC- .2N2
Linear
EMAC FLMIPO
EAC T Raso FLMIPO
Layer 1
EMAC FLMLP1
EAC T Ran FLMMP1
MLP
Layer 2
-MAC. FLMLP2
EAC T Rasz FLMLP2
architecture, the energy cost of SNN can be estimated by combining the spike firing rate and simulation timestep I if the FLOPs of ANN is known. Table [ shows the energy consumption of Conv, self-attention, and MLP modules of the same scale in vanilla and our Spike-driven Transformer.
5
Experiments
We evaluate our method on both static datasets ImtgeNet (77, CIFAR-10/100 78 and neuromorphic
datasets CIFARIO-DY, 77), DVS128 Gesture [8)
Experimental Setup on ImageNet. For the convenience of comparison, we generally continued the experimental setup (20). The input size is set to 224 x 224. The batch size is set to 128 or 256 during 310 training epochs with a cosine-decay learning rate whose initial value is 0.0005. The optimizer Lamb. The image divided into N 196 patches using the SPS module. Standard data augmentation techniques, like random augmentation, mixup, are also employed in training. Details of the training and experimental setup on ImageNet are given the supplementary material.
Accuracy analysis on ImageNet. Our experimental results on ImageNet are given in Table We first compare our model performance with the baseline spiking Transformer (i.e., SpikFormer [20). The five network architectures consistent with those in SpikFormer are adopted by this work. We can see that under the same parameters, our accuracies are significantly better than the corresponding baseline models. For instance, the Spike-driven Transformer-8-3 8-344 is 2.0% higher than SpikFormer-8 8-334. worth noting that the Spike-driven Transformer-8. 66.34M, which is 1.5% higher than the corresponding SpikFormer. We further expand the inference resolution to 288 x 288, obtaining 77.1 %, which is the SOTA result of the SNN field on ImageNet.
obtains 76.32% (input 224x224) with
We then compare our results with existing Res- SNNs. Whether is vanilla Res-SNN 39 , MS-
Table 2: Spike Firing Rate (SFR) of Spike-driven Self-attention in 8-512. Average SFR is the mean of SFR over 4, and SDSA blocks.
Res-SNN [26] or SEW-Res-SNN 25 the accu-
racy of Spike-driven Transformer-8 768 (77.1%)
27 also achieves 77.1% accuracy by plugging an ad- ditional attention auxiliary module [81] [2] in MS-Res-SNN. but destroys the spike-driven nature and requires more parameters (78.37M vs. 66.34M) and training time (1000epoch vs. 3 310epoch). Furthermore, the proposed Spike-
is the highest. Au-MS-RRe-SSNN
SDSA
Average SFR
Qs
0.0091
K3
0.0090
g(Qs, s. Kg)
0.0713
Vs
0.1649
Output of SDSA(.), V
0.0209
driven Transformer outperforms by more than 72% at various network scales, while Res-SNNs have lower performance with a similar amount of parameters. For example, Spike-driven Transformer- 6-512 (This work) vs. SEW-Res-SNN-34 vs. MS-Res-SNN-344 Param, 23,27M vs. 21.79M vs.
L.80M; Acc. 74.11% % vs. 67.04% 69.15%.
7

---

Table 3: Evaluation on ImageNet. Power is the average theoretical energy consumption when predicting an image from the test set. The power data in this work is evaluated according Table [ and data for other works were obtained from related papers. Spiking Transformer-L- represents model with encoder blocks and channels. "The input erops are enlarged 288 288 in inference. The default inference input resolution for other models is 224x224.
Spike
Param (M)
Power (mJ)
Time Step
Methods
Architecture
Acc
driven
Hybrid training
ResNet-34
21.79
250
61.48
TET 84
SEW-ResNet-34
21.79
68.00
Spiking ResNet
85
ResNet-50
25.56
70.93
350
72.75
tdBN 33
Spiking-ResNet-34
/
21.79
6.39
6
63.72
SEW-ResNet-34
21.79
4.04
67.04
SEW-ResNer-500
25.56
4.89
67.78
SEW ResNet 25
SEW-ResNee-101
44.55
8.91
68.76
SEW-ResNet-152
60.19
12.89
69.26
MS-ResNet-18
11.69
4.29
63.10
MS ResNet 26
MS-ResNet-34
21.80
5.11
69.42
MIS-ResNet-1044
77.28
10.19
76.02
Att-MS-ResNNtt
11.87
0.48
63.97
Att MS ResNet 27
Att-MS-ResNet-34
22.12
0.57
69.15
Att-MS-ResNet 104
78.37
7.30
77.08
ResNet
Res-CNN-100
77.28
54.21
76.87
Transformer
Transformer-8 8--11
29.68
41.77
80.80
Spiking Transformer- 8-384
16.81
7.73
70.24
Spiking (ransformer-6-511
23.37
9.41
72.46
Spikformer 20
Spiking Transformer---5111
29.68
11.57
73.38
Spiking Transformer-10-33 10-512
36.01
13.89
73.68
Spiking Transformer-
66.34
21.47
74.81
Spiking Transformer-8
16.81
3.90
72.28
Spiking Transformer-6-512
23.37
3.56
74.11
Spike-driven
Spiking Transformer-8-512
29.68
1.13
71.68
Transformer (Ours)
Spiking Transformer-8-511
29.68
4.50
74.57
Spiking Transformer- 10-512
36.01
5.53
74.66
Spiking Transformer-8 8-
66.34
6.09
77.07
Power analysis on ImageNet. Compared with prior works, the Spike-driven Transformer shines energy cost (Table We first make an intuitive comparison energy consumption in the SNN field. Spike-driven Transformer-8-511 (This work) vs. SEW-Res-SNN-ON vs. ASS-Res-NNN- Power, 4.50mJ vs. 4.89mJ vs. 5m1 Acc, 74.57% vs. 67.78% vs. 69.42%. That our model achieves +6. 79%% and +5.15% accuracy higher than previous SEW and MS Res-SNN backbones with lower energy consumption. What more attractive is that the energy efficiency of the Spike- driven Transformer will be further expanded as the model scale grows because its computational complexity linear in both token and channel dimensions. For instance, in an 8-layer network, as the channel dimension increases from 384 to 512 and 768. SpikFormer (20 has 1.98 x 17.8x(7.73)))) (7.73mJ/3. 7331//3900m) 2.57 (1..111..mmmm 57mm/4.mmmmmmm and 3.52 (21 11.44m/66.099 09mJ) higher energy consumption than our Spike- driven Transformer. At the same time, our task performance on these three network structures has improved by +2.0%, +1.2 % and +1.5%, respectively.
Then we compare the energy cost between Spike- driven and ANN Transformer. Under the same structure, such as 8-512, the power required by the ANN-ViT (41.77mJ) 9.3x that of the spike- driven counterpart (4.50mJ). Further, the energy advantage will extend to 36.7x if we set the Spike-driven version (1.13mJ) Although the accuracy T (here we are direct training) will be lower than T' 4, it can be compensated
Table 4: Energy Consumption of Self-attention. Ej and Eg (including energy consumption to generate Q. K, V) represent the power of self- attention mechanism in ANN and spike-driven.
Models
E1 (pJ)
Eg (pJ)
E1/ Eg
8-384
6.7c8
1.6e7
42.6
8-512
1.2c9
2.1c7
57.2
8-768
2.7c9
3.1c7
87.2

---

Table 5: Experimental Results CIFARI0/100, DVS128 Gesture and CIFARIO-DVS.
CIFARIO-DVS
DVSI28 Gesture
CIFAR-10
CIFAR-100
Methods
Spike-drives driven
T
Acc
Acc
T
Acc
T
Acc
IdBN
39
X
10
67.8
96.9
6
93.2
PLIF 87
/
20
74.8
20
97.6
8
93.5
Dspike
88
X
10
75.4
6
94.3
6
74.2
DSR [9
V
10
77.3
20
95.4
20
78.5
Spikformer
X
16
80.9
98.3
95.5
78.2
DIET-SNN
90
x
5
92.7
5
69.7
ANN(ResNet19)
X
1
94.97
1
75.4
ANN(Transfommrrd-88)
X
96.7
1
81.0
This Work
/ 16 80.0
99.3
4
95.6
4
78.4
by special training methods [66] in future work.
Sparse spike firing is the key for Spike-driven Transformer achieve high energy efficiency. As shown in Table the Spike Firing Rate (SFR) of the self-attention part is very low, where the SFR of Qs and QK are both less than 0.01. Since the mask (Hadamard product) operation does not consume energy, the number of additions required by the SUM, (Qs @ Ks) is less than 0.02NN D times. The operation between the vector output by g(Q.,, Ks) and Vs is still column mask that does not consume energy. Consequently, in the whole self-attention part, the energy consumption of spike-driven self-attention can be lower than 87.2x of ANN self-attention (see Table
Experimental results on CIFAR-10/100, CIFARIO-DVS, and DVS128 Gesture are conducted in Table 5 These four datasets are relatively small compared to ImageNet. CIFAR- 10/100 are static image classification datasets. Gesture and CIFARIO-DVS are neuromorphic action classification datasets, which need to convert event stream into frame sequences before processing. DVS128 Gesture is a gesture recognition dataset. CIFARIO-DV'S is neuromorphic dataset converted from CIFAR-10 by shifting image samples to be captured by the DVS camera. We basically keep the experimental setup in (20). including the network structure, training settings, etc., and details are given the supplementary material. As shown Table 5 we achieve SOTA results Gesture (99.3%) and CIFAR-10 (95.6%), and comparable results SOTA on other datasets.
Table 6: Studies on Spiking Transformer-2-51.
Model
CIFAR-10
CIFAR-100
Baseline (20)
93.12
73.17
+ SDSA
93.09 (-0.03)
72.83 (-0.34)
MS
93.93 (+0.81)
74.63 (+1.46)
This work
93.82 (+0.73)
74.41 (+1.24)
6.0
234
3/6
Ablation study. To im- plement the spike-driven paradigm in Transformer, we design a new SDSA module and reposition the residual connections the entire network based the SpikFormer (20.. We orga- nize ablation studies CI- FAR10/100 to analyze their impact. Results given in 5 We adopt spik- ing Transformer-2-512 12 the baseline structure. It can be observed that SDSA in- curs a slight performance loss. As discussed Section 3.3 SDSA actually masks some unimportant channels directly. Fig. we plot the attention maps (the detailed drawing method is given in the supplementary material). and we can observe: i) SDSA can optimize intermediate features, such as masking background information: ii) SDSA greatly reduces the spike firing rate of Vs, thereby reducing energy cost. On the other hand, the membrane shorteut leads to significant accuracy improvements, consistent
Attention Map
635
126
Input Image
E27
SM
100
Attention Map 2
am
Block
Block 2
3
Block
Figure 3: Attention Map Based on Spike Firing Rate (SFR). Vs is the Value tensor. V/S is the output of SDSA(.). The spike-driven self- attention mechanism masks unimportant channels in Vs to obtain V's. Each pixel on Vs and Vs represents the SFR at a patch. The spatial resolution of each attention map is 14 14 (196 patches). The redder the higher the SFR, the bluer the smaller the SFR.

---

with the experience of Conv-basee MS-SNN 26 25
Comprehensively, the proposed Spike-driven Transformer simultaneously achieves better accuracy and lower energy consumption (Table
Conclusion
We propose a Spike-driven Transformer that combines the low power of SNN and the excellent accuracy of the Transformer. There is only sparse addition in the proposed Spike-driven Transformer. To this end, we design a novel Spike-Driven Self-Attention (SDSA) module and rearrange the location of residual connections throughout the network. The complex and energy-intensive matrix multiplication, softmax, and scale in the vanilla self-attention are dropped. Instead, we employ mask, addition, and spike neuron layer realize the function of the self-attention mechanism. Moreover, SDSA has linear complexity with both token and channel dimensions. Extensive experiments are conducted on static image and neuromorphic datasets, verifying the effectiveness and efficiency the proposed method. We hope our investigations pave the way for further research on Transformer-base SNNs and inspire the design of next-genenn gext-generttion neuromorphic chips.
Acknowledgement
This work was supported by Beijing Natural Science Foundation for Distinguished Young Scholars (JQ21015), National Science Foundation for Distinguished Young Scholars (62325603), and National Natural Science Foundation of China (62236009, U22A20103).
References
[1] Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural Networks, 10(9):1659-1677 1997.
[2] Paul A Merolla, John Arthur, Rodrigo Alvarez-lcaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. million spiking-neuron integrated circuit with scalable communication network and interface. Science, 345(6197)668--777 2014.
[3] Mike Davies, Narayan Srinivasa, Tsung- Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82-99, 2018.
[4] Jing Pei, Lei Deng, al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):00-111, 2019.
[5] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607-6111 2019.
[6] Bojian Yin, Federico Corradi. and Sander M Bohté. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence, 3(10):905-99 913, 2021.
[7] Catherine D Schuman, Shruti R Kulkarni, Maryam Parsa, Parker Mitchell, Bill Kay, et al. Opportunities for neuromorphic computing algorithms and applications. Nature Computational Science, 101) 2022.
[8] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, pages 5998-6998999 -6008, 2017.
[9] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE transactions on Pattern Analysis and Machine Intelligence, 45(1):87-110, 2022.
[10] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s): 2022.
10

---

[11] Yudong Li, Yunlin Lei, and Xu Yang. Spikeformer: A novel architecture for training high- performance low-latency spiking neural network. arXiv preprint arXiv:2211.10686, 2022.
[12] Nathan Leroux, Jan Finkbeiner, and Emre Neftci. Online transformers with spiking neurons for fast prosthetic hand control. arXiv preprint arXiv:2303.11866, 2023.
[13] liqing Zhang, Bo Dong, Haiwei Zhang, Jianchuan Ding, Felix Heide, Baocai Yin, and Xin Yang. Spiking transformers for event- based single object tracking. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 8801-8810, 2022.
[14] Jiyuan Zhang, Lulu Tang, Zhaofei Yu, Jiwen Lu, and Tiejun Huang. Spike transformer: Monocular depth estimation for spiking camera. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, pages 34 14-52. Springer, 2022.
[15] Zhengkun Tian, Jiangyan Yi, Jianhua Tao, Ye Bai, Shuai Zhang, and Zhengqi Wen. Spike- Triggered Autoeeeereeeessivve Transformer for End-to-End Speech Recognition. In Proc. Interspeech 2020, pages 5026-5030, 2020.
[16] Rui-Jie Zhu, Qihang Zhao, and Jason K Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint arXiv:2302. 13939, 2023.
[17] Etienne Mueller, Viktor Studenyak, Daniel Auge, and Alois Knoll. Spiking transformer networks: A rate coded approach for processing sequential data. In 2021 7th International Conference on Systems and Informatics (ICSAI), pages 1-5, IEEE, 2021.
[18] Minglun Han, Qingyu Wang, Tielin Zhang, Yi Wang. Duzhen Zhang, and Bo Xu. Complex dy- namic neurons improved spiking transformer network for efficient automatic speech recognition. Thirtt-Seventh AAAI Conference on Artificial Intelligence (AAAI 2023), 2023.
[19] Shihao Zou, Yuxuan Mu, Xinxin Zuo, Sen Wang, and Li Cheng. Event-based human pose tracking by spiking spatiotemporal transformer. arXiv preprint S.Xiv:2303.09681.. 2023.
[20] Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. In The Eleventh International Conference on Learning Representations, 2023.
[21] Guangyao Chen, Peixi Peng, Guoqi Li, and Yonghong Tian. Training full spike neural networks via auxiliary accumulation pathway. arXiv preprint arXiv:2301. 2023.
[22] Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Han Zhang, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual learning for transformer-bar based spiking neural network. arXiv preprint arXiv:2304.1195: 2023.
[23] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
[24] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler, Efficient transformers: A survey. ACM Computing Surveys, 55(6):1-25) 28, 2022.
[25] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 041211556-2066 2021.
[26] Yifan Hu, Yujie Wu, Lei Deng, Man Yao, and Guoqi Li. Advancing residual learning towards powerful deep spiking neural networks. arXiv preprint arXiv:21 2021.
[27] Man Yao, Guangshe Zhao, Hengyu Zhang, Hu Yifan, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1P.1-18. 01 2023.
(28) Guoqi Li, Lei Deng, Huajin Tang, Gang Pan, Yonghong Tian, Kaushik Roy, and Wolfgang Maass. Brain inspired computing: A systematic survey and future trends. 2023.
11

---

[29] Eugene M Izhikevich. Simple model of spiking neurons. IEEE Transactions on Neural Networks, (46):1569-15722 2003.
[30] Tielin Zhang, Xiang Cheng, Shuncheng Jia, Mu-ming Poo, Yi Zeng, and Bo Xu. Self- backpropagation of synaptic modifications elevates the efficiency of spiking and artificial neural networks. Science Advances, 7(43)=eabh0146 2021.
[31] Yujie Wu, Rong Zhao, Jun Zhu, Feng Chen, Mingkun Xu, Guoqi Li. Sen Song, Lei Deng, Guanrui Wang. Hao Zheng, et al. Brain-inspired global-local learning incorporated with neuromorphic computing. Nature Communications, 13(1))65, 2022.
[32] Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-lest in networks of spiking neurons. Advances in Neural Information Processing Systems, 31, 2018.
[33] Arjun Rao, Philipp Plank, Andreas Wild, and Wolfgang Maass, long short-term memory for ai applications in spike-based neuromorphic hardware. Nature Machine Intelligence, 4(5):467-479 2022.
[34] Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin, Zhaoxu Yang, and Guoqi Li. Temporal-wise attention spiking neural networks for event streams classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10221- 10230, 2021.
[35] Man Yao, Jiakui Hu, Guangshe Zhao, Yaoyuan Wang, Ziyang Zhang. Bo Xu, and Guoqi Li. Inherent redundancy in spiking neural networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 16924-16934, 2023.
[36] Man Yao, Hengyu Zhang, Guangshe Zhao, Xiyu Zhang, Dingheng Wang, Gang Cao, and Guoqi Li. Sparser spiking activity can be better: Feature refine-and ande-anddmask spiking neural network for event-based visual recognition. Neural Networks, 166:410-423, 2023.
[37] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in Neuroscience, 12:331, 2018.
[38] Emre O Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of maadient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6): 2019.
[39] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11062-11070, 2021.
[40] Lei Deng, Yujie Wu, Yifan Hu, Ling Liang, Guoqi Li, Xing Hu, Yufei Ding, Peng Li, and Yuan Xie. Comprehensive snn compression using admm optimization and activity regularization. IEEE Transactions on Neural Networks and Leurning Systems, 2021.
[41] Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Leurning Representations, 2021.
[42] Jibin Wu, Chenglin Xu, Xiao Han, Daquan Zhou, Malu Zhang, Haizhou Li, and Kay Chen Tan. Progressive tandem learning for pattern recognition with deep spiking neural networks. IEEE Trunsuctions on Pattern Analysis and Machine Intelligence, 44(1 44(11):7824-7840 2021.
[43] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 1311-1318, 2019.
[44] Johannes Schemmel, Daniel Brüderle, Andreas Grübl, Matthias Hock, Karlheinz Meier, and Sebastian Millner. A wafer-scale neuromorphic hardware system for large-scale neural modeling. In 2010 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1947-1950. IEEE, 2010.
12

---

[45] Eustace Painkras, Luis A Plana, Jim Garside, Steve Temple, Francesco Galluppi, Cameron Patterson, David R Lester, Andrew D Brown, and Steve B Furber. Spinnaker: 1-w 18-core system-on-chip for caaasivvel--praallle neural network simulation. IEEE Journal of Solid-state Circuits, 4848):1943-1953, 2013.
[46] Ben Varkey Benjamin, Peiran Gao, Emmett McQuinn, Swadesh Choudhary, Anand R Chan drasekaran, Jean-Marie Bussat, Rodrigo Alvarez- Icaza, John V Arthur, Paul A Merolla, and Kwabena Boahen. Neurogrid: A mixed-analog- multichip system for large-scale neural simulations. Proceedings of the IEFE, 102(5):699 -1669999771611 2014.
[47] Juncheng Shen, De Ma, Zonghua Gu, Ming Zhang, Xiaolei Zhu, Xiaoqiang Xu, Qi Xu, Yangjing Shen, and Gang Pan. Darwin: A neuromorphic hardware co-processor based on spiking neural networks. Science China Information Sciences, 59(2):1-5, 2016.
[48] Arindam Basu, Lei Deng, Charlotte Frenkel, and Xueyong Zhang. Spiking neural network integrated circuits: A review of trends and future directions. In 2022 IEEE Custom Integrated Circuits Conference (CICC), pages 1 1-8, 2022.
[49] Ole Richer, Ning Qiao, Qian Liu, and Sadique Sheik. Event-driven spiking convolutional neural network. WIPO Patent, page WO2020207982AI, 2020.
[50] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[51] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1810:008100 2018.
[52] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representa- fions (ICLR), 2020.
[53] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[54] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficien image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[55] Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang. and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems, 35: 19934-11229,, 2022.
[56) Haokui Zhang, Wenze Hu, and Xiaoyu Wang. Parc-net: Position aware circular convolution with merits from convnets and transformer. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, pages 613-630. Springer, 2022.
[57] Yongming Rao, Zuyan Liu, Wenliang Zhao, Zhou, and Jiwen Lu. Dynamic spatial sparsifi- cation for efficient vision transformers and convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[58] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 558-567, 2021.
[59] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's ss clothing for faster inference. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1 12259-
12269, 2021.
13

---

[60] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021.
[61] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in Neural Information Processing Systems, 32, 2019.
[62] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient attention with many heads. In Computer Vision-FCCC 2022 Workshops: Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, pages 35-49. Springer, 2023.
[63] Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. In International Conference on Learning Representations, 2022.
[64] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVÉ Winter Conference on Applications of Computer Vision, pages 3531-3539, 2021.
[65] Hao Peng. Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. In International Conference on Learning Representations, 2021.
[66] Abdelrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang. and Fahad Shahbaz Khan. Swiftformer: Efficient additive attention for transformer-based real-time mobile vision applications. arXiv preprint arXiv:2303. 15446, 2023.
[67] A. L. Hodgkin and A. F. Huxley. A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of Physiology. 117(4):500-544 1952.
[68] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[69] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision ECCV 2016, pages 630-645, Cham, 2016. Springer International Publishing.
[70] Xingting Yao, Fanrong Li, Zitao Mo, and Jian Cheng. GLIF: A unified gated leaky integrate- and-fire neuron for spiking neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Prucessing Systems, 2022.
[71] Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate and efficient time domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence, 3(10):905991,, 2021.
[72] Yufei Guo, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Yinglei Wang, Xuhui Huang, and Zhe Ma. Im-loss: information maximization loss for spiking neural networks. Advances in Neural Information Processing Systems, 35: 156-166, 2022.
[73] Zhaodong Chen, Lei Deng, Bangyan Wang, Guoqi Li, and Yuan Xie. A comprehensive and modularized statistical framework for gradient norm equality in deep neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1): 2022.
(74] Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pen- nington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning, pages 5393-5402. PMLR, 2018
[75] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2003.00555, 2022.
14

---

[76] Meng-Hao Guo, Tian-Xing Xu, Jiang-Jiam Jiang Liu, Zheng-Ning Liu, Peng- Tao Jiang, Tai-Jiang Mu, Song-Hal Zhang. Ralph Martin, Ming-Ming -Ming Cheng, and Shi-Min Hu. Attention mechanisms in computer vision: A survey. Computational Visual Media, 8(3):331 2022.
[77] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255 2009.
[78] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[79] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. stream dataset for object classification. Frontiers in Neuroscience, 11:309, 2017.
event-
[80] Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, Jeff Kusnitz, Michael Debole, Steve Esser, Tobi Delbruck, Myron Flickner, and Dharmendra Modha. A low power, fully event-based gesture recognition system. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition (CVPR), pages 7243-7252, 2017.
[81] Jie Hu, Li Shen, and Gang Sun. Squeezz-and-eeciiation networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7132-7141, 2018.
[82] Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3-19, 2018.
[83] Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations, 2020.
[84] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. In International Conference on Learning Representations, 2022.
[85] Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, pages 1-6, 2021.
[86] Sayeed Shafayet Chowdhury, Nitin Rathi, and Kaushik Roy. One timestep is all you need: training spiking neural networks with ultra low latency. arXiv preprint arXiv:2110.05929 2021.
[87] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2661-2671, 2021.
[88] Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng. Yongging Hai, and Shi Gu. Differ- entiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks. In Pro- ceedings of the International Conference on Neural Information Processing Systems (NeurIPS), volume 34, pages 13426-23439) 2021.
[89] Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi Zhi-Quan Luo. Training high-performance low-latency spiking neural networks by differentiation on spike representation. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 12453,, 2022.
[90] Nitin Rathi and Kaushik Roy. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEF Transactions on Neural Networks and Learning Systems, pages 1-9, 2021.
[91] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Praning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017.
15

---

[92] Souvik Kundu, Massoud Pedram, and Peter A Beerel. snn: Harnessing the inherent robustness of energy-effiiiee deep spiking neural networks by training with crafted input noise. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5209-5218, 2021.
[93] Priyadarshini Panda, Sai Aparna Aketi, and Kaushik Roy. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience, 14:653, 2020.
[94] Mark Horowitz. 1.1 computing's s energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pages 10-14. IEEE, 2014.
[95] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger. Andrew J. Davison, Jörg Conradt, Kostas Daniilidis, and Davide Scaramuzza. Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):154-180, 2022.
[96] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618-626. 2017.
16


Reviews:
[{"review_id": "ixkW7EFouU", "reviewer": "Reviewer_GiWX", "summary": "The authors propose a variant of transformer networks with spiking neurons based on the LIF neuron. The submission reformulates the self-attention to use sparse addition and masking, and modifies the residual connections to transmit information in the domain of membrane potentials.\nThey achieve a state of the art result on Imagenet.\nThey present an energy analysis.\n", "questions": "Can the authors provide an additional explanation for equations 15 and 16?", "limitations": "not much discussed", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It increases the amount of binary spike-based computations in the transformer.\nThe experimental results are strong.\nA small ablation study is performed.", "weaknesses": "The explanation for  equations 15 and 16 can be improved\nThe explanation of power estimates can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback and your time in reading our paper. Due to space constraints in the paper, the explanations for Eq.(15) and (16) are rough, and we put some details of energy consumption evaluation in the Supplementary Material. After careful inspection, we found that there is a typo in Eq.(16), we accidentally wrote $V^{i}$ instead of $Q^{i}$. We're sorry for this and hope that the responses below can answer your concerns.\n\n> *Question*: Can the authors provide an additional explanation for equations 15 and 16?\n\n**A**: Here we first briefly introduce three typical attention mechanisms in existing Transformers from the perspective of computational complexity: Vanilla Self-Attention(VSA) [1], Linear Attention [2], Hydra Attention [3]. Then, we introduce the proposed Spike-Driven Self-Attention (SDSA) and analyze its computational complexity.\n\n***\nWe can understand self-attention from the perspective of computational complexity. Specifically, two matrix multiplications between float-point $Q$, $K$, $V$ in $\\mathbb{R}^{N\\times D}$ are included in the VSA, where $N$ is the token number, $D$ is the channel dimension. Generally, VSA performs multi-head self-attention, i.e., divide $Q$, $K$, $V$ into $H$ heads in the channel dimension. In the $i$-th head, $Q^{i}$, $K^{i}$, $V^{i}$ in $\\mathbb{R}^{N \\times D/H}$. After the self-attention operation is performed on the $H$ heads respectively, the outputs are concatenated together.\n\n(1) Vanilla self-attention [1]. $Q$ and $K$ are matrix multiplied first, and then their output is matrix multiplied with $V$. The computational complexity of VSA is $O(N^2D)$, which has a **quadratic** relationship with the toke number $N$.\n\n(2) Linear attention [2,3]. $K$ and $V$ are matrix multiplied first, and then their output is matrix multiplied with $Q$. The computational complexity of linear attention is $O(ND^2/H)$, which has a **linear** relationship with the toke number $N$.\n\n(3) Hydra attention [3]. Consider an extreme case in linear attention, set $H=D$. That is, in each head, $Q^{i}$, $K^{i}$, $V^{i}$ in $\\mathbb{R}^{N \\times 1}$. Then the computational complexity of hydra attention is $O(ND)$, which has a **linear** relationship with both the toke number $N$ and the channel dimension $D$.\n\n***\n(4) **Spike-driven self-attention (This work)**. We show that SDSA has the same computational complexity as hydra attention (Lines179-184), i.e., $O(ND)$. We here assume $T=1$ for mathematical understanding. Note, in SNN, a spike neuron layer $SN(\\cdot)$ first converts $Q$, $K$, $V$ into spike tensor $Q_{S}$, $K_{S}$, and $V_{S}$. The proposed SDSA Version 1 (SDSA-V1) is executed as: \n\n$SDSA(Q, K, V)$ = $g(Q_{S}, K_{S}) \\otimes V_{S}$ = $SN(SUM_{c}(Q_{S} \\otimes  K_{S})) \\otimes V_{S}$ (14)\n\nwhere $\\otimes$ is the Hadamard product, $g(\\cdot)$ is used to compute the attention map, $SUM_{c}(\\cdot)$ represents the sum of each column. The outputs of both $g(\\cdot)$ and $SUM_{c}(\\cdot)$ are $D$-dimensional row vectors. The Hadamard product between spike tensors is equivalent to the mask operation. **Since the Hadamard product among $Q_{S}$, $K_{S}$, and $V_{S}$ can be exchanged**, Eq.(14) can also be written as (SDSA-V2):\n\n$SDSA(Q, K, V)$ = $Q_{S} \\otimes g(K_{S}, V_{S})$ = $Q_{S} \\otimes SN(SUM_{c}(K_{S} \\otimes  V_{S}))$ (15)\n\nIn Eq.(15), $K_{S}$ and $V_{S}$ participate in the operation first, **thus it is a kind of linear attention. Further, we consider the special operation of Hadamard product.** Specifically, the output of $SUM_{c}(K_{S} \\otimes  V_{S})$ is a $D$-dimensional row vector, and the value of the $i$-th element in $SUM_{c}(K_{S} \\otimes  V_{S})$ is $D_{i}$, then\n\n$D_{i}$ = $SUM_{c}(K_{S}^{i} \\otimes  V_{S}^{i})$ = $(K_{S}^{i})^{\\rm{T}}  \\odot  V_{S}^{i}$ = $SN(K^{i})^{\\rm{T}} \\odot SN(V^{i})$,\n\nwhere $\\odot$ is the dot product operation, $K_{S}^{i} = SN(K^{i})$ and $V_{S}^{i} = SN(V^{i})$ are the $i$-th column vectors in $K_{S}$ and $V_{S}$, respectively. **In simple terms, taking the Hadamard product of two column vectors $a$ and $b$ and summing them is equivalent to multiplying $b$ times the transpose of $a$, i.e., $SUM_{c}(a \\otimes b) = a^{\\rm{T}} \\odot b$**. Therefore, we can get the V2 version of SDSA:\n\n$SDSA(Q^{i}, K^{i}, V^{i})$ = $SN(Q^{i})g(K^{i}, V^{i})$ = $SN(Q^{i})SN(SUM_{c}(K_{S}^{i} \\otimes  V_{S}^{i}))$ = $SN(Q^{i}) SN(SN(K^{i})^{\\rm{T}} \\odot {SN}(V^{i}))$ (16)\n\nwhere $Q^{i}, K^{i}, V^{i}$ in $\\mathbb{R}^{N\\times 1}$ are the $i$-th vectors in $Q, K, V$ respectively. The output of $g(K^{i}, V^{i})$ is a scalar, 0 or 1. Since the operation between $SN(Q^{i})$ and $g(K^{i}, V^{i})$ is a mask, the whole SDSA only needs to calculate $D$ times for $g(K^{i}, V^{i}) = SN(D_{i})$. Note that only $N$ additions need to be performed in $D_{i}$ = $SUM_{c}(K_{S}^{i} \\otimes  V_{S}^{i})$. Thus, the computational complexity of SDSA is $O(0+ND)$, which is linear with both $N$ and $D$. Vectors $K_{S}^{i}$ and $K_{S}^{i}$ are very sparse, typically less than 0.01 (see Supplementary Material). Together, the whole SDSA only needs about $0.02ND$ times of addition. \n\n***\n[1] Attention is all you need. In: NeurIPS (2017)\n\n[2] Transformers are RNNs: fast autoregressive transformers with linear attention. In: ICML (2020)\n\n[3] Hydra attention: Efficient attention with many heads. In: ECCV (2022)"}, {"review_id": "3mIxWae8pq", "reviewer": "Reviewer_Lu4B", "summary": "This paper proposes an improved spike transformer by replacing Spike-Element-Wise shortcut in an existing spike transformer (ref [20]) with Membrane Shortcut from spike ResNet (ref [26]) .", "questions": "The outputs of LIF spiking neurons at all timesteps are used as the inputs to the Transformer. This may be the tested method in [20], but why is this a better idea than using the output at the last timestep? As shown in table-2, the power is directly proportional to the timestep count. So this seems a critical parameter to optimize over.\n\nIn Eq (12), S_L has the shape TxNxD. So outputs at all timesteps are used in classification head?\n\nIn Eq (6), the input s has no 2d spatial dimensions. What are the 2d dimensions in Conv2d()?", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "Match SOTA ImageNet top-1 precision achieved by ResNet [26] with slightly lower estimated energy consumption.", "weaknesses": "Combining known techniques from two papers without modification is considered an incremental improvement.\nIt is not clear why Transformer is a better choice than ConvNet in the context of Spike Neural Network. In ViT papers, the motivation for the Transformer is its model capacity can scale easily to handle large datasets and it has much better parallelism on TPU/GPU than ConvNet. None of these has been indicated as the goal for SNN. As shown in table-2, the ResNet SNN model in ref[27] is equally competitive as the proposed model.\nThe notation is a bit confusing in a few equations. The s in Eq (5) is binary. This should be distinguished from floating-point numbers in Eq (4). But the same symbol R is used for both. The same observation holds for Eq (8)(10)(11).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We first list your advice and questions, then give our detailed answers. **And, we really hope that you would re-consider your rating, given this work's notable contribution to the SNN field.**\n\n> *Summary and Weakness 1*: This paper proposes an improved spike transformer by replacing Spike-Element-Wise shortcut in an existing spike transformer (ref [20]) with Membrane Shortcut from spike ResNet (ref [26]). Combining known techniques from two papers without modification is considered an incremental improvement.\n\n**A**: We would like to discuss the contribution and significance of this work with you in depth.\n\n(1) **Goals and status quo in SNN.** The ambition of SNNs is to be a low-power alternative to ANNs. The key to low-power of SNN is the **spike-driven**. There are **two main hurdles** in achieving this goal: First, there is a performance gap between SNNs and ANNs; Second, the low-power of SNNs can only be truly realized when run on a neuromorphic chip, thus SNNs are in fact a computing paradigm that requires algorithm-hardware co-design.\n\n(2) **Technical Contributions.** We incorporate the spike-driven paradigm into Transformer by the proposed Spike-driven Transformer.\n\n- **SDSA operator.** There are only two types of operators in SNN, spike-driven Conv and MLP. **SDSA is the first spike-driven Self-attention operator** that implements the self-attention using mask and addition without any multiplication (e.g., softmax and scale). SDSA is computationally linear in both tokens and channels. Its energy cost is $87.2\\times$ lower than vanilla self-attention.\n\n- **Rearrange shortcuts.** We use membrane potential shortcuts to avoid integer (multi-bit spikes).\n\n- **Spike-driven Transformer** has only sparse addition and achieves SOTA performance on ImageNet.\n\n(3) **Significance.**\n\n- **Energy efficiency.** Compared with the prior SNNs with the same parameter amount, spike-driven Transformer has higher performance but lower energy consumption. Compared with the ANN counterpart, the energy efficiency of spike-driven Transformer is as high as $36.7 \\times$.\n\n- **Performance.** Achieved a breakthrough from 0 to 1, we incorporated the spike-driven paradigm into Transformer and achieved SOTA results. There is huge room for future performance improvement.\n\n- **Algorithms drive the hardware design.** SDSA is the first spike-driven self-attention operator, which is expected to advance the design of next-generation neuromorphic chips.\n\n**Thus, the contribution of this work is by no means a mere substitution of shortcuts on the basis of prior work. We can confidently say that this work will change the field of SNNs in terms of algorithm and neuromorphic chip design.**\n\n> *Weakness 2*: It is not clear why Transformer is a better choice than ConvNet in the context of Spike Neural Network. In ViT papers, the motivation for the Transformer is its model capacity can scale easily to handle large datasets and it has much better parallelism on TPU/GPU than ConvNet. None of these has been indicated as the goal for SNN. As shown in table-2, the ResNet SNN model in ref[27] is equally competitive as the proposed model.\n\n**A**: This is an important issue. In fact, the proposed architecture is a Conv and ViT hybrid network to extract information in a local-global manner. This is somewhat similar to MetaFormer.\n\n- **MetaFormer[1,2]** argues that there is general architecture abstracted from Transformers by not specifying the token mixer. Setting the operators in the token mixer to Identity mapping, Conv, MLP, or Self-attention will result in different accuracies. The results show that the Self-attention, which can extract global information, can achieve the highest performance.\n\n- **Spike-driven Transformer** can be considered as a kind of MetaFormer (CAFormer in [2], Conv + ViT). The proposed novel SDSA is a token mixer.\n\nWe argue that from the perspective of MetaFormer, SDSA operators bring global information to the network based on the infrastructure. Interestingly, the attention SNN in ref[27] also adds a global attention module on vanilla Res-SNN. In contrast to Multiply-and-Accumulate attention module in ref[27], the global SDSA operator is purely additive. Given the fact that this work is the first to incorporate spike-driven into Transformer, it is reasonable to think that it has more room for improvement.\n\n> *Weakness2,Q3*: The notation is a bit confusing in a few equations. The s in Eq (5) is binary. This should be distinguished from floating-point numbers in Eq (4). But the same symbol R is used for both. The same observation holds for Eq (8)(10)(11). In Eq (6), the input s has no 2d spatial dimensions. What are the 2d dimensions in Conv2d()?\n\n**A**: We apologize for the confusion caused by the imprecise notation. In Eq.(6), before executing $Conv2d(\\cdot)$, $s\\in \\mathbb{R}^{T\\times N\\times D}$ will be transposed into $s\\in \\mathbb{R}^{T\\times C\\times H \\times W}$.\n\n> *Q1,2*: The outputs of LIF spiking neurons at all timesteps are used as the inputs to the Transformer. This may be the tested method in [20], but why is this a better idea than using the output at the last timestep? As shown in table-2, the power is directly proportional to the timestep count. So this seems a critical parameter to optimize over. In Eq (12), S_L has the shape TxNxD. So outputs at all timesteps are used in classification head?\n\n**A**: Yes, in SNN, it is a default operation to use the output of all timesteps for classification [3], because it is more accurate. Generally, for image classification, the larger the timestep, the higher the performance and inference energy cost, and the training cost also increases rapidly. Therefore, this is a trade-off problem.\n\n---\n[1] Metaformer is actually what you need for vision. In CVPR 2022.\n\n[2] Metaformer baselines for vision. In arXiv 2022.\n\n[3] Temporal efficient training of spiking neural network via gradient re-weighting. In ICLR 2022."}, {"review_id": "qxeZ1m2gE6", "reviewer": "Reviewer_8FkB", "summary": "This paper proposed a Spike-Driven Self-Attention (SDSA) module, which uses Hadamard product, column-wise summation, and spiking neuron layer to replace the matrix multiplication and softmax operation. Experiments on static and neuromorphic image classification demonstrate competitive performance and energy efficiency.", "questions": "1. The ablation study shows the membrane shortcut brings significant accuracy improvement. I slightly doubt the importance of the SDSA. Can the authors try the combination of the membrane shortcut and some attention-free transformers like [1][2] to check if they can also achieve similar performance?\n\n\n[1] J Lee-Thorp et al., “FNet: Mixing Tokens with Fourier Transforms”, NAACL 2022\n[2] W Yu et al, “MetaFormer Is Actually What You Need for Vision”, CVPR 2022", "limitations": "The paper has few limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.. The authors proposed a novel form of linear self-attention module, SDSA, which leverages the features of the spikes and increases computational and memory efficiency.\n2. The authors show extensive results from different models and different datasets. The performance of accuracy and energy efficiency is very strong.\n3. SDSA reduces the computational complexity with a slight accuracy drop, and the authors show some attention maps to validate the effectiveness of the SDSA.", "weaknesses": "1. There is still a noticeable drop between the spike-driven transformer and the original transformer. Can the authors show some results of the model with more time steps and report the limitations of the model?\n2. Can the authors show the performance and energy consumption as the number of time steps decreases?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We first list your advice and questions, then give our detailed answers.\n\n> *Weakness 1 and 2:* There is still a noticeable drop between the spike-driven transformer and the original transformer. Can the authors show some results of the model with more time steps and report the limitations of the model? Can the authors show the performance and energy consumption as the number of time steps decreases?\n\n**A:** Generally, in directly training SNNs, for image classification, the larger the timestep, the higher the performance and inference energy cost, and the training cost also increases rapidly. However, as the timestep increases, the performance gain becomes smaller and smaller. Therefore, this is a trade-off problem. In existing deep SNNs [3,4,5], researchers generally exploit $T=4$ by default. Due to time and resource constraints, We only tested up to $T=6$ on spiking Transformer-384.\n\n| Timestep | $T=1$ | $T=2$ | $T=4$ | $T=6$|\n|---|---|---|---|---|\n| Acc (\\%) | 71.7 | 72.9 | 74.6 | 74.8 |\n| Power (mJ) | 1.13 | 2.33 | 4.50 | 7.17 |\n\n**A major limitation of SNNs is that, although SNNs are theoretically highly energy-efficient, they often require specialized hardware (such as neuromorphic chips [6]) for verification.** Thus, SNNs are in fact a sparse computing paradigm that requires algorithm-hardware co-design. This is why we believe that this paper will have a profound impact on the SNN field. **First, our model raises the performance ceiling in the field of SNNs**. Given the fact that this work is the first to incorporate spike-driven into Transformer, it is reasonable to think that it has more room for improvement. **On the other hand, to the best of our knowledge, the proposed Spike-Driven Self-Attention (SDSA) operator is the third class of existing operators besides spike-driven Conv and MLP. So, we think our work will also have a profound impact on future neuromorphic chip design.**\n\n> *Question:* The ablation study shows the membrane shortcut brings significant accuracy improvement. I slightly doubt the importance of the SDSA. Can the authors try the combination of the membrane shortcut and some attention-free transformers like [1][2] to check if they can also achieve similar performance?\n\n**A:** **We would be happy to discuss this with you. MetaFormer has inspired us a lot, and we understand your idea.** We organized some experiments, set $T=1$, 100epoch, and spiking Transformer 8-384.The experimental results are as follows:\n\n| Model | Baseline(SDSA, This work) | +SSA [7] | +Fourier [1] | +Pooling [2] |\n|---|---|---|---|---|\n| Acc (\\%) | 61.0 | 63.7 | Not convergent | 41.2 |\n\nWe suspect that the performance gap between Baseline(SDSA) and +SSA is due to insufficient training. Therefore, we set $T=4$ and 310epoch and retrain. The experimental results are as follows:\n\n| Model | Baseline(SDSA, This work) | +SSA [6] |\n|---|---|---|\n| Acc (\\%) | 72.3 | 72.7 |\n\nAccording to our understanding of MetaFormer and the above experimental results, the following preliminary conclusions can be drawn:\n\n(1) **We think that spike-driven Transformer can be considered as a kind of MetaFormer (CAFormer in [8], Conv + ViT). The proposed SDSA is a token mixer.** In this perspective, when we replace the token mixer from SDSA (this work) to SSA[7], the accuracy of both $T=1$ and $T=4$ will improve. However, it should be noted that there is only binary spike-based addition in the proposed SDSA and the computational complexity is $O(ND)$, while the SSA in [7] contains multiplication (multi-bit integer multiplication and scale operations) and the computational complexity is $O(ND^2)$.\n\n(2) **Directly employing the Fourier or Pooling operator to replace SDSA as the token mixer does not work well.** The reason has not yet been confirmed. But we think it will be a good perspective to understand and improve spike-driven Transformer in the future.\n\n---\n[1] FNet: Mixing Tokens with Fourier Transforms, In NAACL, 2022.\n\n[2] MetaFormer Is Actually What You Need for Vision. In CVPR 2022.\n\n[3] Deep Residual Learning in Spiking Neural Networks. In NeurIPS, 2021.\n\n[4] Temporal efficient training of spiking neural network via gradient re-weighting. In ICLR 2022.\n\n[5] Attention Spiking Neural Networks. In IEEE T-PAMI, 2023.\n\n[6] Towards artificial general intelligence with hybrid Tianjic chip architecture, In Nature, 2019.\n\n[7] Spikformer: When Spiking Neural Network Meets Transformer. In ICLR, 2023.\n\n[8] Metaformer baselines for vision. In arXiv 2022."}, {"review_id": "ShYKrFQugd", "reviewer": "Reviewer_qknk", "summary": "This paper proposes a Spike-driven Transformer that incorporates the spike-driven paradigm of SNNs into the Transformer and is hardware-friendly for neuromorphic chips. The authors use SDSA to transform the multiplication operation between Query, Key, and Value into a combination of mask and sparse addition operations and modify the self-attention structure to achieve linear complexity. As a result, this approach can achieve up to 87.2× lower energy consumption compared to the traditional self-attention method. Furthermore, membrane shortcut is introduced to ensure all spiking neurons communicate via binary spikes. The paper provides insights into implementing each module on neuromorphic chips and achieves competitive results on multiple datasets while reducing energy consumption compared to previous works.", "questions": "How is the membrane shortcut implemented on a neuromorphic chip?  Is it necessary to use it instead of SEW shortcut? The article explains that membrane shortcuts are used to optimize the distribution of membrane potential, but for spiking neurons, wouldn't it be more appropriate to perform such an optimization in the time domain?", "limitations": "No", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The authors introduce the Spike-Driven Self-Attention (SDSA) module as a replacement for the self-attention module in the current spiking Transformer. The proposed module exhibits superior performance while significantly reducing energy consumption, resulting in an architecture that is better suited for deployment on neuromorphic chips.", "weaknesses": "This paper explores the feasibility of implementing Conv, MLP, and Self-attention models driven by spikes on neuromorphic chips. However, it does not delve into the implementation details of membrane shortcuts. This approach involves the direct transmission of membrane potential signals between spiking neurons and may contradict the spike-driven approach discussed in the original text.\n\nThe effectiveness of a spikformer-based framework with only SDSA, without MS, remains unclear as most experiments in the literature combine both techniques. This weakens the argument for the performance of the spike-driven Transformer proposed in this paper. Although an ablation study was conducted on MS and SDSA in Table 5, it was limited to the CIFAR10/100 dataset and Spiking Transformer-2-512. The results suggest that the use of MS may be crucial to achieve state-of-the-art performance.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We have carefully studied your comments and argue that **your concerns can be addressed**. \n\nWe incorporate the spike-driven paradigm into Transformer. In spike-driven Transformer, there are only sparse additions. This limitation is quite severe, but we still achieved SOTA on ImageNet in SNN. To achieve this, we designed a novel Spike-Driven Self-Attention (SDSA) and tuned shortcuts. **What's more, the SDSA operator is the third class of operators besides spike-driven Conv and MLP, which will inspire future neuromorphic chip designs.** Since this work is the first to incorporate spike-driven into Transformer, it is reasonable to think that it has more room for improvement.\n\n**This work will undoubtedly impact the SNN domain in algorithm and neuromorphic chip design. Given the significant technical contributions and importance of this work to the field, we sincerely hope that you will reconsider your rating** \n\n> *W1 and Q1*. it does not delve into the implementation details of Membrane Shortcuts (MS). This approach... How is MS implemented on a neuromorphic chip? \n\n**A: The MS does not conflict with spike-driven** \n\n- **Spike-driven** implies spike-based event-driven, which restricts all multiplications associated with spike tensors to be implemented as additions.\n\n- **Spike-driven on a neuromorphic chip.** Spike-driven paradigm is implemented on neuromorphic chips in the form of **addressable algorithms[1]**. The standard protocol for spike communication is the asynchronous Address-Event Representation (AER), from simple point-to-point links to complex networks-on-a-chip. So, spike-driven router occupies an important position in chip design. \n\n- **MS on a neuromorphic chip.** The design of a spike-driven router is affected by many factors, e.g., chip architecture, layout, clock, manufacturing process, applications. Let's take Speck of SynSense as an example to illustrate how to implement MS. Speck is an asynchronous chip focused on processing event streams. In Speck [2], Whenever an event arrives at an SNN core with its address information, the corresponding kernel value and destination neuron position are obtained by address searching, the destination neuron states are then asynchronously updated according to the synaptic operation. And, asynchronous spike-driven convolution is independent to the arrival of other input events and cores, the operation can be efficiently parallel distributed for multiple events at different positions. Specific to using MS in Speck, when a spiking neuron receives a spike, the membrane potential must change. Then, another addressing function can be used to pass this membrane potential to the corresponding neuron in the subsequent layer for merging.\n\n> *Q2*. Is it necessary to use MS shortcut instead of SEW shortcut? \n\n**A:** It depends on the specific situation. We give a very strict restriction that there can only be the addition in the whole Transformer. In this case, the SEW cannot exist because it would bring integer multiplication. If we relax this restriction or there are neuromorphic chips that support integer multiplication, SEW can also exist.\n\n> *Q3*. The article explains that membrane shortcuts are used to optimize the distribution of membrane potential, but for spiking neurons, wouldn't it be more appropriate to perform such an optimization in the time domain?\n\n**A:** It is indeed feasible to perform membrane potential optimization in the time domain on some small time-domain tasks, e.g., using long short-term memory[3]. But when processing large static datasets, such as ImageNet, where deep SNNs are required, membrane potential optimization using residual connections in the spatial-domain to avoid performance degradation is mandatory[4]. \n\n> *W2*. The effectiveness of a spikformer-based framework with only SDSA, without MS, remains unclear... This weakens the argument for the performance of the spike-driven Transformer. The results suggest that the use of MS may be crucial to achieve SOTA.\n\n**A:** We tend to understand spike-driven Transformer from the view of MetaFromer[6], which argues that there is general architecture abstracted from Transformers by not specifying the token mixer. Setting the operators in the token mixer will result in different accuracies. So, MS and SEW are changes to the architecture, while SDSA and SSA are adjustments to operators. To verify this, we set $T=4$, structure-8-384 on ImageNet as follows:\n\n|Model|SEW+SSA(Base[5])|SEW+SDSA|MS+SSA|MS+SDSA(This work)|\n|---|---|---|---|---|\n|Acc(\\%)|70.2|68.1|72.7|72.3|\n\nWe can see that MS brings gain to architecture performance, MS+SSA(72.7) vs. SEW+SSA(70.2), and MS+SDSA(72.3) VS. SEW+SDSA(68.1). And, compared to SSA, SDSA indeed does perform worse. But it should be noted that there is only addition in the proposed SDSA and the complexity is $O(ND)$, while the SSA in [5] contains multiplication (multi-bit integer multiplication and scale operations) and the complexity is $O(ND^2)$. Note, SDSA is a severely restricted operator, but it works pretty well. So, once the basic architecture is determined, the operator has a trade-off between accuracy and energy.\n\nWe appreciate that you can discuss this issue with us. **This actually points out two directions for further optimization: architecture and spike-driven operators.**\n\n---\n[1] Bottom-Up and Top-Down Approaches for the Design of Neuromorphic Processing Systems: Tradeoffs and Synergies Between Natural and Artificial Intelligence. In Proceedings of the IEEE, 2023.\n\n[2] Event-driven spiking convolutional neural network. WIPO Patent, page WO2020207982A1, 2020.\n\n[3] A long short-term memory for AI applications in spike-based neuromorphic hardware. In Nature Machine Intelligence, 2022.\n\n[4] Attention Spiking Neural Networks. In IEEE T-PAMI, 2023.\n\n[5] Spikformer: When Spiking Neural Network Meets Transformer. In ICLR, 2023.\n\n[6] Metaformer is actually what you need for vision. In CVPR 2022."}]

Author rebuttal:

Meta-review: This work proposes improving the spike transformer by using the membrane shortcut from spike ResNet and a new spike-driven self-attention operator that only uses binary spike-based addition. This is used to achieve SOTA ImageNet top-1 in the SNN field while using lower estimated energy. This will likely be an impactful paper in the SNN field.

Advantages:
- SOTA results for ImageNet in the SNN field.
- Lower estimated energy consumption from the newly proposed SDSA operator.
- The new SDSA operator has lower computational complexity than SSA.

Weaknesses:
- The notation in the equations is a bit confusing.
- MS (membrane shortcut) does not align with SNN mechanisms and its potential implementation is somewhat unclear.
Decision: Accept (poster)