Spikformer: When Spiking Neural Network Meets Transformer 
Authors: Zhaokun Zhou
Yuesheng Zhu
Chao He
Yaowei Wang
Shuicheng YAN
Yonghong Tian
Li Yuan
Venue: ICLR 2023 poster (2023)
Keywords: Transformer
Spiking Neural Network

Abstract: We consider two biologically plausible structures, the Spiking Neural Network (SNN) and the self-attention mechanism. The former offers an energy-efficient and event-driven paradigm for deep learning, while the latter has the ability to capture feature dependencies, enabling Transformer to achieve good performance. It is intuitively promising to explore the marriage between them. In this paper, we consider leveraging both self-attention capability and biological properties of SNNs, and propose a novel Spiking Self Attention (SSA) as well as a powerful framework, named Spiking Transformer (Spikformer). The SSA mechanism in Spikformer models the sparse visual feature by using spike-form Query, Key, and Value without softmax.  Since its computation is sparse and avoids multiplication, SSA is efficient and has low computational energy consumption. It is shown that Spikformer with SSA can outperform the state-of-the-art SNNs-like frameworks in image classification on both neuromorphic and static datasets. Spikformer (66.3M parameters) with comparable size to SEW-ResNet-152 (60.2M,69.26%) can achieve 74.81% top1 accuracy on ImageNet using 4 time steps, which is the state-of-the-art in directly trained SNNs models. Code is avaiable at https://github.com/ZK-Zhou/spikformer.

Published as a conference paper at ICLR 2023
SPIKFORMER: WHEN SPIKING NEURAL NETWORK MEETS TRANSFORMER
Zhaokuun Zhou
1.2 Yuesheng Zhu" Li Yuan'
*Chao He
2 Yaowei Wang
Shuicheng Yan
1.2 Yonghong Tian
I Peking University
PPeng Cheng Laboratory 4 Shenzhen EEGSmart Technology Co., Ltd. (yuanli-ece) eww.eeuu........e. edu.cn
SSea Al Lab
ABSTRACT
We consider two biologically plausible structures, the Spiking Neural Network (SNN) and the self-attention mechanism. The former offers an energy-efficient and event-driven paradigm for deep learning, while the latter has the ability to capture feature dependencies, enabling Transformer to achieve good performance. It is intuitively promising to explore the marriage between them. In this paper, we consider leveraging both self-attention capability and biological properties of SNNs, and propose a novel Spiking Self Attention (SSA) as well as a powerful framework, named Spiking Transformer (Spikformer). The SSA mechanism in Spikformer models the sparse visual feature by using spike-form form Query. Key, and Value without softmax. Since its computation is sparse and avoids multiplication, SSA is efficient and has low computational energy consumption. It is shown that Spikformer with SSA can outperform the state-of-the-art SNNs-like frameworks in image classification on both neuromorphic and static datasets. Spikformer (66.3M parameters) with comparable size can achieve 74.81% topl accuracy on ImageNet using 4 time steps, which is the state-of-the-ar in directly trained SNNs models. Codes is avaiable at (Spikformer
152 (60.2M, 69.26%)
INTRODUCTION
As the third generation of neural network (Maass 1997), the Spiking Neural Network (SNN) is very promising for its low power consumption, event-driven characteristic, and biological plausibility (Roy et al.) 2019). With the development of artificial neural networks (ANNs). SNNs able to lift performance by borrowing advanced architectures from ANNs, such as ResNet-like SNNs (Hu ct al. 2021a, Fang et al. 222 Zheng al. [202]: Hu et al. 2021b), Spiking Recurrent Neural Networks (Loth Rezaabad & Vishwanath| 2020) and Spiking Graph Neural Networks (7hu et al.
2022). Transformer, originally designed for natural language processing (Vaswani et al.) 2017). has
flourished for various tasks in computer vision, including image classification (Dosovitskiy et al.
2020: Yuan al. 2021a), object detection (Carion et al. 2020 Zhu al.) 2020 Liu al. [021]
semantic segmentation (Wang et al. [021]] Yuan et al. 2021b) and low-level image processing (Chen al. 2021). Self-attention the key part of Transformer, selectively focuses on information of
interest, and is also an important feature of the human biological system (Whitington et al.) 2022
Caucheteux King 2022). Intuitively, it is intriguing to explore applying self-attention SNNs for more advanced deep learning, considering the biological properties of the two mechanisms.
however non-trivial to port the self-attention mechanism into SNNs. In vanilla self-attention (VSA) (Vaswani al. 2017), there are three components: Query, Key, and Value. As shown in Figure [a), standard inference of VSA is firstly obtaining a matrix by computing the dot product of poitt-ointtfomm Query and Key; then softmax, which contains exponential calculations and division operations, is adopted to normalize the matrix to give the attention map which will be used to weigh the Value. The above steps in VSA do not conform the calculation characteristics of SNNs, i.e., avoiding multiplication. Moreover, the heavy computational overhead of VSA almost prohibits
"Iddcctes the corresponding author.

---

Published as a conference paper at ICLR 2023
Attn MAn
-084 Pollme 1.7e
5.5 10 SOP
KG
Ätin Map
VI=
V
4.95pT
X
2:4
-1.5
①
1.1
-0:11
2.8
3.3
X
X
4.8
-1.8
dxN
of X
KTy
Nxd
NXN
N'd
Nx
Nxd ouNi
X
Softmax
4.4 10 SOP
dxd
0IN 22x FL 354.23
3.66J)
NX
(a) Vanilla Self Attention
(b) Spiking Self Attention
Figure Illustration of vanilla self-attention (VSA) and our Spiking Self Attention (SSA). A red spike indicates a value of I at that location. The blue dashed boxes provide examples of matrix dot product operation. For convenience, we choose one of the heads of SSA, where N is the number of input patches and d is the feature dimension of one head. FLOPss is the floating point operations and SOPs is the theoretical synaptic operations. The thcorctical energy consumption perform one calculation between Query, Key and Value in one time step is obtained from senccderr-booke 512-embedding dimdiinndimeeiionn Spikformer on ImageNet test set according to (Kundu ct al. 2021b Hu et al. 2021a)) More details about the calculation of theoretical SOP and energy consumption are included appendix.
(a) In VSA. Q7, Kr, VT are float-point forms. After the dot-product of Qr and K,, the softmax function regularizes negative values in the attention map positive values. (b) In SSA, all value attention map is non-negative and the computation is sparse using spike-form Q, K, V (5.5 x 10 VS. 77 x 10" in VSA). Therefore, the computation in SSA consumes less energy compared with VSA (354. 2p.)). In addition, the SSA is decomposable (the calculation order of Q, K and V is changeable).
C.2
applying it directly to SNNs. Therefore, in order to develop Transformer on SNNs, we need to design a new effective and computation-effiiien self-attention variant that can avoid multiplications.
We thus present Spiking Self Attention (SSA), illustrated in Figure I b). SSA introduces self- attention mechanism to SNNs for the first time, which models the interdependence using spike sequences. In SSA, the Query, Key, and Value are in spike form which only contains of 0 and 1. The obstacles to the application of self-attention in SNNs are mainly caused by softmax. 1) As shown in Figure [ the attention map calculated from spike- form Query and Key has natural non- non-negativeness, which ignores irrelevant features. Thus, we do not need the softmax to keep the attention matrix non-negative, which is its most important role in VSA (Qin al.] 2022). 2) The input and the Value of the SSA are in the form of spikes, which only consist of O and I and contain less fine aie-ggrined feature compared to the float-point input and Value of the VSA in ANNs. So the float-point Query and Key and softmax function are redundant for modeling such spike sequences. Tab. Iillustraaes that our SSA competitive with VSA in the effect of processing spike sequences. Based on the above insights, we discard softmax normalization for the attention map in SSA. Some previous Transformer variants also discard softmax or replace it with a linear function. For example, Performer (Choromanski et al. 2020), positive random feature is adopted to approximate softmax; CosFormer (Qin et al. 2022 replaces softmax with ReLU and cosine function.
With such designs of SSA, the calculation of spike-form Query, Key, and Value avoids multiplications and can be done by logical AND operation and addition. Also, its computation is very efficient. Due to sparse spike-form Query, Key and Value (shown in appendix [D.]] and simple computation, the number of operations in SSA is small, which makes the energy consumption of SSA very low. Moreover, our SSA is decomposable after deprecation of softmax, which further reduces its computational complexity when the sequence length is greater than the feature dimension of one head, depicted in Figure [b) ① 2.
Based on the proposed SSA, which well suits the calculation characteristics of SNNs, we develop the Spiking Transformer (Spikformer). An overview of Spikformer is shown in Figure 2 boosts the performance trained on both static datasets and neuromorphic datasets. To the best of our knowledge, is the first time to explore the self-attention mechanism and directly-trainet Transformer in the SNNs. To sum up, there are three-fold contributions of our work:
We design novel spike-form self-attention named Spiking Self Attention (SSA) for the properties SNNs. Using sparse spike-form Query, Key, and Value without softmax, the calculation of SSA avoids multiplications and is efficient.
We develop the Spiking Transformer (Spikformer) based on the proposed SSA. To the best of our knowledge, this is the first time implement self-attention and Transformer in SNNs.
2

---

Published as a conference paper at ICLR 2023
Extensive experiments show that the proposed architecture outperforms the of-e-oo-thhe-rr SNNs on both static and neuromorphic datasets. is worth noting that we achieved more than 74% accuracy on ImageNet with 4 time steps using directly-trained SNN model for the first time.
2 RELATED WORK
Vision Transformers. For the image classification task, a standard vision transformer (ViT) includes a patch splitting module, the transformer encoder layer(s), and linear classification head. The Transformer encoder layer consists of self-attention layer and a multi perception layer block. Self- attention is the core component making ViT successful. By weighting the image-patches feature value through the dot-product of query and key and softmax function, self-attention can capture the global dependence and interest representation [Katharopoulos et al. 20200 Qin et al. (022). Some works have been carried out to improve the structures of ViTs. Using convolution layers for patch splitting has been proven to be able to accelerate convergence and alleviate the data-hungry problem of ViT (Xiao et al., 202 bb; Hassani et al.] 202T). There are some methods aiming to reduce the computational complexity of self-attention improve its ability of modeling visual dependencies (0a
(Song) 2021] Yang et al. 2021] Rao et al.) 2021; Choromanski et al., (2020). This paper focuses on exploring the effectiveness of self-attention in SNNs and developing powerful spiking transformer
model for image classification.
Spiking Neural Networks. Unlike traditional deep learning models that convey information using
continuous decimal values, SNNs use discrete spike sequences to calculate and transmit information. Spiking neurons receive continuous values and convert them into spike sequences, including the
Leaky Integrate-and-FFree (LIF) neuron (Wu et al.] (018). PLIF (Fang et al., 2021b), etc. There are two ways get deep SNN models: ANN-to-Sie -NNN conversion and direct training. In ANN- to-SNN conversion (Cao et al. Hunsserger & Eliasmith 2015 Rueckauer et al. 2017 Bu
af. 2021 Meng et al., 2022 Wang et al. 2022), the high- nigh-performance pre-trained ANN is converted to SNN by replacing the ReLU activation layers with spiking neurons. The converted SNN requires large time steps to accurately approximate ReLU activation, which causes large latency (Han et al. 2020). In the area of direct training, SNNs are unfolded over the simulation time steps and trained in a way of backpropagation through time (Lee et al. 2016 Shrestha & Orchard 2018). Because the event-triggered mechanism in spiking neurons is non non- differentiable, the surrogate gradient is used for backpropagation (Lee et al.) 2020] Neftci et al. 2019 Xiao et al. (222 Ia) adopts implicit differentiation on the equilibrium state to train SNN. Various models from ANNs have been ported to SNNs. However, the study of self-attention on SNN is currently blank. Yao et al.
(2021) proposed temporal attention reduce the redundant time step. Zhang et al. (2022a b) both use ANN-Transformer to process spike data, although they have 'Spiking Transformer' in the title. Mueller al. (001)) provides ANN-SNN conversion Transformer, but remains vanilla self-attention which does not conform the characteristic of SNN. In this paper, we will explore the feasibility of implementing self-attention and Transformer SNNs.
As the fundamental unit of SNNs, the spike neuron receives the resultant current and accumulates membrane potential which is used to compare with the threshold to determine whether to generate the spike. We uniformly use LIF spike neurons in our work. The dynamic model of LIF is described as:
H[4 Vlc 1]+ 1 (X(4 (V(L 1] Vrrsect)
7
Sit] = ( Vea),
(2)
Vl[ = H[ (1 (1- s(t) + S(t.
(3)
where is the membrane time constant, and X(I is the input current time step L. When the membrane potential H() exceeds the firing threshold Ven, the spike neuron will trigger a spike S11]. ((v) is the Heaviside step function which equals for membrane potential after the trigger event which equals H [t] if no spike is generated, and otherwise equals to the reset potential Vreset-
and otherwise. V represents the
METHOD
We propose Spiking Transformer (Spikformer), which incorporates the self-attention mechanism and Transformer into the spiking neural networks (SNNs) for enhanced learning capability. Now we explain the overview and components of Spikformer one by one.
3

---

Published as a conference paper at ICLR 2023
Spikformer Encoder Block x L
SPS
Clacaficstin Head
Comy
BV
MLP
Spiking Position Embedding
Spiking Self Attention
Element-wise Add
Spike Neuron
Layer
Dost Producct
MP
Max Pooling
NB
Spike Feature
Spike Sequence
Linear
The overview of Spiking Transformer (Spikformer), which consists of a spiking patch splitting modulc (SPS), Spikformer encoder and Linear classification head. We empircally find that the layer normalization (LN) does not apply to SNNs, so we use batch normalization (BN) instead.
Figure 2:
3.1 OVERALL ARCHITECTURE
overview of Spikformer depicted in Figure 2 Given a 2D image sequence X
the Spiking Patch Splitting (SPS) module linearly projects it to D dimensional spike-form feature vector and splits into a sequence of N flattened spike-form patches z. Float-point-form position embedding cannot be used in SNNs. We employ conditional position embedding generator (Chu eet al. 2021) to generate spike-form relative position embedding (RPE) and add the RPE to patches sequence get Xo. The conditional position embedding generator contains 2D convolution layer (Conv2d) with kernel size 3, batch normalization (BN), and spike neuron layer (SN). Then we pass the X, the L-block Spikformer encoder. Similar the standard VIT encoder block, Spikformer encoder block consists of Spiking Self Attention (SSA) and an MLP block. Residual connections are applied in both the SSA and MLP block. As the main component in Spikformer encoder block. SSA offers an efficient method to model the local-global information of images using spike-form Query (Q), Key (K), and Value (V) without softmax, which will be analyzed in detail in Sec. (3.3 A global average-pooling (GAP) is utilized on the processed feature from Spikformer encoder and outputs the D-dimension feature which will sent to the cully-comnected-lay classification head (CH) to output the prediction Y. Spikformer can be written as follows:
ST)
T RTXCXHXW I RTRND
(4)
RPF (CCIn((CC))(())))
RPE E TTNND
(5)
X, = x + RPE,
Xo E TXNxD
(6)
X SSA(X--1) + X1-1,
X/ E RRNND ,1 I...L
(7)
X; MLP(X/) + X/,
X1 E E TxNxD 1ANxD,I=1...1
T 1...L
(8)
Y = CH(G)A((L)
(9)
3.2 SPIKING PATCH SPLITTING
As shown in Figure the Spiking Patch Splitting (SPS) module aims to linearly project an image to dimensional spike-form feature and split the feature into patches with a fixed size. SPS can contain multiple blocks. Similar the convolutional stem in Vision Transformer (Xiao et al. 2021 Hassani et al. 2021), we apply a convolution layer in each SPS block to introduce inductive bias into
In the neuromorphic dataset the data shape is / E I TxCHHXW where T. C. H. and W denote time step. channel, height and width, respectively. A 2D image 7 RCXHxW in static datasets need to be repeated T times form sequence of images.
4

---

Published as a conference paper at ICLR 2023
Spikformer. Specifically, given an image sequence I R?
I MP (SN (SN((((CCV)(())))
(10)
where the Conv2d and MP represent the 2D convolution layer (stride-1, max-pooling, respectively. The number of SPS blocks can be more than 1. When using multiple SPS blocks, the number of output channels in these convolution layers is gradually increased and finally matches the embedding dimension of patches. For example, given output embedding dimension D and a four-block SPS module, the number of output channels in four convolution layers is D/8, D/4, D/ D/2, D. While the 2D-max-pa -D-max-pooiing layer is applied to down-sampl, the feature size after SPS block with a fixed size. After the processing of SPS, is split into an image patches sequence x E TxNxD
kernel size) and
3.3 SPIKING SELF ATTENTION MECHANISM
Spikformer encoder is the main component of the whole architecture, which contains the Spiking Self Attention (SSA) mechanism and MLP block. In this section we focus on SSA, starting with a review of vanilla self-attention (VSA). Given an input feature sequence X E xxxNNx the VSA in ViT has three float-point key components, namely query (Qz), key (K=), and value (Vx) which are calculated by learnable linear matrices Wo, W K, Wv E R RDXD and X:
Qx = XWQ: K5= = XWK, Vi= = XWv
(11)
where F denotes the float-point form. The output of vanilla self-attention can computed as:
@
VSA(Qr, Kr, Vz)
Softmax
V5
(12)
where d= D/H is the feature dimension of one head and H is the head number. Converting the Moot-point-oomm Value (Vz) into spike form (V) can realize the direct application of VSA in SNNs, which can be expressed as:
QFK
VSA(Q=, KF, V) Softmax
(13)
However, the calculation of VSA is not applicable in SNNs for two reasons. 1) The float-point matrix multiplication of Q,, K and softmax function which contains exponent calculation and division operation, do not comply with the calculation rules of SNNs. 2) The quadratic space and time complexity of the sequence length of VSA do not meet the efficient computational requirements of SNNs.
We propose Spiking Self-Attention (SSA), which is more suitable for SNNs than the VSA, as shown in Figure (bb) and the bottom of Figure 2 The query (Q), key (K), and Value (V) are computed through learnable matrices firstly. Then they become spiking sequences via different spike neuron layers:
-NN(BN((WQ)).K SN (BN(WN(WW))), = SNV(BN(XWW)
(14)
where Q, K. V RTNNX We believe that the calculation process of the attention matrix should use pure spike-form fore-ffommmm Query and Key(only containing O and 1). Inspired by vanilla self-attention VVaswani et al.) 2017). we add a scaling factor s to control the large value of the matrix multplicatiion result. 8 does not affect the property of SSA. As shown in Figure as:
2 the spike-friendly SSA is defined
SSA' (Q,K,V) SN (Q SSA K, V) = S. (BN(Linaaaa (Q, K. V)))..
V Q
(15)
(16)
The single-head SSA introduced here can easily be extended to the multi-head SSA, which detailed in the appendix A SSA is independently conducted each time step and seeing more details in appendix B As shown in Eq. (15). SSA cancels the use of softmax to normalize the attention matrix in Eq. (1) and directly multiplies Q, K and V. An intuitive calculation example is shown in Figure [[b.. The softmax is unnecessary in our SSA, and it even hinders the implementation of self-attention to SNNs. Formally, based on Eq. [4]. the spike sequences and K output by the spiking neuron
5

---

Published as a conference paper at ICLR 2023
layer Q and SN& & respectively, are naturally non-negative (0 or 1), resulting in non non-negative attention map. SSA only aggregates these relevant features and igmores the irrelevant information. Hence it does not need the softmax to ensure the non-negativeness of the attention map. Moreover, compared to the float-point-frm X5 and V in ANNs, the input X and the Value V of self-attention in SNNs are in spike form, containing limited information. The vanilla self-attention (VSA) with float-pinttform Q,,, K5 and softmax is redundant for modeling the spike-form X, V, which cannot get more information from X, V than SSA. That is, SSA is more suitable for SNNs than the VSA.
We conduct experiments to validate the above insights by comparing the proposed SSA with four different calculation methods of the attention map, as shown in Tab. A1 denotes multiply- ing the float-points Q and K directly to get the attention map, which preserves both positive and negative correlation. ARELU uses the multiplication between (eLU(Q) and ReI.U(K) to obtain the attention map. AreLU retains the positive values of Q K and sets the negative values to 0,
Table Analysis of the SSA's rationality. We replace SSA with other attention vari- ants and keep the remaining network structure in Spikformer unchanged. We show the accuracy (Acc) on CIFARIO-DY
(Li al. (2177). CIFARI0/100 (Krizhevsky
2009).
OPs (M) the number of operations (For As, At.nkyPall.tt AR.L. and
Awoftmaxx OPs is FLOPs, and SOPs ignored: For ASSA. it SOPs.) and P (aJ) the thcorctical energy consumption to perform one calculation among Q, K, V.
CIFARIO-DVS
CIFARIO
CIFARIO0
Acc/OPs (M)P (nJ)
A
79.40/16.8/77
93.96/6.3/29
76.94/6.3/29
ALeskyRoLU
79.80/16.8/77
93.85/6.3/29
76.73/6.3/29
ARRLU
79.40/16.877
94.34/6.3/29
77.00/6.3/29
Asoftmax
80.00/19. 1/88
94.97/6, 6/30
77.92/6.6/3
ASSA
80.90/0.66/0.594
95.19/1.1/0.00
77.86/1.3/1.177
while ALenkyRet.t Rel.U still retains the negative points. Asottmax means the attention map is generated following VSA. The above four methods use the same Spikformer framework and weight the spike-form V. From Tab. [Tb the superior performance of our ASSA over A and ReLa proves the superiority of SN. The reason why ASsA is better than ARELU may be that AssA has better non-linearity in self-attention. By comparing with Asoftmax: ASSA is competitive, which even surpasses Agoftmax on CIFAR TODVS and CIFAR1O. This can be attributed to SSA being more suitable for spike sequences (X and V) with limited information than VSA. Furthermore, the number of operations and theoretical energy consumption required by the ASSA to complete the calculation of Q, K, V is much lower than that of the other methods.
SSA is specially designed for modeling spike sequences. The Q, K, and V are all in spike form, which degrades the matrix prodput calculation to logical AND operation and summation operation. We take a row of Query g and a column of Key A as a calculation example: Σi= g:ki = >n; =1 kip Also, as shown in Tab. [ SSA has a low computation burden and energy consumption due to sparse spike-form Q, K and V (Figure. 4p and simplified calculation. In addition, the order of calculation between Q, K and V is changeable: QK" first and then V, or TFV first and then Q. When the sequence length N is bigger than one head dimension d, the second calculation order above will incur less computation complexity (O(Nd)) than the first one (O(N2d)). SSA maintains the biological plausibility and computationally efficient properties throughout the whole calculation process.
4 EXPERIMENTS
We conduct experiments on both static datasets CIFAR, ImageNet (Deng al. 2009). and neuro- morphic datasets CIFARIO-DYY 10-DVS, DVS128 Gesture (Amir et al.) 2017) to evaluate the performance of Spikformer. The models for conducting experiments are implemented based on Pytorch (Paszke et al., 2019). SpikingJelly H and Pytorch image models library (Timm) We train the Spikformer from scratch and compare it with current SNNs models in Sec. 4. and 4.2
We conduct ablation
studies to show the effects of the SSA module and Spikformer in Sec.
4.1 STATIC DATASETS CLASSIFICATION
ImageNet contains around 1.3 million 1, 000-class images for training and 50, 000 images for validation. The input size of our model ImageNet is set to the default 224 224. The optimizer

---

Published as a conference paper at ICLR 2023
Table 2: Evaluation ImageNet. Param refers to the number of parameters. Power is the average theoretical energy consumption when predicting an image from ImageNet test set, whose calculation detail is shown in Eq. 22 Spikformer-L-D represents a Spikformer model with L Spikformer encoder blocks and D feature embedding dimensions. The train loss, test loss and test accuracy
curves are shown in appendix D.2
OPs refers SOPs in SNN and FLOPs in ANN-VIT.
Param (M)
OPs
Power (mJ)
Time Step
Methods
Architecture
Acc
(G)
Hybrid training Rathi al.
2020
ResNet-34
21.79
250
61.48
Spiking-ResNet-34
21.79
6
64.79
TET Deng al. 2021
SEW-ResNet-34
21.79
4
68.00
ResNet-34
21.79
65.28
59.295
350
71.61
Spiking ResNet Hu al.
2021a
ResNet-50
25.56
78.29
70.934
350
72.75
Zheng STBP-dBNN Zheng et al. 2021
Spiking-ResNet-34
21.79
6.50
6.393
6
63.72
SEW-ResNet-34
21.79
3.88
4.035
67.04 4
SEW-ResNet-50
25.56
4.83
4,890
4
67.78
SEW ResNet al.
SEW-ResNet-101
44.55
9.30
8.913
4
68.76
SEW-ResNet-15
60.19
13.72
12,891
69.26
Transformer
Transforme---551
29.68
8.33
38.340
80.80
Spikformer-8 8-344
16.81
6.82
7.734
4
70.24
Spikformer-6- 6-00ommer---5111 6-212
23.37
8.69
9,417
4
72.46
Spikformer
Spikformer-8-5
29.68
11.09
11.577
4
73.38
Spikformer- 10-512
36.01
13.67
13.899
4
73.68
Spikformer-8-76
66.34
22.09
21.477
4
74.81
is Adam W and the batch size is set to 128 or 256 during 310 training epochs with a cosine-decay learning rate whose initial value is 0.0005. The scaling factor 0.125 when training on ImageNet and CIFAR. A four-block SPS splits the image into 196 16 x 16 patches. Following (Yuan et al.) 2021a), standard data augmentation methods, such as random augmentation, mixup, and cutmix, also used in training. We try variety of models with different embedding dimensions and numbers of transformer blocks for ImageNet, which has been shown in Tab. 2
We also give a comparison of synaptic operations (SOPs) (Merolla et al.. 2014) and theoretical energy consumption. From the results, can be seen that our Spikformer achieves a significant accuracy boost the ImageNet compared with the current best SNNs models. In particular, our comparison first starts from our smallest model with other models. The Spikformer-8-36 with 16.81M parameters has 70.24% top-1 accuracy when trained from scratch on ImageNet, which outperforms the best the current best direct-train model SeW-ResNet- 152: 69. 26% with 60.19M. In addition, the SOPs and the theoretical en- Spikformer-S 8-384 (6. 82G. 7.734mJ) are lower com- pared with the SEW-ResNed 1ee (13.72G, 12.89ImJ). The 29.68M model Spikformer-8-511 12 has already achieved state-of-the-art performance with 73.38% which is even higher than the converted model (Hu et al. (02 (72.75%) using 350 time steps. As the number of Spikformer blocks increases, the classification accuracy of our model ImageNet is also 10-552 getting higher. The Spikformer- 10-512 obtains 73.68% with 42.35M. The same happens when gradually increasing the embedding dimension, where Spikformer-8-768 further improves the performance to 74.81% 1% and significantly outperforms the SEW-ResNet-155 model by 5.55%. ANN- ViT-8-512 is 7.42% higher than @pikformer-8-5111 but the theoretical energy consumption is 3.31 x of Spikformer- ---trmmee---11122--- 8-512. In Figure we show the attention map examples of the last encoder block in Spikformer-8-512 the fourth time step. SSA can capture image regions associated with classification semantics and set irrelevant regions to 0 (black region), and is shown to be effective, event-driven, and energy-efficient.
Input Image
A ttention Map
ergy consumption
Figure 3: Attention map examples of SSA. The black region 0.
CIFAR provides 50, 000 train and 10, 000 test images with 32 x 32 resolution. The batch size is set to 128. A four-block SPS (the first two blocks do not contain the max- max-pooling layer) splits the image into 64 4 x patches. Tab. 3 shows the accuracy of Spikformer compared with other models on CIFAR. As shown in Tab, B Spikformer-4- 384 achieves 95.1 9% accuracy CIFAR10, which is better than the TET (94.44%) and ResNet-19
7

---

Published as a conference paper at ICLR 2023
Table 3: Performance comparison of our method with existing methods CIFAR10/100. Our
method improves network performance across all tasks. ddnotes sellemmpleenntaiionn results by
Deng et al. (202T). Note that Hybrid training (Rathi et 2020) adopts ResNet-20 for CIFARIO and VGG-11 for CIFARI00.
Param (M)
Time Step
CIFARIO Acc
CIFARI00
Methods
Architecture
Acc
Hybrid Raahi
Rathi ct al. 2020
VGG-11
9.27
125
92.22
67.87
Diet-SNN Rath & Koy 2020
ResNet-20
0.27
10/5
92.54
64.07
STBP Wu et a.. 2018
CIFARNet
17.34
12
89.83
STBP NeuNorm WD al. 2019
CIFARNet
17.54
90.53
TSSL-BP Zhang 14
CIFARNet
17.54
91.41
PROP-ddBNiZeennn TET Deig et al. 2021
ResNet-19
12,63
92.92
70.86
ResNet-19
12.63
94,44
74.47
ResNet-
12.63
94.97
75.35
ANN
Transfomer-4-334
9.32
96.73
81.02
Spikformer-4-25
4.15
93.94
75.96
Spikformer
Spikformer-2-366
5.76
94.80
76.95
Spikformer-44384
9.32
95,19
77.86
Spikformer-4338/ 400E
9.32
95.51
78.21
ANN (94.97%). The performance is improved as the dimensions or blocks increase. Specifically, Spikformer-43384 improves by 1.25% compared to Spikformer-4-256 and improves by 0.39% compared to Spikformer-2-3363 We also find that extending the number of training epochs 400 can improve the performance (Spikformer--388) 400E achieves 0.32% and 0.35% advance compared to Spikformer-4-384 on CIFAR 10 and CIFARI00), The improvement of the proposed Spikformer on complex datasets such as CIFARIOS 100 i is even higher. Spikformer-4-384 (77.86%, 9.32M) obtains significant improvement of 2.51% compared with ResNet- 19 ANN (75.35%, 12.63M) model. The ANN-Transformer model is 1 1.54% and 3.16% higher than Spikformer-43384 respectively. As shown in appendix [D.5 transfer learning can achieve higher performance on CIFAR based on pre-trained Spikformer, which demonstrates high transfer ability.
4.2
NBUROMORPHIC DATASETS CLASSIFICATION
DVS128 Gesture is gesture recognition dataset that contains 11 hand gesture categories from 29 individuals under 3 illumination conditions. CIFARIO-DVS is also neuromorphic dataset converted from the static image dataset by shifting image samples to be captured by the DVS camera, which provides 000 training samples and 1, 000 test samples.
For the above two datasets of image size 128 x 128, we adopt fourkblock SPS. The patch embedding dimension is 256 and the patch size 16 x 16. We use shallow Spikformer with transformer encoder blocks. The SSA contains 8 and 16 heads for DVS128 Gesture and CIFAR 10-DVS, re- spectively. The time the spiking neuron is or 16. The training epoch is 200 for DVS128 Gesture and 106 for CIFARIO-DVS The optimizer is AdamW and the batch size is set to 16. The learning rate is initialized to 0.1 and reduced with cosine decay. We apply data augmentation on CIFARIO-DVS according to (Li al. 2022). We use a learnable parameter the scaling factor to control the QK QV result.
The classification performance of Spikformer well as the compared state-of -fee----rf---tttt models neuromorphic datasets is shown Tab. can be seen that our model achieves good performance on both datasets by using 2.59M model. On DVS128 Gesture, 16-time steps, which is higher than NEw-ResNet (97.9%). Our result is also competitive compared with TA-SNN (98.6%, 60 time steps) Yaoo al. 2021) which uses floating-poin spikes in the forward propagation. On CIFARIO-DVE we achieve 1.6% and 3.6% better accuracy than the SOTA methods DSR (77.3%) with binary spikes using 10 steps and 16 steps respectively. TET not an architecture- based but a based method which achieves 83.2% using long epochs (300) and 9.27M VGGSNN, so do not compare with it in the table.
obtain accuracy of 98.2% with
4.3 ABLATION STUDY
Time step The accuracy regarding different simulation time steps of the spike neuron shown in When the time step method 1.87% lower than the network with T on CIFAR10.
Tab. 5,
8

---

Published as a conference paper at ICLR 2023
Table 4: Performance comparison to the state-of-the-art (SOTA) methods neuromorphic datasets. Bold font means the best; denotes with Data Augmentation.
CIFARIO-DVS
DVS128
Method
Spikes
T Step
Acc
T Step
Acc
LIAF-Net Wu et al.
2021
TNNI 5-2011
X
10
70.4
60
97.6
TA-SNN Yao et al.
2021
1CC-2021
X
10
72.0
60
98.6
Rollout Kugele ct al.
2020
Front. Neurosci 2020
48
66.8
240
97.2
DECOLLE (Kaiser
2020
Neurosci
500
95.5
tdBN Zheng et al. 2021
2e2rt-2021
10
67.8
40
96.9
PLIF Fang et al. 2021b
97.6
SEW-ResNop (Fang et all
202
NeudPS-2021
16
74.4
16
97.9
Dspike (Li
75.4*
SALT Kim & Panda 2021
Neural Netw-2021
20
67.1
DSR (Meng et al.
2022
77.3"
10
78.9
10
96.9
Spikformer
80.9
98.3
Spikformer-8-511 with time step still achieves 70.14%. The above results show Spikformer is robust under low latency (fewer time steps) conditions.
SSA We conduct ablation studies SSA further identify its advantage. We first test its ef- fect by replacing SSA with standard vanilla self-attention. We test two cases where Value is in floating point form (Spikformer-L-DDw VSA Vz)) and in spike form (Spikformer-L-Dw VSA).
We also test the different atten- tion variants on ImageNet follow- ing Tab. D On CIFARIO, the performance of Spikformer with SSA is competitive compared to Spikformer-4-384, VSA and even
Table Ablation study results on SSA, time step.
Datasets
Models
Time Step
Topl-Ace (%)
93.51//4.33
93 53.55/66.28 28
CIFARI0/100 Spikformer-43384. S8A
05.19/77.88
Spikformer-4-384.. VSA Vyr, On Im- ageNet, our Spikformer-8-512. SSA outperforms Spikformer-8-512,, VSA by 0.68%. On CIFARION 100 and Ima- geNet, the accuracy of Spikformer-T- Dw v is better than Spikformer because of the float-point-fomm Value. The reason why the Spikformer- 8-512 1. Spikformer-8-512, ReLU, and Spikformer-8 8-11 LeakyReLU do not converge is the value of dot-product value of Query, Key, and Value large, which makes the surrogate gradient of the output spike neuron layer disappear. More details are in the appendix D.4 In comparison, the dot-prood not-product value the designed SSA is in controllable range, which is determined by the sparse spike-form foree-form&t Q, K and V. and makes Spikformer, SSA easy to converge.
95.34/78.61
Spikformer-44 VSA
94.97/77.99
Spikformer VSA Vyr
95.17/78.37 4
Spikformer-8-512..
X
Spikformer-8-5122. ReLU
X
Spikformer-8-512 LeakyRoLU
X
Spikformer-8-511 VSA
72.70
ImageNet
Spiklormer-8-511 12 VSA V.r
73.96
70.14
Spikformer-8-512u 88A
09
73. 73.38
73.70
CONCLUSION
In this work explored the feasibility of implementing the self atlf-attention mechanism and Transformer in Spiking Neuron Networks and propose Spikformer based on a new Spiking Self-Attention (SSA). Unlike the vanilla self-attention mechanism in ANNs. SSA is specifically designed for SNNs and spike data. We drop the complex operation of softmax in SSA, and instead perform matrix dot- product directly spike-form Query, Key, and Value, which is efficient and avoids multiplications. In addition, this simple self-attention mechanism makes Spikformer work surprisingly well on both static and neuromorphic datasets. With directly training from seratch, Spiking Transformer outperforms the state-of-the-an SNNs models. We hope our investigations pave the way for further research on transformer-bat based SNNs models.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Our codes are based on SpikingJelly (Fan et al. 2020). an open-source SNN framework, and Pytorch image models library (Timm)( Wightman ) The experimental results in this paper reproducible. We explain the details of model training and dataset augmentation in the main text and supplement in the appendix. Our codes of Spikformer models uploaded supplementary material and will be available on GitHub after review.
ACKNOWLEDGEMEENT
This work supported by Nature Science Foundation of China NN.62202014 and No.62006007). Shenzhen Basic Research Program (AJJC/1202228111511776000 and the National Innovation 2030 Major ST Project of China (No. (N0.20220AAAA0014203))
REFERENCES
Amon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, Jeff Kusnitz, Michael Debole, Steve Esser, Tobi Delbruck, Myron Flickner, and Dharmendra Modha. A low power, fully event-based gesture recogmition system. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7243-7252, 2017.
Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann-snn conversion for high-accuracy and ultra-low-latee latency spiking neural networks. In International Conference Learning Representations (ICLR), 2021.
Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(())544-6) 66, 2015.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision (ECCV). pp. 213-229. Springer, 2020.
Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communications biology, 5(1):1-10, 2022.
Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12299-12310, 2021.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXivv2009. 14794, 2020.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS),
volume 34, pp. 9355-9336, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting. In International Conference on Learning Representa- tions (ICLR), 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representa- tions (ICLR), 2020.
10

---

Published as a conference paper at ICLR 2023
Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Yonghong Tian, and other contributors. Spikingielly, / http:tuw..ccm.aaaeeeee/////////a
2020. Accessed: YYYY-MM-DD.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep Residual Learning in Spiking Neural Networks. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), volume 34, pp. 21056-21069, 2021a.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2661-
2671, 2021b.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 13558-
13567, 2020.
Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:2104.07700 2021.
Mark Horowitz. 1.1 computing's s energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14. IEEE, 2014.
Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-6, 2021a. doi: 10.1009
Yifan Hu, Yujie Wu, Lei Deng, and Guoqi Li. Advancing residual learning towards powerful deep spiking neural networks. arXiv preprint arXiv:21 2021b.
Eric Hunsberger and Chris Eliasmith. Spiking deep networks with lif neurons. arXiv preprint arXiv:1510.08829, 2015.
Jacques Kaiser, Hesham Mostafa, and Emre Neftci. Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE). Frontiers in Neuroscience, 14:424, 2020. doi:
003338//nins.2222000444
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning (ICML), pp. 5156-5165, 2020.
Youngeun Kim and Priyadarshini Panda. Optimizing Deeper Spiking Neural Networks for Dynamic Vision Sensing. Neural Networks, 144:686-698, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca. Efficient Processing of Spatio-temporal Data Streams with Spiking Neural Networks. Frontiers in Neuroscience, 14:439, 2020.
Souvik Kundu, Gourav Datta, Massoud Pedram, and Peter A Beerel. Spike-thrift: Towards energy- efficient deep spiking neural networks by limiting spiking activity via attention- -uidedd compression. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3953-3962, 202
Souvik Kundu, Massoud Pedram, and Peter A Beerel. Hire-snn: Harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 5209-5218, 2021b.
Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling spike-base based backpropagation for training deep neural network architectures. Frontiers in neuroscience, 11: 2020.
11

---

Published as a conference paper at ICLR 2023
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in neuroscience, 10:508, 2016.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. CifarlO-dvs: event-stream dataset for object classification. Frontiers in neuroscience, 11:309, 2017.
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks. Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), volume 34, pp. 22439, 2021.
Yuhang Li, Youngeun Kim, Hyoungseob Park, Tamar Geller, and Priyadarshini Panda. Neuromorphic data augmentation for training spiking neural networks. arXiv preprint arXiv:2203.06145 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 10012-10022, 2021.
Ali Lotfi Rezaabad and Sriram Vishwanath. Long short-term memory spiking networks and their applications. In Proceedings of the International Conference on Neuromorphic Systems 2020
(ICONS), pp. 2020.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 1099);1659-16711 1997.
Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Training Hiih--eeffommncc Low-Latency Spiking Neural Networks by Differentiation on Spike Representa- tion. ArXiv preprint arXiv:2205.004: 2022.
Paul Merolla, John Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking- neuron integrated circuit with a scalable communication network and interface. Science, 345 (6197):668 673, 2014.
Etienne Mueller, Viktor Studenyak, Daniel Auge, and Alois Knoll. Spiking transformer networks: A rate coded approach for processing sequential data. In 2021 7th International Conference Systems and Informatics (ICSAI), pp. 1-5. IEEE, 2021.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-basee optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 1--66,,:--3,) 2019.
Priyadarshini Panda, Sai Aparna Aketi, and Kaushik Roy. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience, 14:653, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Proceedings the International Conference on Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Ling- peng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. arXiv preprint 0007092202.08791,, 2022.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), volume 34, pp. 13937-13949, 2021.
Nitin Rathi and Kaushik Roy. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks, arXiv preprint arXiv:2008.0m 036558. 2020.
12

---

Published as a conference paper at ICLR 2023
Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXhv:2005.01807, 2020.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing, Nature, 575(7784);007-611 2019.
Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conver- sion of continuous- valued deep networks to efficient event-driven networks for image classification. Frontiers in neuroscience, 1 11:682, 2017.
Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), volume 31, 2018.
leong-geun Song. Vit High performance linear vision transformer without softmax. arXiv preprint arXiv:2109..433333 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), volume 30, 2017.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 568-578, 2021.
Yuchen Wang, Malu Zhang, Yi Chen, and Hong Qu. Signed neuron with memory: Towards simple, accurate and high-efficient ann-snn conversion. In International Joint Conference on Artificial Intelligence, 2022.
James C. R. Whittington, Joseph Warren, and Tim E.J. Behrens. Relating transformers to mod- els and neural representations of the hippocampal formation. In International Conference on Learning Representations (ICLR), 2022. URL Cettps://github.comm// /htmmn/
B8DVo951YBO
Ross Wightman. Pytorch image models.
pytorch- image-modee 2019.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for
training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct Training for Spiking Neural Networks: Faster, Larger, Better. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 1311-1318, 2019. doi: 10. 33101.330111111
Zhenzhi Wu, Hehui Zhang, Yihan Lin, Guoqi Li, Meng Wang. and Ye Tang. LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processing. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-14, 2021. doi: 10.1 10.1109/NNNLS.220...7772
Mingqing Xiao, Qingyan Meng. Zongpeng Zhang, Yisen Wang, and Zhouchen Lin. Training feedback spiking neural networks by implicit differentiation on the equilibrium state. volume 34, pp. 14516-14528, 2021a.
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), volume 34, pp. 30392-30400, 2021b.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal attention for long-range interactions in vision transformers. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), volume 34, pp. 10008-30022 2021.
13

---

Published as a conference paper at ICLR 2023
Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin, Zhaoxu Yang. and Guoqi Li. Temporal wise attention spiking neural networks for event streams classification. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 10221-10230, 2021.
Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng. Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. arXiv preprint arXiv:2209. 13929, 2022.
Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence, 3(10):905-99) 913,
2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 558-567, 2021a.
Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan. Volo: Vision outlooker for visual recognition. arXiv preprint arXiv:2106. 131 2021b.
Jiqing Zhang, Bo Dong, Haiwei Zhang, Jianchuan Ding, Felix Heide, Baocai Yin, and Xin Yang. Spiking transformers for event-based single object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8801-8810, 2022a.
Jiyuan Zhang, Lulu Tang, Zhaofei Yu, Jiwen Lu, and Tiejun Huang. Spike transformer: Monocular depth estimation for spiking camera. In Proceedings of the European Conference on Computer Vision (ECCV), 2022b.
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), volume 33, pp. 10022-12033, 2020.
Hanle Zheng. Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going Deeper With Directly- -rriine Larger Spiking Neural Networks. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 11062-1 100701 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
Zulun Zhu, Jiaying Peng, Jintang Li, Liang Chen, Qi Yu, and Siqiang Luo. Spiking graph convolu- tional networks. Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI), pp. 2434-24400 2022. doi: 10. 20224/33/icaii222223388
14

---

Published as a conference paper at ICLR 2023
APPENDIX
A MULTIHEAD SPIKING SELF ATTENTION
In practice we reshape the Q, K, V E RTXNXD into multi-head form 7x*HxXxd, where H x d. Then we split Q, K, V into H parts and run H SSA operations, in parallel, which are called H-head SSA. The Multihead Spiking Self Attention (MSSA) is shown in follows:
Q (21.92.-1 .4H), K (k1, 12. ka), V (01, U2
(H) g.k, RxNxx (17)
MSSA (Q, KV [SSA] (g1, Ai, (1) K2 02): SSA (qn, vn)]
(18)
M, SN((N(Linear(MMSS) (Q, K, V))))
(19)
B SPIKING SELF ATTENTION AND TIME STEP
In practice, T is independent dimension for spike neuron layer. In other layers, it is merged with the batch size.
C EXPERIMENT DETAILS
C.I TRAINING
Unlike the standard ViT, Dropout and Droppath are not applied in Spikformer. We remove the layer norm before each self-attention and MLP block, and add batch norm after each linear layer instead. In all Spikformer models, the hidden dimension of MLP blocks is 4 x D, where D is the embedding dimension. in Eq. 200. we select the Sigmoid function as the surrogate function with 4.
Sigmoid(x) + exp (-ax)
(20)
For DVSI28 Gesture, we place ID max-pooling layer after and K increase the density of the data, which improves the accuracy from 97 97.9% to 98.3% in 16 time steps. We set the threshold voltage Va, of the spike neuron layer after QKTV to 0.5, while the others are set 1.
C.2 THEBORETCCAL SYNAPTIC OPERATION AND ENERGY CONSUMPTION CALCULATION
The calculation theoretical energy consumption requires first calculating the synaptic operations:
SOPs(I) fr x T FLOPs(()
(21)
where is block/layer Spikformer, fr is the firing rate the input spike train of the block/layer and T is the simulation time step of spike neuron. FLOPs(I) refers to floating point operations of I, which is the number of multiply-ame caltiply--nd-accuusee accumuaccmultte (MAC) operations. And SOPs the number of spike-based accumulate (AC) operations. We estimate the theoretical energy consumption of
Spikformer according to (Kundu et al.) 20216.10 Hu et al. 2021b Horowitz) 2014 Kundu et al. 2021a Yin et al. 2021 Panda al. [200 Yao et al. (022). We assume that the MAC and AC operations implemented on the 45nm hardware [12], where EMAG = 4.6pJ and EAC 0.9pJ. The theoretical
energy consumption of Spikformer calculated:
Egpik
EMAC x FLLNN Conv
M
+ EAC
Σ
SOP SNN Conv
+
Σ
SOP
SNN FC +
SOP!
SSA
(22)
n=
m=
1=1
15

---

Published as a conference paper at ICLR 2023
where FLSNN Cone is the first layer to encode static RGB images into spike-form. Then the SOPs of m SNN Conv layers, n SNN Fully Connected Layer (FC) and SSA are added together and multiplied by EAC. For ANNs, the theoretical energy consumption of block b is calculated:
Power(b) 4.6pJ FLOPs(b)
(23)
For SNNs, Power(b) is:
Power (b) = 0.9pJ x SOPs(b)
(24)
bdy
fire rate
block
fire rate Qf
Figure 4: Fire rate of Query, Key and Value of blocks in Spikformer-8-512 on ImageNet test set.
test accuracy %
training loss
tost loss
Splerman 20
epach
epoch
epoch
(a) Training loss
(b)
Testing loss
(c) Test accuracy
Figure 5: Training loss, testing loss and test accuracy on ImageNet.
D ADDITIONAL RESULTS
D. FIRE RATE OF QUERY, KEY AND VALUE
As shown in 4 the Query, Key and Value are very spare in SSA, causing sparse computation of SSA.
D.2
LOSS AND ACCURACY IMAGENET
We show the training loss, testing loss and test accuracy of Spikformer in Figue. Both training and testing losses decrease as the number of Spikformer blocks increases or the embedding dimension increases.
16

---

Published as a conference paper at ICLR 2023
Table 6: Additional result CIFARIO/100. Spikformer-433844 IF the integrate-and-Hie neuron.
Time
Topl-Ace
Models
Step
(%)
Spikformer--4384.w
92.39/74.28
Spikformer-4-3844 ReLU
92.98/74.32
Spikformer-4-386 LoakyRol.U
92.88/74.31
Spikformer-4-386
1
93.11/74.37
Spikformer-4-384, IF
95.33/78.14
Table 7: Transfer Learning on CIFARIO/100.
Models
CIFARIO
CIFARI00
Spikfonmer-4338
95.54
79.96
Spikformer-8-33
96.64
82.09
Spikformer-8-512
97.03
83.83
D.3 ADDITIONAL ACCURACY RESULTS ON CIFAR
We conduct additional experiments on CIFAR as shown in Tab.
D.4 ANALYSIS OF SELF-ATTENTION VARIANTS NOT CONVERGING ON IMAGENET
The reason that the three models do not converge in Tab. is explain follows. As shown in Figure. [a)), the gradient of sigmoid surrogate function vanishes when the difference between the average input value Vi and the firing threshold VAA large or too small. We collect the output value of Q * after one training eopch of Spikformer-85512.. 1. Spikformer-8-511 Spikformer-& ---mmer----1. LeakyReLU, and Spikformer-8-512 SSA. which will be sent to the spike neuron layer as the input value V,, as shown in Eq. (15) Compared to the other three variants, shown in Spikformer-8-512 SSA is controlled in a suitable range.
ReLU.
Figure. 6 (b), the value QK Therefore, SSA has stable surrogate gradients during training and converges easily.
Sigmold suringate function
Outgan
vallie
V: Vi
block
aa) Sigmoid
(b) QKT value
Figure 6: (a) the sigmoid surrogate function and its gradient curve. (b) the value of QKTV
D.5 TRANSFER LEARNING
We transfer Spikformer to the downstream CIFAR dataset. The pre-trained Spikformer-4-3 and Doikformer-8-384455 on ImageNet are finetuned with 60 epochs. The input size of CIFAR is 224 x 224. The remaining hyperparameters are the same as the ones directly trained on CIFAR. As shown in Tab. Spikformer shows high transfer ability.
17


Reviews:
[{"review_id": "neo-r6nE08", "reviewer": "Reviewer_cw3J", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors provide a feasible implementation of SNN-oriented self-attention mechanism and Vision Transformer, obtaining superior results on both static and neuromorphic benchmarks.", "summary_of_the_review": "1. In Eqn. (16), additional SN-BN-Linear is applied to SSA’. What is the role of this extra part? \n2. Table 1. ①Why is the ratio of (SOPs of A_SSA)/(FLOPs of A_ReLU) much smaller than those on other datasets? ② It is stated in the paper that the VSA with float-point-form Q, K and softmax is redundant for spike-form X, V, which cannot get more information from X, V than SSA. Does it mean that a continuous attention map cannot work better than the spike-formed attention map? Can the authors provide more explanations for this and why the opposite conclusion is drawn from Table 5? ③ It seems that sparsity has been taken into account for both SOPs and FLOPs (in ReLU), but this is not explicitly stated and seems confusing. \n3. The authors use the data 77fJ/SOP for their energy estimation. However, it only stands for the Energy per Synaptic Event reported in ROLLS, while there are other operations like producing a spike which consumes 3.7pJ. The authors should provide a more rigorous comparison in terms of energy consumption.\n4. I noticed that the model used skip connections, and I am curious about how the two branches of spikes are merged. If the two spike trains are added together directly, the data will not be limited to the pure spike form and then does the first linear layer in the subsequent block still avoid multiplications? \n5. What is the scaling factor s in Eqn. (15)? \n6. In Table 3, it would be better to compare with ANN-Transformers instead of ANN-ResNet19.", "strength_and_weaknesses": "Strength:\n1.\tThe first time to implement self-attention and transformer in large-scale model and dataset.\n2.\tThe novel module pays attention to spike-based operations and avoids multiplications to obtain high energy efficiency.\n3.\tThe model achieves new state-of-the-art results on various datasets.\n\nWeakness:\n1.\tThe paper provides much about the experimental results, while the contents for the method and motivation seem insufficient.\n2.\tNot much insights have been given as to why spiking self-attention should be designed in this way.\n3.\tOther concerns detailed in Summary.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fEyizhTxeIc", "reviewer": "Reviewer_6yty", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the spiking version of the transformer by designing spike-form of Query, Key and Value of self-attention. The experiments are performed on both static and neuromorphic datasets for the classification tasks.", "summary_of_the_review": "Overall, the significance of this work is limited, and the authors over-claimed their contributions.", "strength_and_weaknesses": "++ The figures are nice\n\n--The novelty of this paper is limited. The major contribution is that the authors apply the transformer framework to SNNs. The performance of the proposed spiking transformer is lower than the transformer of ANNs. This research cannot give full play to SNN strengths, like temporal processing capability, and will cause researchers to go astray. \n\n--The authors over-claimed their contributions. The authors claim that they are the first to implement Transformer in SNNs. Actually, there exist many spiking transformer works, e.g., [1-3].\n\n-- The authors use floating point operations to add spikes, which defeats the purpose of SNNs.\n\n--The authors are not familiar with the research progress of SNNs. There are many of the latest works, which are ignored by the authors. E.g. [4-7].\n \n--The comparison is unfair. As the network structure of the spiking transformer differs from SEW-ResNet-152, there is no doubt the performance will be better. I do not think it is a fair comparison. Please show the performance of the corresponding ANN. \n\n--Ignoring all that, the results are not convincing. The performance of the proposed method is not SOTA. The authors only chose favourable comparisons. For example, TET achieves 83.17% accuracy on CIFAR10-DVS dataset with 10 steps, while the proposed method gets 80.9% with 16 time-steps. \n\n--The authors did not compare the training cost of the methods. Please add it in Table 2. \n\n--The ANN2SNN method achieves better performance with the increase of time-steps. How about spiking transformer? If you train the network with time-steps 4 on ImageNet, please show the performance of the network when the time-steps are 2,3,10,20,50.\n \n--Please compare SOPs and Power in Table 3 and 4.\n\n[1] Zhang, Jiqing, et al. \"Spiking Transformers for Event-Based Single Object Tracking.\" CVPR. 2022.\n\n[2] Zhang, Jiyuan, et al. \"Spike Transformer: Monocular Depth Estimation for Spiking Camera.\"ECCV. 2022\n\n[3] Mueller, Etienne, et al. \"Spiking Transformer Networks: A Rate Coded Approach for Processing Sequential Data.\" 2021 7th International Conference on Systems and Informatics (ICSAI). IEEE, 2021.\n\n[4] Xiao, Mingqing, et al. \"Training feedback spiking neural networks by implicit differentiation on the equilibrium state.\" Advances in Neural Information Processing Systems 34 (2021): 14516-14528.\n\n[5] Meng, Qingyan, et al. \"Training much deeper spiking neural networks with a small number of time-steps.\" Neural Networks153 (2022): 254-268.\n\n[6] Bu, Tong, et al. \"Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks.\" International Conference on Learning Representations. 2022.\n\n[7] Wang, Yuchen, et al. \"Signed Neuron with Memory: Towards Simple, Accurate and High-Efficient ANN-SNN Conversion.\" International Joint Conference on Artificial Intelligence. 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Eq9LnYIZOQq", "reviewer": "Reviewer_vg9k", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce Spikformer a SNN version of Transformer, which includes a novel implementation of self-attention with spiking representations. The authors show that their proposed architecture outperforms SOTA SNNs on both traditional image tasks as well as neuromorphic tasks.  ", "summary_of_the_review": "Overall a solid contribution to the field of SNNs research and one that should deserve a spot in the conference. ", "strength_and_weaknesses": "Strengths \nThe paper's motivation is a relatively simple idea (implement a SNN version of Transformers), but its implementation is non-trivial given the constraints of SNNs. The authors' solution is a simple and efficient one, which sets solid ground for future iterations. The paper strengths are the novel architecture contributions and the SOTA results when compared with other SNNs.\n\nWeaknesses\nEven though it is not the authors' motivation, it can be argued that this paper does not offer an alternative architecture to Transformers. Transformers likely outperform Spikformers in many ways and most importantly in prediction quality. On the other hand, we are also not learning about new biological features that are important for brain computing. This paper is about advancing the field of SNN research, which has not yet produced an algorithm that can broadly compete with ANNs. Yes, we know for a fact that NNs in the brain are SNNs, but we still do not know why, and this paper is obviously not helping us clarify that.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "CVeuKxjZc0Q", "reviewer": "Reviewer_qGTH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of adding spiking neurons to transformer architecture. ", "summary_of_the_review": "Based on the quality of this paper, I intend to give acceptance. If the concerns can be adequately addressed, I will increase my score. ", "strength_and_weaknesses": "Strength:\n\n+ Clear paper structure, neat paper presentation. \n\n+ Compared to spiking convolutional architectures the accuracy are higher. \n\nWeakness: \n\n- Seems like the method to alleviate the full precision multiplication is to add more LIF neurons. \n\n- Can authors at least provide transfer learning results since high transferability is the core argument for artificial transformers? Also, more LIF may provide higher training latency in practice. Can the authors compare training GPUs between SpikeTransformer and SpikeCNN?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster