Title: Spike-driven Transformer
Venue: NeurIPS 2023
Accepted: yes
Keywords: Spiking Neural Networks; Transformer; Neuromorphic Computing; Event-driven; Linear Attention
Abstract: Spiking Neural Networks (SNNs) provide an energy-efficient deep learning option due to their unique spike-based event-driven (i.e., spike-driven) paradigm. In this paper, we incorporate the spike-driven paradigm into Transformer by the proposed Spike-driven Transformer with four unique properties: (1) Event-driven, no calculation is triggered when the input of Transformer is zero; (2) Binary spike communication, all matrix multiplications associated with the spike matrix can be transformed into sparse additions; (3) Self-attention with linear complexity at both token and channel dimensions; (4) The operations between spike-form Query, Key, and Value are mask and addition. Together, there are only sparse addition operations in the Spike-driven Transformer. To this end, we design a novel Spike-Driven Self-Attention (SDSA), which exploits only mask and addition operations without any multiplication, and thus having up to $87.2\times$ lower computation energy than vanilla self-attention. Especially in SDSA, the matrix multiplication between Query, Key, and Value is designed as the mask operation. In addition, we rearrange all residual connections in the vanilla Transformer before the activation functions to ensure that all neurons transmit binary spike signals. It is shown that the Spike-driven Transformer can achieve 77.1\% top-1 accuracy on ImageNet-1K, which is the state-of-the-art result in the SNN field.

Spike-driven Transformer

Abstract

Spiking Neural Networks (SNNs) provide an energy-efficient deep learning option due to their unique spike-based event-driven (i.e., spike-driven) paradigm. In this paper, we incorporate the spike-driven paradigm into Transformer by the proposed Spike-driven Transformer with four unique properties: i) Event-driven, no calculation is triggered when the input of Transformer is zero; ii) Binary spike communication, all matrix multiplications associated with the spike matrix can be transformed into sparse additions; iii) Self-attention with linear complexity at both token and channel dimensions; iv) The operations between spike-form Query, Key, and Value are mask and addition. Together, there are only sparse addition operations in the Spike-driven Transformer. To this end, we design a novel SpikeDriven Self-Attention (SDSA), which exploits only mask and addition operations without any multiplication, and thus having up to 87 . 2 × lower computation energy than vanilla self-attention. Especially in SDSA, the matrix multiplication between Query, Key, and Value is designed as the mask operation. In addition, we rearrange all residual connections in the vanilla Transformer before the activation functions to ensure that all neurons transmit binary spike signals. It is shown that the Spikedriven Transformer can achieve 77.1% top-1 accuracy on ImageNet-1K, which is the state-of-the-art result in the SNN field. All source code and models are available at https://github.com/BICLab/Spike-Driven-Transformer .

1 Introduction

One of the most crucial computational characteristics of bio-inspired Spiking Neural Networks (SNNs) [1] is spike-based event-driven (spike-driven): i) When a computation is event-driven, it is triggered sparsely as events (spike with address information) occur; ii) If only binary spikes (0 or 1) are employed for communication between spiking neurons, the network's operations are synaptic ACcumulate (AC). When implementing SNNs on neuromorphic chips, such as TrueNorth [2], Loihi [3], and Tianjic [4], only a small fraction of spiking neurons at any moment being active and the rest being idle. Thus, spike-driven neuromorphic computing that only performs sparse addition operations is regarded as a promising low-power alternative to traditional AI [5, 6, 7].

Although SNNs have apparent advantages in bio-plausibility and energy efficiency, their applications are limited by poor task accuracy. Transformers have shown high performance in various tasks for their self-attention [8, 9, 10]. Incorporating the effectiveness of Transformer with the high energy efficiency of SNNs is a natural and exciting idea. There has been some research in this direction, but all so far have relied on 'hybrid computing'. Namely, Multiply-and-ACcumulate (MAC) operations dominated

(b) Spike-Driven Self-Attention (SDSA)

Hadamard product (Element-wise mask)

Spiking neuron

Column summation

∑

Dot product

×

Broadcasted element-wise multiplication

∎

Column mask

Linear

Linear

Linear

…

…

…

…

…

…

…

𝑄

…

…

…

…

…

…

…

…

…

…

…

…

…

…

𝐾

𝑉

(a) Vanilla Self-Attention (VSA)

Transpose

𝑛 × 𝑑

𝑑 × 𝑛

Scale

Softmax

𝑛 × 𝑛

Linear

𝑛 × 𝑑

𝑂 𝐷𝑁 2 + 𝐷𝑁 2

𝑛 × 𝑑

𝑋

Linear

Linear

Linear

…

…

…

…

…

…

…

𝑄

𝐾

𝑉

…

…

…

…

…

…

…

…

…

…

…

…

…

…

Binary spike input

𝑛 × 𝑑

𝑛 × 𝑑

𝑛 × 𝑑

𝑂 𝑁𝐷 + 0

Linear

𝑛 × 𝑑

∑

1 × 𝑑

∎

SDSA

Version 1

Linear

Linear

Linear

∅ ∙

:

∅ ∙

:

𝑄

𝐾

𝑉

…

…

…

…

…

…

…

1

2

3

n

1

2

d

…

…

…

1

2

h=d

…

…

…

1

2

h=d

𝑛 × 𝑑

Linear

…

1 × 𝑑

…

𝑂 0 + 𝑁𝐷

1 × 𝑑

𝑛 × 𝑑

𝑆

Binary spike input

×

SDSA

Version 2

=

෠

𝑋

መ

𝑆

መ

𝑆

𝑆

by vanilla Transformer components and AC operations caused by spiking neurons both exist in the existing spiking Transformers. One popular approach is to replace some of the neurons in Transformer with spiking neurons to do a various of tasks [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22], and keeping the MAC-required operations like dot-product, softmax, scale, etc.

Though hybrid computing helps reduce the accuracy loss brought on by adding spiking neurons to the Transformer, it can be challenging to benefit from SNN's low energy cost, especially given that current spiking Transformers are hardly usable on neuromorphic chips. To address this issue, we propose a novel Spike-driven Transformer that achieves the spike-driven nature of SNNs throughout the network while having great task performance. Two core modules of Transformer, Vanilla Self-Attention (VSA) and Multi-Layer Perceptron (MLP), are re-designed to have a spike-driven paradigm.

The three input matrices for VSA are Query ( Q ), Key ( K ), and Value ( V ), (Fig. 1(a)). Q and K first perform similarity calculations to obtain the attention map, which includes three steps of matrix multiplication, scale and softmax. The attention map is then used to weight the V (another matrix multiplication). The typical spiking self-attentions in the current spiking Transformers [20, 19] would convert Q , K , V into spike-form before performing two matrix multiplications similar to those in VSA. The distinction is that spike matrix multiplications can be converted into addition, and softmax is not necessary [20]. But these methods not only yield large integers in the output (thus requiring an additional scale multiplication for normalization to avoid gradient vanishing), but also fails to exploit the full energy-efficiency potential of the spike-driven paradigm combined with self-attention.

We propose Spike-Driven Self-Attention (SDSA) to address these issues, including two aspects (see SDSA Version 1 in Fig. 1(b)): i) Hadamard product replaces matrix multiplication; ii) matrix column-wise summation and spiking neuron layer take the role of softmax and scale. The former can be considered as not consuming energy because the Hadamard product between spikes is equivalent to the element-wise mask. The latter also consumes almost no energy since the matrix to be summed column-by-column is very sparse (typically, the ratio of non-zero elements is less than 0.02). We also observe that SDSA is a special kind of linear attention [23, 24], i.e., Version 2 of Fig. 1(b). In this view, the spiking neuron layer that converts Q , K , and V into spike form is a kernel function.

Additionally, existing spiking Transformers [20, 12] usually follow the SEW-SNN residual design [25], whose shortcut is spike addition and thus outputs multi-bit (integer) spikes. This shortcut can satisfy event-driven, but introduces integer multiplication. We modified the residual connections throughout the Transformer architecture as shortcuts between membrane potentials [26, 27] to address this issue (Section 3.2). The proposed Spike-driven Transformer improves task accuracy on both static and neuromorphic event-based datasets. The main contributions of this paper are as follows:

We propose a novel Spike-driven Transformer that only exploits sparse addition. This is the first time that the spike-driven paradigm has been incorporated into Transformer, and the proposed model is hardware-friendly to neuromorphic chips.

We design a Spike-Driven Self-Attention (SDSA). The self-attention operator between spike Query, Key, Value is replaced by mask and sparse addition with essentially no energy consumption. SDSA is computationally linear in both tokens and channels. Overall, the energy cost of SDSA (including Query, Key, Value generation parts) is 87 . 2 × lower than its vanilla self-attention counterpart.

We rearrange the residual connections so that all spiking neurons in Spike-driven Transformer communicate via binary spikes.

Extensive experiments show that the proposed architecture can outperform or comparable to StateOf-The-Art (SOTA) SNNs on both static and neuromorphic datasets. We achieved 77.1% accuracy on ImageNet-1K, which is the SOTA result in the SNN field.

2 Related Works

Bio-inspired Spiking Neural Networks can profit from advanced deep learning and neuroscience knowledge concurrently [28, 5, 7]. Many biological mechanisms are leveraged to inspire SNN's neuron modeling [1, 29], learning rules [30, 31], etc. Existing studies have shown that SNNs are more suited for incorporating with brain mechanisms, e.g., long short-term memory [32, 33], attention [34, 27, 35], etc. Moreover, while keeping its own spike-driven benefits, SNNS have greatly improved its task accuracy by integrating deep learning technologies like network architecture [26, 25, 36], gradient backpropagation [37, 38], normalization [39, 40], etc. Our goal is to combine SNN and Transformer architectures. One way is to discretize Transformer into spike form through neuron equivalence [41, 42], i.e., ANN2SNN, but this requires a long simulation timestep and boosts the energy consumption. We employ the direct training method, using the first SNN layer as the spike encoding layer and applying surrogate gradient training [43].

Neuromorphic Chips. As opposed to the compute and memory separated processors used in ANNs, neuromorphic chips use non-von Neumann architectures, which are inspired by the structure and function of the brain [5, 7, 28]. Because of the choice that uses spiking neurons and synapses as basic units, neuromorphic chips [44, 45, 46, 2, 47, 3, 4] have unique features, such as highly parallel operation, collocated processing and memory, inherent scalability, and spike-driven computing, etc. Typical neuromorphic chips consume tens to hundreds of mWs [48]. Conv and MLP in neuromorphic chips are equivalent to a cheap addressing algorithm [49] since the sparse spike-driven computing, i.e., to find out which synapses and neurons need to be involved in the addition operation. Our Transformer design strictly follows the spike-driven paradigm, thus it is friendly to deploy on neuromorphic chips.

Efficient Transformers. Transformer and its variants have been widely used in numerous tasks, such as natural language processing [8, 50, 51] and computer vision [52, 53, 54]. However, deploying these models on mobile devices with limited resources remains challenging because of their inherent complexity [24, 55]. Typical optimization methods include convolution and self-attention mixing [55, 56], Transformer's own mechanism (token mechanism [57, 58], self-attention [59, 60], multihead [61, 62] and so on) optimization, etc. An important direction for efficient Transformers is linear attention on tokens since the computation scale of self-attention is quadratic with token number. Removing the softmax in self-attention and re-arranging the computation order of Query, Key, and Value is the main way to achieve linear attention [60, 63, 23, 64, 65, 62, 66]. For the spiking Transformer, softmax cannot exist, thus spiking Transformer can be a kind of linear attention.

SPS

Spike-driven Encoder Block

×𝑳

+

MLP

×3

Conv 2D

BN

MP

Conv 2D

BN

MP

Conv 2D

BN

+

RPE

Spike-Driven Self-Attention (SDSA)

Mask & Add

Linear

BN

𝑄

𝐾

𝑉

Linear

BN

Linear

BN

Linear

BN

MP

Layer

+

Element-wise

Add

Membrane

Potential

Shortcut

PSM

𝑢

𝑈0

SDSA

𝑈𝑙

′

Classification

Head

Linear

BN

Linear

BN

+

3 Spike-driven Transformer

We propose a Spike-driven Transformer, which incorporates Transformer into the spike-driven paradigm with only sparse addition. We first briefly introduce the spike neuron layer, then introduce the overview and components of the Spike-driven Transformer one by one.

The spiking neuron model is simplified from the biological neuron model [67, 29]. Leaky Integrateand-Fire (LIF) spiking neuron [1], which have biological neuronal dynamics and are easy to simulate on a computer, is uniformly adopted in this work. The dynamics of the LIF layer [37] is governed by

where t denote the timestep, U [ t ] means the membrane potential which is produced by coupling the spatial input information X [ t ] and the temporal input H [ t -1] , where X [ t ] can be obtained through operators such as Conv, MLP, and self-attention. When membrane potential exceeds the threshold u th , the neuron will fire a spike, otherwise it will not. Thus, the spatial output tensor S [ t ] contains only 1 or 0. Hea( · ) is a Heaviside step function that satisfies Hea( x ) = 1 when x ≥ 0 , otherwise Hea( x ) = 0 . H [ t ] indicates the temporal output, where V reset denotes the reset potential which is set after activating the output spiking. β < 1 is the decay factor, if the spiking neuron does not fire, the membrane potential U [ t ] will decay to H [ t ] .

3.1 Overall Architecture

Fig. 2 shows the overview of Spike-driven Transformer that includes four parts: Spiking Patch Splitting (SPS), SDSA, MLP, and a linear classification head. For the SPS part, we follow the design in [20]. Given a 2D image sequence I ∈ R T × C × H × W , the Patch Splitting Module (PSM), i.e., the first four Conv layers, linearly projects and splits it into a sequence of N flattened spike patches s with D dimensional channel, where T (images are repeated T times in the static dataset as input), C , H , and W denote timestep, channel, height and width of the 2D image sequence. Another Conv layer

is then used to generate Relative Position Embedding (RPE). Together, the SPS part is written as:

where u and U 0 are the output membrane potential tensor of PSM and SPS respectively, SN ( · ) denote the spike neuron layer. Note, in Eq 6, before executing Conv 2 d ( · ) , s ∈ R T × N × D will be transposed into s ∈ R T × C × H × W . We pass the U 0 to the L -block Spike-driven Transformer encoder, which consists of a SDSA and a MLP block. Residual connections are applied to membrane potentials in both SDSA and MLP block. SDSA provides an efficient approach to model the local-global information of images utilizing spike Q , K , and V without scale and softmax (see Sec. 3.3). A Global Average-Pooling (GAP) is utilized on the processed feature from spike-driven encoder and outputs the D -dimension channel which will be sent to the fully-connected-layer Classification Head (CH) to output the prediction Y . The three parts SDSA, MLP and CH can be written as follows:

where U ′ l and S ′ l represent membrane potential and spike tensor output by SDSA at l -th layer.

3.2 Membrane Shortcut in Spike-driven Transformer

Residual connection [68, 69] is a crucial basic operation in Transformer architecture. There are three shortcut techniques in existing Conv-based SNNs [27]. Vanilla Res-SNN [39], similar to vanilla Res-CNN [68], performs a shortcut between membrane potential and spike. Spike-Element-Wise (SEW) Res-SNN [25] employs a shortcut to connect the output spikes in different layers. Membrane Shortcut (MS) Res-SNN [26], creating a shortcut between membrane potential of spiking neurons in various layers. There is no uniformly standard shortcut in the current SNN community, and SEW shortcut is adopted by existing spiking Transformers [20, 12]. As shown in Eq. 7, Eq. 9 and Eq. 11, we leverage the membrane shortcut in the proposed Spike-driven Transformer for four reasons:

Spike-driven refers to the ability to transform matrix multiplication between weight and spike tensors into sparse additions. Only binary spikes can support the spike-driven function. However, the values in the spike tensors are multi-bit (integer) spikes, as the SEW shortcut builds the addition between binary spikes. By contrast, as shown in Eq. 8, Eq. 10, Eq. 11, SN is followed by the MS shortcut, which ensures that there are always only binary spike signals in the spike tensor.

High performance . The task accuracy of MS-Res-SNN is higher than that of SEW-Res-SNN [26, 27, 25], also in Transformer-based SNN (see Table 5 in this work).

Bio-plausibility . MSshortcut can be understood as an approach to optimize the membrane potential distribution. This is consistent with other neuroscience-inspired methods to optimize the internal dynamics of SNNs, such as complex spiking neuron design [70], attention mechanism [27], long short-term memory [33], recurrent connection [71], information maximization [72], etc.

Dynamical isometry . MS-Res-SNN has been proven [26] to satisfy dynamical isometry theory [73], which is a theoretical explanation of well-behaved deep neural networks [74, 75].

3.3 Spike-driven Self-Attention

Vanilla Self-Attention (VSA). Given a float-point input feature sequence X ∈ R N × D , float-point Query ( Q ), Key ( K ), and Value ( V ) in R N × D are calculated by three learnable linear matrices, respectively. Standard scaled dot-product self-attention (Fig. 2(a)) is computed as [52]:

where d = D/H is the feature dimension of one head and H is the head number, √ d is the scale factor. The time complexity of VSA is O ( N 2 D + N 2 D ) .

Spike-Driven Self-Attention (SDSA) Version 1. As shown in Fig. 2(b) left part, given a spike input feature sequence S ∈ R T × N × D , float-point Q , K , and V in R T × N × D are calculated by three learnable linear matrices, respectively. Note, the linear operation here is only addition, because the input S is a spike tensor. A spike neuron layer SN ( · ) follows, converting Q , K , V into spike tensor Q S , K S , and V S . SDSA Version 1 (SDSA-V1) is presented as:

where ⊗ is the Hadamard product, g ( · ) is used to compute the attention map, SUM c ( · ) represents the sum of each column. The outputs of both g ( · ) and SUM c ( · ) are D -dimensional row vectors. The Hadamard product between spike tensors is equivalent to the mask operation.

Discussion on SDSA. Since the Hadamard product among Q S , K S , and V S in R N × D (we here assume T = 1 for mathematical understanding) can be exchanged, Eq. 14 can also be written as:

Note, Eq.14 and Eq.15 are equivalent functionally. In this view, Eq.15 is a linear attention [23, 63] whose computational complexity is linear in token number N because K S and V S can participate in calculate first. This is thanks to the softmax operation in the VSA is dropped here. The function of softmax needs to be replaced by the kernel function. Specific to our SDSA, SN ( · ) is the kernel function. Further, we can assume a special case [62], H = D , i.e., the number of channels per head is one. After the self-attention operation is performed on the H heads respectively, the outputs are concatenated together. Specifically,

where Q i , K i , V i in R N × 1 are the i -th vectors in Q,K,V respectively, ⊙ is the dot product operation. The output of g ( K i , V i ) is a scalar, 0 or 1. Since the operation between SN ( Q i ) and g ( K i , V i ) is a mask, the whole SDSA only needs to be calculated H = D times for g ( K i , V i ) . The computational complexity of SDSA is O (0 + ND ) , which is linear with both N and D (see Fig. 1(b) right part). Vectors K i and V i are very sparse, typically less than 0.01 (Table 2). Together, the whole SDSA only needs about 0 . 02 ND times of addition, and its energy consumption is negligible.

Interestingly, Eq. 15 actually converts the soft vanilla self-attention to hard self-attention, where the attention scores in soft and hard attention are continuous- and binary-valued, respectively [76]. Thus, the practice of the spike-driven paradigm in this work leverages binary self-attention scores to directly mask unimportant channels in the sparse spike Value tensor. Although this introduces a slight loss of accuracy (Table 5), SDSA( · ) consumes almost no energy.

4 Theoretical Energy Consumption Analysis

Three key computational modules in deep learning are Conv, MLP, and self-attention. In this Section, We discuss how the spike-driven paradigm achieves high energy efficiency on these operators.

Spike-driven in Conv and MLP. Spike-driven combines two properties, event-driven and binary spike-based communication. The former means that no computation is triggered when the input is zero. The binary restriction in the latter indicates that there are only additions. In summary, in spike-driven Conv and MLP, matrix multiplication is transformed into sparse addition, which is implemented as addressable addition in neuromorphic chips [49].

Spike-driven in Self-attention. Q S , K S , V S in spiking self-attention involve two matrix multiplications. One approach is to perform multiplication directly between Q S , K S , V S , which is then converted to sparse addition, like spike-driven Conv and MLP. The previous work [20] did just that. We provide a new scheme that performs element-wise multiplication between Q S , K S , V S . Since all elements in spike tensors are either 0 or 1, element multiplication is equivalent to a mask operation with no energy consumption. Mask operations can be implemented in neuromorphic chips through addressing algorithms [49] or AND logic operations [4].

Energy Consumption Comparison. The times of floating-point operations (FLOPs) is often used to estimate the computational burden in ANNs, where almost all FLOPs are MAC. Under the same

architecture, the energy cost of SNN can be estimated by combining the spike firing rate R and simulation timestep T if the FLOPs of ANN is known. Table 1 shows the energy consumption of Conv, self-attention, and MLP modules of the same scale in vanilla and our Spike-driven Transformer.

5 Experiments

Weevaluate our method on both static datasets ImageNet [77], CIFAR-10/100 [78], and neuromorphic datasets CIFAR10-DVS [79], DVS128 Gesture [80].

Experimental Setup on ImageNet. For the convenience of comparison, we generally continued the experimental setup in [20]. The input size is set to 224 × 224 . The batch size is set to 128 or 256 during 310 training epochs with a cosine-decay learning rate whose initial value is 0 . 0005 . The optimizer is Lamb. The image is divided into N = 196 patches using the SPS module. Standard data augmentation techniques, like random augmentation, mixup, are also employed in training. Details of the training and experimental setup on ImageNet are given in the supplementary material.

Accuracy analysis on ImageNet. Our experimental results on ImageNet are given in Table 3. We first compare our model performance with the baseline spiking Transformer (i.e., SpikFormer [20]). The five network architectures consistent with those in SpikFormer are adopted by this work. We can see that under the same parameters, our accuracies are significantly better than the corresponding baseline models. For instance, the Spike-driven Transformer-8-384 is 2.0% higher than SpikFormer-8-384. It is worth noting that the Spike-driven Transformer-8-768 obtains 76.32% (input 224 × 224) with 66 . 34 M, which is 1.5% higher than the corresponding SpikFormer. We further expand the inference resolution to 288 × 288, obtaining 77.1%, which is the SOTA result of the SNN field on ImageNet.

We then compare our results with existing ResSNNs. Whether it is vanilla Res-SNN [39], MSRes-SNN [26] or SEW-Res-SNN [25], the accuracy of Spike-driven Transformer-8-768 (77.1%) is the highest. Att-MS-Res-SNN [27] also achieves 77.1% accuracy by plugging an additional attention auxiliary module [81, 82] in MS-Res-SNN, but it destroys the spike-driven nature and requires more parameters (78.37M vs. 66.34M) and training time (1000epoch vs. 310epoch). Furthermore, the proposed Spike-

driven Transformer outperforms by more than 72% at various network scales, while Res-SNNs have lower performance with a similar amount of parameters. For example, Spike-driven Transformer6-512 (This work) vs. SEW-Res-SNN-34 vs. MS-Res-SNN-34: Param, 23.27M vs. 21.79M vs. 21.80M; Acc, 74.11% vs. 67.04% vs. 69.15%.

Power analysis on ImageNet. Compared with prior works, the Spike-driven Transformer shines in energy cost (Table 3). We first make an intuitive comparison of energy consumption in the SNN field. Spike-driven Transformer-8-512 (This work) vs. SEW-Res-SNN-50 vs. MS-Res-SNN-34: Power, 4.50mJ vs. 4.89mJ vs. 5.11mJ; Acc, 74.57% vs. 67.78% vs. 69.42%. That is, our model achieves +6.79% and +5.15% accuracy higher than previous SEW and MS Res-SNN backbones with lower energy consumption. What is more attractive is that the energy efficiency of the Spikedriven Transformer will be further expanded as the model scale grows because its computational complexity is linear in both token and channel dimensions. For instance, in an 8-layer network, as the channel dimension increases from 384 to 512 and 768 , SpikFormer [20] has 1.98 × (7.73mJ/3.90mJ), 2.57 × (11.57mJ/4.50mJ), and 3.52 × (21.47mJ/6.09mJ) higher energy consumption than our Spikedriven Transformer. At the same time, our task performance on these three network structures has improved by +2.0% , +1.2% , and +1.5% , respectively.

Then we compare the energy cost between Spikedriven and ANN Transformer. Under the same structure, such as 8-512, the power required by the ANN-ViT (41.77mJ) is 9.3 × that of the spikedriven counterpart (4.50mJ). Further, the energy advantage will extend to 36.7 × if we set T = 1 in the Spike-driven version (1.13mJ). Although the accuracy of T = 1 (here we are direct training) will be lower than T = 4 , it can be compensated

Table 4: Energy Consumption of Self-attention. E 1 and E 2 (including energy consumption to generate Q,K,V ) represent the power of selfattention mechanism in ANN and spike-driven.

by special training methods [86] in future work.

Sparse spike firing is the key for Spike-driven Transformer to achieve high energy efficiency. As shown in Table 2, the Spike Firing Rate (SFR) of the self-attention part is very low, where the SFR of Q S and Q K are both less than 0.01. Since the mask (Hadamard product) operation does not consume energy, the number of additions required by the SUM c ( Q S ⊗ K S ) is less than 0 . 02 ND times. The operation between the vector output by g ( Q S , K S ) and V S is still a column mask that does not consume energy. Consequently, in the whole self-attention part, the energy consumption of spike-driven self-attention can be lower than 87 . 2 × of ANN self-attention (see Table 4).

Experimental results on CIFAR-10/100, CIFAR10-DVS, and DVS128 Gesture are conducted in Table 5. These four datasets are relatively small compared to ImageNet. CIFAR10/100 are static image classification datasets. Gesture and CIFAR10-DVS are neuromorphic action classification datasets, which need to convert the event stream into frame sequences

before processing. DVS128 Gesture is a gesture recognition dataset. CIFAR10-DVS is a neuromorphic dataset converted from CIFAR-10 by shifting image samples to be captured by the DVS camera. We basically keep the experimental setup in [20], including the network structure, training settings, etc., and details are given in the supplementary material. As shown in Table 5, we achieve SOTA results on Gesture (99.3%) and CIFAR-10 (95.6%), and comparable results to SOTA on other datasets.

Ablation study. To implement the spike-driven paradigm in Transformer, we design a new SDSA module and reposition the residual connections in the entire network based on the SpikFormer [20]. We organize ablation studies on CIFAR10/100 to analyze their impact. Results are given in Table 5. We adopt spiking Transformer-2-512 as the baseline structure. It can be observed that SDSA incurs a slight performance

Input

Image

Attention

Map 1

Attention

Block 1

Block 2

Block 3

Block 4

𝑉

!

𝑉 "

!

0.39

0.34

0.38

0.48

0.36

0.28

0.30

0.37

0.33

0.23

0.21

0.26

0.30

0.17

0.12

0.14

0.27

0.12

0.04

0.03

0.09

0.05

0.04

0.02

0.07

0.04

0.03

0.02

0.05

0.02

0.02

0.01

0.03

0.01

0.01

0.01

0.01

0.00

0.00

0.00

loss. As discussed in Section 3.3, SDSA actually masks some unimportant channels directly. In Fig. 3, we plot the attention maps (the detailed drawing method is given in the supplementary material), and we can observe: i) SDSA can optimize intermediate features, such as masking background information; ii) SDSA greatly reduces the spike firing rate of ˆ V S , thereby reducing energy cost. On the other hand, the membrane shortcut leads to significant accuracy improvements, consistent

with the experience of Conv-based MS-SNN [26, 25]. Comprehensively, the proposed Spike-driven Transformer simultaneously achieves better accuracy and lower energy consumption (Table 3).

6 Conclusion

We propose a Spike-driven Transformer that combines the low power of SNN and the excellent accuracy of the Transformer. There is only sparse addition in the proposed Spike-driven Transformer. To this end, we design a novel Spike-Driven Self-Attention (SDSA) module and rearrange the location of residual connections throughout the network. The complex and energy-intensive matrix multiplication, softmax, and scale in the vanilla self-attention are dropped. Instead, we employ mask, addition, and spike neuron layer to realize the function of the self-attention mechanism. Moreover, SDSA has linear complexity with both token and channel dimensions. Extensive experiments are conducted on static image and neuromorphic datasets, verifying the effectiveness and efficiency of the proposed method. We hope our investigations pave the way for further research on Transformer-based SNNs and inspire the design of next-generation neuromorphic chips.

References

Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural Networks , 10(9):1659-1671, 1997.

Paul A Merolla, John V Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science , 345(6197):668-673, 2014.

Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro , 38(1):82-99, 2018.

Jing Pei, Lei Deng, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature , 572(7767):106-111, 2019.

Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature , 575(7784):607-617, 2019.

Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence , 3(10):905-913, 2021.

Catherine D Schuman, Shruti R Kulkarni, Maryam Parsa, J Parker Mitchell, Bill Kay, et al. Opportunities for neuromorphic computing algorithms and applications. Nature Computational Science , 2(1):10-19, 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pages 5998-6008, 2017.

Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE transactions on Pattern Analysis and Machine Intelligence , 45(1):87-110, 2022.

Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR) , 54(10s):1-41, 2022.

Yudong Li, Yunlin Lei, and Xu Yang. Spikeformer: A novel architecture for training highperformance low-latency spiking neural network. arXiv preprint arXiv:2211.10686 , 2022.

Nathan Leroux, Jan Finkbeiner, and Emre Neftci. Online transformers with spiking neurons for fast prosthetic hand control. arXiv preprint arXiv:2303.11860 , 2023.

Jiqing Zhang, Bo Dong, Haiwei Zhang, Jianchuan Ding, Felix Heide, Baocai Yin, and Xin Yang. Spiking transformers for event-based single object tracking. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , pages 8801-8810, 2022.

Jiyuan Zhang, Lulu Tang, Zhaofei Yu, Jiwen Lu, and Tiejun Huang. Spike transformer: Monocular depth estimation for spiking camera. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII , pages 34-52. Springer, 2022.

Zhengkun Tian, Jiangyan Yi, Jianhua Tao, Ye Bai, Shuai Zhang, and Zhengqi Wen. SpikeTriggered Non-Autoregressive Transformer for End-to-End Speech Recognition. In Proc. Interspeech 2020 , pages 5026-5030, 2020.

Rui-Jie Zhu, Qihang Zhao, and Jason K Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint arXiv:2302.13939 , 2023.

Etienne Mueller, Viktor Studenyak, Daniel Auge, and Alois Knoll. Spiking transformer networks: A rate coded approach for processing sequential data. In 2021 7th International Conference on Systems and Informatics (ICSAI) , pages 1-5. IEEE, 2021.

Minglun Han, Qingyu Wang, Tielin Zhang, Yi Wang, Duzhen Zhang, and Bo Xu. Complex dynamic neurons improved spiking transformer network for efficient automatic speech recognition. Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI 2023) , 2023.

Shihao Zou, Yuxuan Mu, Xinxin Zuo, Sen Wang, and Li Cheng. Event-based human pose tracking by spiking spatiotemporal transformer. arXiv preprint arXiv:2303.09681 , 2023.

Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. In The Eleventh International Conference on Learning Representations , 2023.

Guangyao Chen, Peixi Peng, Guoqi Li, and Yonghong Tian. Training full spike neural networks via auxiliary accumulation pathway. arXiv preprint arXiv:2301.11929 , 2023.

Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Han Zhang, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual learning for transformer-based spiking neural network. arXiv preprint arXiv:2304.11954 , 2023.

A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning (ICML) , 2020.

Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys , 55(6):1-28, 2022.

Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems , 34:21056-21069, 2021.

Yifan Hu, Yujie Wu, Lei Deng, Man Yao, and Guoqi Li. Advancing residual learning towards powerful deep spiking neural networks. arXiv preprint arXiv:2112.08954 , 2021.

Man Yao, Guangshe Zhao, Hengyu Zhang, Hu Yifan, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , PP:1-18, 01 2023.

Guoqi Li, Lei Deng, Huajin Tang, Gang Pan, Yonghong Tian, Kaushik Roy, and Wolfgang Maass. Brain inspired computing: A systematic survey and future trends. 2023.

Eugene M Izhikevich. Simple model of spiking neurons. IEEE Transactions on Neural Networks , 14(6):1569-1572, 2003.

Tielin Zhang, Xiang Cheng, Shuncheng Jia, Mu-ming Poo, Yi Zeng, and Bo Xu. Selfbackpropagation of synaptic modifications elevates the efficiency of spiking and artificial neural networks. Science Advances , 7(43):eabh0146, 2021.

Yujie Wu, Rong Zhao, Jun Zhu, Feng Chen, Mingkun Xu, Guoqi Li, Sen Song, Lei Deng, Guanrui Wang, Hao Zheng, et al. Brain-inspired global-local learning incorporated with neuromorphic computing. Nature Communications , 13(1):65, 2022.

Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-learn in networks of spiking neurons. Advances in Neural Information Processing Systems , 31, 2018.

Arjun Rao, Philipp Plank, Andreas Wild, and Wolfgang Maass. A long short-term memory for ai applications in spike-based neuromorphic hardware. Nature Machine Intelligence , 4(5):467-479, 2022.

Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin, Zhaoxu Yang, and Guoqi Li. Temporal-wise attention spiking neural networks for event streams classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1022110230, 2021.

Man Yao, Jiakui Hu, Guangshe Zhao, Yaoyuan Wang, Ziyang Zhang, Bo Xu, and Guoqi Li. Inherent redundancy in spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16924-16934, 2023.

Man Yao, Hengyu Zhang, Guangshe Zhao, Xiyu Zhang, Dingheng Wang, Gang Cao, and Guoqi Li. Sparser spiking activity can be better: Feature refine-and-mask spiking neural network for event-based visual recognition. Neural Networks , 166:410-423, 2023.

Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in Neuroscience , 12:331, 2018.

Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine , 36(6):51-63, 2019.

Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11062-11070, 2021.

Lei Deng, Yujie Wu, Yifan Hu, Ling Liang, Guoqi Li, Xing Hu, Yufei Ding, Peng Li, and Yuan Xie. Comprehensive snn compression using admm optimization and activity regularization. IEEE Transactions on Neural Networks and Learning Systems , 2021.

Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Learning Representations , 2021.

Jibin Wu, Chenglin Xu, Xiao Han, Daquan Zhou, Malu Zhang, Haizhou Li, and Kay Chen Tan. Progressive tandem learning for pattern recognition with deep spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(11):7824-7840, 2021.

Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 1311-1318, 2019.

Johannes Schemmel, Daniel Brüderle, Andreas Grübl, Matthias Hock, Karlheinz Meier, and Sebastian Millner. A wafer-scale neuromorphic hardware system for large-scale neural modeling. In 2010 IEEE International Symposium on Circuits and Systems (ISCAS) , pages 1947-1950. IEEE, 2010.

Eustace Painkras, Luis A Plana, Jim Garside, Steve Temple, Francesco Galluppi, Cameron Patterson, David R Lester, Andrew D Brown, and Steve B Furber. Spinnaker: A 1-w 18-core system-on-chip for massively-parallel neural network simulation. IEEE Journal of Solid-state Circuits , 48(8):1943-1953, 2013.

Ben Varkey Benjamin, Peiran Gao, Emmett McQuinn, Swadesh Choudhary, Anand R Chandrasekaran, Jean-Marie Bussat, Rodrigo Alvarez-Icaza, John V Arthur, Paul A Merolla, and Kwabena Boahen. Neurogrid: A mixed-analog-digital multichip system for large-scale neural simulations. Proceedings of the IEEE , 102(5):699-716, 2014.

Juncheng Shen, De Ma, Zonghua Gu, Ming Zhang, Xiaolei Zhu, Xiaoqiang Xu, Qi Xu, Yangjing Shen, and Gang Pan. Darwin: A neuromorphic hardware co-processor based on spiking neural networks. Science China Information Sciences , 59(2):1-5, 2016.

Arindam Basu, Lei Deng, Charlotte Frenkel, and Xueyong Zhang. Spiking neural network integrated circuits: A review of trends and future directions. In 2022 IEEE Custom Integrated Circuits Conference (CICC) , pages 1-8, 2022.

Ole Richer, Ning Qiao, Qian Liu, and Sadique Sheik. Event-driven spiking convolutional neural network. WIPO Patent , page WO2020207982A1, 2020.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representa- tions (ICLR) , 2020.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10012-10022, 2021.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning , pages 10347-10357. PMLR, 2021.

Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems , 35:12934-12949, 2022.

Haokui Zhang, Wenze Hu, and Xiaoyu Wang. Parc-net: Position aware circular convolution with merits from convnets and transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI , pages 613-630. Springer, 2022.

Yongming Rao, Zuyan Liu, Wenliang Zhao, Jie Zhou, and Jiwen Lu. Dynamic spatial sparsification for efficient vision transformers and convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 558-567, 2021.

Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1225912269, 2021.

Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations , 2021.

Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in Neural Information Processing Systems , 32, 2019.

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient attention with many heads. In Computer Vision-ECCV 2022 Workshops: Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII , pages 35-49. Springer, 2023.

Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. In International Conference on Learning Representations , 2022.

Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3531-3539, 2021.

Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. In International Conference on Learning Representations , 2021.

Abdelrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Swiftformer: Efficient additive attention for transformer-based real-time mobile vision applications. arXiv preprint arXiv:2303.15446 , 2023.

A. L. Hodgkin and A. F. Huxley. A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of Physiology , 117(4):500-544, 1952.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision ECCV 2016 , pages 630-645, Cham, 2016. Springer International Publishing.

Xingting Yao, Fanrong Li, Zitao Mo, and Jian Cheng. GLIF: A unified gated leaky integrateand-fire neuron for spiking neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence , 3(10):905-913, 2021.

Yufei Guo, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Yinglei Wang, Xuhui Huang, and Zhe Ma. Im-loss: information maximization loss for spiking neural networks. Advances in Neural Information Processing Systems , 35:156-166, 2022.

Zhaodong Chen, Lei Deng, Bangyan Wang, Guoqi Li, and Yuan Xie. A comprehensive and modularized statistical framework for gradient norm equality in deep neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(1):13-31, 2022.

Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning , pages 5393-5402. PMLR, 2018.

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.00555 , 2022.

Meng-Hao Guo, Tian-Xing Xu, Jiang-Jiang Liu, Zheng-Ning Liu, Peng-Tao Jiang, Tai-Jiang Mu, Song-Hai Zhang, Ralph R Martin, Ming-Ming Cheng, and Shi-Min Hu. Attention mechanisms in computer vision: A survey. Computational Visual Media , 8(3):331-368, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 248-255, 2009.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs: an eventstream dataset for object classification. Frontiers in Neuroscience , 11:309, 2017.

Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, Jeff Kusnitz, Michael Debole, Steve Esser, Tobi Delbruck, Myron Flickner, and Dharmendra Modha. A low power, fully event-based gesture recognition system. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7243-7252, 2017.

Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 7132-7141, 2018.

Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 3-19, 2018.

Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations , 2020.

Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. In International Conference on Learning Representations , 2022.

Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems , pages 1-6, 2021.

Sayeed Shafayet Chowdhury, Nitin Rathi, and Kaushik Roy. One timestep is all you need: training spiking neural networks with ultra low latency. arXiv preprint arXiv:2110.05929 , 2021.

Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2661-2671, 2021.

Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS) , volume 34, pages 23426-23439, 2021.

Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Training high-performance low-latency spiking neural networks by differentiation on spike representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12444-12453, 2022.

Nitin Rathi and Kaushik Roy. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems , pages 1-9, 2021.

Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations , 2017.

Souvik Kundu, Massoud Pedram, and Peter A Beerel. Hire-snn: Harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5209-5218, 2021.

Priyadarshini Panda, Sai Aparna Aketi, and Kaushik Roy. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience , 14:653, 2020.

Mark Horowitz. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) , pages 10-14. IEEE, 2014.

Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J. Davison, Jörg Conradt, Kostas Daniilidis, and Davide Scaramuzza. Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(1):154-180, 2022.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision , pages 618-626, 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Spike-driven Transformer that incorporates the spike-driven paradigm of SNNs into the Transformer and is hardware-friendly for neuromorphic chips. The authors use SDSA to transform the multiplication operation between Query, Key, and Value into a combination of mask and sparse addition operations and modify the self-attention structure to achieve linear complexity. As a result, this approach can achieve up to 87.2× lower energy consumption compared to the traditional self-attention method. Furthermore, membrane shortcut is introduced to ensure all spiking neurons communicate via binary spikes. The paper provides insights into implementing each module on neuromorphic chips and achieves competitive results on multiple datasets while reducing energy consumption compared to previous works.\n\nSTRENGTHS:\nThe authors introduce the Spike-Driven Self-Attention (SDSA) module as a replacement for the self-attention module in the current spiking Transformer. The proposed module exhibits superior performance while significantly reducing energy consumption, resulting in an architecture that is better suited for deployment on neuromorphic chips.\n\nWEAKNESSES:\nThis paper explores the feasibility of implementing Conv, MLP, and Self-attention models driven by spikes on neuromorphic chips. However, it does not delve into the implementation details of membrane shortcuts. This approach involves the direct transmission of membrane potential signals between spiking neurons and may contradict the spike-driven approach discussed in the original text.\n\nThe effectiveness of a spikformer-based framework with only SDSA, without MS, remains unclear as most experiments in the literature combine both techniques. This weakens the argument for the performance of the spike-driven Transformer proposed in this paper. Although an ablation study was conducted on MS and SDSA in Table 5, it was limited to the CIFAR10/100 dataset and Spiking Transformer-2-512. The results suggest that the use of MS may be crucial to achieve state-of-the-art performance.\n\nQUESTIONS:\nHow is the membrane shortcut implemented on a neuromorphic chip?  Is it necessary to use it instead of SEW shortcut? The article explains that membrane shortcuts are used to optimize the distribution of membrane potential, but for spiking neurons, wouldn't it be more appropriate to perform such an optimization in the time domain?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposed a Spike-Driven Self-Attention (SDSA) module, which uses Hadamard product, column-wise summation, and spiking neuron layer to replace the matrix multiplication and softmax operation. Experiments on static and neuromorphic image classification demonstrate competitive performance and energy efficiency.

STRENGTHS:
1.. The authors proposed a novel form of linear self-attention module, SDSA, which leverages the features of the spikes and increases computational and memory efficiency.
2. The authors show extensive results from different models and different datasets. The performance of accuracy and energy efficiency is very strong.
3. SDSA reduces the computational complexity with a slight accuracy drop, and the authors show some attention maps to validate the effectiveness of the SDSA.

WEAKNESSES:
1. There is still a noticeable drop between the spike-driven transformer and the original transformer. Can the authors show some results of the model with more time steps and report the limitations of the model?
2. Can the authors show the performance and energy consumption as the number of time steps decreases?

QUESTIONS:
1. The ablation study shows the membrane shortcut brings significant accuracy improvement. I slightly doubt the importance of the SDSA. Can the authors try the combination of the membrane shortcut and some attention-free transformers like [1][2] to check if they can also achieve similar performance?


[1] J Lee-Thorp et al., “FNet: Mixing Tokens with Fourier Transforms”, NAACL 2022
[2] W Yu et al, “MetaFormer Is Actually What You Need for Vision”, CVPR 2022

LIMITATIONS:
The paper has few limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper proposes an improved spike transformer by replacing Spike-Element-Wise shortcut in an existing spike transformer (ref [20]) with Membrane Shortcut from spike ResNet (ref [26]) .

STRENGTHS:
Match SOTA ImageNet top-1 precision achieved by ResNet [26] with slightly lower estimated energy consumption.

WEAKNESSES:
Combining known techniques from two papers without modification is considered an incremental improvement.
It is not clear why Transformer is a better choice than ConvNet in the context of Spike Neural Network. In ViT papers, the motivation for the Transformer is its model capacity can scale easily to handle large datasets and it has much better parallelism on TPU/GPU than ConvNet. None of these has been indicated as the goal for SNN. As shown in table-2, the ResNet SNN model in ref[27] is equally competitive as the proposed model.
The notation is a bit confusing in a few equations. The s in Eq (5) is binary. This should be distinguished from floating-point numbers in Eq (4). But the same symbol R is used for both. The same observation holds for Eq (8)(10)(11).

QUESTIONS:
The outputs of LIF spiking neurons at all timesteps are used as the inputs to the Transformer. This may be the tested method in [20], but why is this a better idea than using the output at the last timestep? As shown in table-2, the power is directly proportional to the timestep count. So this seems a critical parameter to optimize over.

In Eq (12), S_L has the shape TxNxD. So outputs at all timesteps are used in classification head?

In Eq (6), the input s has no 2d spatial dimensions. What are the 2d dimensions in Conv2d()?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
The authors propose a variant of transformer networks with spiking neurons based on the LIF neuron. The submission reformulates the self-attention to use sparse addition and masking, and modifies the residual connections to transmit information in the domain of membrane potentials.
They achieve a state of the art result on Imagenet.
They present an energy analysis.

STRENGTHS:
It increases the amount of binary spike-based computations in the transformer.
The experimental results are strong.
A small ablation study is performed.

WEAKNESSES:
The explanation for  equations 15 and 16 can be improved
The explanation of power estimates can be improved.

QUESTIONS:
Can the authors provide an additional explanation for equations 15 and 16?

LIMITATIONS:
not much discussed
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07