SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Spike-driven in Self-attention.  $Q_{S}$ ,  $K_{S}$ ,  $V_{S}$  in spiking self-attention involve two matrix multiplications. One approach is to perform multiplication directly between  $Q_{S}$ ,  $K_{S}$ ,  $V_{S}$ , which is then converted to sparse addition, like spike-driven Conv and MLP. The previous work [20] did just that. We provide a new scheme that performs element-wise multiplication between  $Q_{S}$ ,  $K_{S}$ ,  $V_{S}$ . Since all elements in spike tensors are either 0 or 1, element multiplication is equivalent to a mask operation with no energy consumption. Mask operations can be implemented in neuromorphic chips through addressing algorithms [49] or AND logic operations [4].
Energy Consumption Comparison. The times of floating-point operations (FLOPs) is often used to estimate the computational burden in ANNs, where almost all FLOPs are MAC. Under the same
Table 1: Energy evaluation.  $FL_{Conv}$  and  $FL_{MLP}$  represent the FLOPs of the Conv and MLP models in the ANNs, respectively.  $R_C, R_M, \overline{R}, \widehat{R}$  denote the spike firing rates (the proportion of non-zero elements in the spike matrix) in various spike matrices. We give the strict definitions and calculation methods of these indicators in the Supplementary due to space constraints.
Vanilla Transformer [52]	Spike-driven Transformer (This work)
SPS	First Conv	EMAC·FLConv	EMAC·T·RC·FLConv
Other Conv	EMAC·FLConv	EAC·T·RC·FLConv
Self-attention	Q,K,V	EMAC·3ND²	EAC·T·R·3ND²
f(Q,K,V)	EMAC·2N²D	EAC·T·R·ND
Scale	EM·N²	-
Softmax	EMAC·2N²	-
Linear	EMAC·FLMLP0	EAC·T·RM0·FLMLP0
MLP	Layer 1	EMAC·FLMLP1	EAC·T·RM1·FLMLP1
Layer 2	EMAC·FLMLP2	EAC·T·RM2·FLMLP2
architecture, the energy cost of SNN can be estimated by combining the spike firing rate  $R$  and simulation timestep  $T$  if the FLOPs of ANN is known. Table 1 shows the energy consumption of Conv, self-attention, and MLP modules of the same scale in vanilla and our Spike-driven Transformer.
We evaluate our method on both static datasets ImageNet [77], CIFAR-10/100 [78], and neuromorphic datasets CIFAR10-DVS [79], DVS128 Gesture [80].
Experimental Setup on ImageNet. For the convenience of comparison, we generally continued the experimental setup in [20]. The input size is set to  $224 \times 224$ . The batch size is set to 128 or 256 during 310 training epochs with a cosine-decay learning rate whose initial value is 0.0005. The optimizer is Lamb. The image is divided into  $N = 196$  patches using the SPS module. Standard data augmentation techniques, like random augmentation, mixup, are also employed in training. Details of the training and experimental setup on ImageNet are given in the supplementary material.
Accuracy analysis on ImageNet. Our experimental results on ImageNet are given in Table 3. We first compare our model performance with the baseline spiking Transformer (i.e., SpikFormer [20]). The five network architectures consistent with those in SpikFormer are adopted by this work. We can see that under the same parameters, our accuracies are significantly better than the corresponding baseline models. For instance, the Spike-driven Transformer-8-384 is  $2.0\%$  higher than SpikFormer-8-384. It is worth noting that the Spike-driven Transformer-8-768 obtains  $76.32\%$  (input  $224 \times 224$ ) with  $66.34\mathrm{M}$ , which is  $1.5\%$  higher than the corresponding SpikFormer. We further expand the inference resolution to  $288 \times 288$ , obtaining  $77.1\%$ , which is the SOTA result of the SNN field on ImageNet.
We then compare our results with existing ResSNNs. Whether it is vanilla Res-SNN [39], MS-Res-SNN [26] or SEW-Res-SNN [25], the accuracy of Spike-driven Transformer-8-768 (77.1%) is the highest. Att-MS-Res-SNN [27] also achieves 77.1% accuracy by plugging an additional attention auxiliary module [81, 82] in MS-Res-SNN, but it destroys the spike-driven nature and requires more parameters (78.37M vs. 66.34M) and training time (1000epoch vs. 310epoch). Furthermore, the proposed Spike
Table 2: Spike Firing Rate (SFR) of Spike-driven Self-attention in 8-512. Average SFR is the mean of SFR over  $T = 4$ , and 8 SDSA blocks.
SDSA	Average SFR
QS	0.0091
KS	0.0090
g(QS, KS)	0.0713
VS	0.1649
Output of SDSA(·), V̂S	0.0209
driven Transformer outperforms by more than  $72\%$  at various network scales, while Res-SNNs have lower performance with a similar amount of parameters. For example, Spike-driven Transformer-6-512 (This work) vs. SEW-Res-SNN-34 vs. MS-Res-SNN-34: Param, 23.27M vs. 21.79M vs. 21.80M; Acc,  $74.11\%$  vs.  $67.04\%$  vs.  $69.15\%$ .
Table 3: Evaluation on ImageNet. Power is the average theoretical energy consumption when predicting an image from the test set. The power data in this work is evaluated according to Table 1, and data for other works were obtained from related papers. Spiking Transformer-  $L - D$  represents a model with  $L$  encoder blocks and  $D$  channels. *The input crops are enlarged to  ${288} \times  {288}$  in inference. The default inference input resolution for other models is  ${224} \times  {224}$  .
Methods	Architecture	Spike-driven	Param(M)	Power(mJ)	TimeStep	Acc
Hybrid training [83]	ResNet-34	✓	21.79	-	250	61.48
TET [84]	SEW-ResNet-34	X	21.79	-	4	68.00
Spiking ResNet [85]	ResNet-50	✓	25.56	70.93	350	72.75
tdBN [39]	Spiking-ResNet-34	✓	21.79	6.39	6	63.72
SEW ResNet [25]	SEW-ResNet-34	X	21.79	4.04	4	67.04
SEW-ResNet-50	X	25.56	4.89	4	67.78
SEW-ResNet-101	X	44.55	8.91	4	68.76
SEW-ResNet-152	X	60.19	12.89	4	69.26
MS ResNet [26]	MS-ResNet-18	✓	11.69	4.29	4	63.10
MS-ResNet-34	✓	21.80	5.11	4	69.42
MS-ResNet-104*	✓	77.28	10.19	4	76.02
Att MS ResNet [27]	Att-MS-ResNet-18	X	11.87	0.48	1	63.97
Att-MS-ResNet-34	X	22.12	0.57	1	69.15
Att-MS-ResNet-104*	X	78.37	7.30	4	77.08
ResNet	Res-CNN-104	X	77.28	54.21	1	76.87
Transformer	Transformer-8-512	X	29.68	41.77	1	80.80
Spikformer [20]	Spiking Transformer-8-384	X	16.81	7.73	4	70.24
Spiking Transformer-6-512	X	23.37	9.41	4	72.46
Spiking Transformer-8-512	X	29.68	11.57	4	73.38
Spiking Transformer-10-512	X	36.01	13.89	4	73.68
Spiking Transformer-8-768	X	66.34	21.47	4	74.81
Spike-drivenTransformer (Ours)	Spiking Transformer-8-384	✓	16.81	3.90	4	72.28
Spiking Transformer-6-512	✓	23.37	3.56	4	74.11
Spiking Transformer-8-512	✓	29.68	1.13	1	71.68
Spiking Transformer-8-512	✓	29.68	4.50	4	74.57
Spiking Transformer-10-512	✓	36.01	5.53	4	74.66
Spiking Transformer-8-768*	✓	66.34	6.09	4	77.07
Power analysis on ImageNet. Compared with prior works, the Spike-driven Transformer shines in energy cost (Table 3). We first make an intuitive comparison of energy consumption in the SNN field. Spike-driven Transformer-8-512 (This work) vs. SEW-Res-SNN-50 vs. MS-Res-SNN-34: Power,  $4.50\mathrm{mJ}$  vs.  $4.89\mathrm{mJ}$  vs.  $5.11\mathrm{mJ}$ ; Acc,  $74.57\%$  vs.  $67.78\%$  vs.  $69.42\%$ . That is, our model achieves  $+6.79\%$  and  $+5.15\%$  accuracy higher than previous SEW and MS Res-SNN backbones with lower energy consumption. What is more attractive is that the energy efficiency of the Spike-driven Transformer will be further expanded as the model scale grows because its computational complexity is linear in both token and channel dimensions. For instance, in an 8-layer network, as the channel dimension increases from 384 to 512 and 768, SpikFormer [20] has  $1.98 \times (7.73\mathrm{mJ} / 3.90\mathrm{mJ})$ ,  $2.57 \times (11.57\mathrm{mJ} / 4.50\mathrm{mJ})$ , and  $3.52 \times (21.47\mathrm{mJ} / 6.09\mathrm{mJ})$  higher energy consumption than our Spike-driven Transformer. At the same time, our task performance on these three network structures has improved by  $+2.0\%$ ,  $+1.2\%$ , and  $+1.5\%$ , respectively.
Then we compare the energy cost between Spikesdriven and ANN Transformer. Under the same structure, such as 8-512, the power required by the ANN-ViT  $(41.77\mathrm{mJ})$  is  $9.3\times$  that of the spikedriven counterpart  $(4.50\mathrm{mJ})$ . Further, the energy advantage will extend to  $36.7\times$  if we set  $T = 1$  in the Spike-driven version  $(1.13\mathrm{mJ})$ . Although the accuracy of  $T = 1$  (here we are direct training) will be lower than  $T = 4$ , it can be compensated
Table 4: Energy Consumption of Self-attention.  $E_{1}$  and  $E_{2}$  (including energy consumption to generate  $Q,K,V$ ) represent the power of self-attention mechanism in ANN and spike-driven.
Models	E1(pJ)	E2(pJ)	E1/E2
8-384	6.7e8	1.6e7	42.6
8-512	1.2e9	2.1e7	57.2
8-768	2.7e9	3.1e7	87.2
Table 5: Experimental Results on CIFAR10/100, DVS128 Gesture and CIFAR10-DVS.
Methods	Spike-driven	CIFAR10-DVS	DVS128 Gesture	CIFAR-10	CIFAR-100
T	Acc	T	Acc	T	Acc	T	Acc
tdBN [39]	X	10	67.8	40	96.9	6	93.2	-	-
PLIF [87]	✓	20	74.8	20	97.6	8	93.5	-	-
Dspike [88]	X	10	75.4	-	-	6	94.3	6	74.2
DSR [89]	✓	10	77.3	-	-	20	95.4	20	78.5
Spikformer [20]	X	16	80.9	16	98.3	4	95.5	4	78.2
DIET-SNN[90]	X	-	-	-	-	5	92.7	5	69.7
ANN(ResNet19)	X	-	-	-	-	1	94.97	1	75.4
ANN(Transformer4-384)	X	-	-	-	-	1	96.7	1	81.0
This Work	✓	16	80.0	16	99.3	4	95.6	4	78.4
by special training methods [86] in future work.
Sparse spike firing is the key for Spike-driven Transformer to achieve high energy efficiency. As shown in Table 2, the Spike Firing Rate (SFR) of the self-attention part is very low, where the SFR of  $Q_{S}$  and  $Q_{K}$  are both less than 0.01. Since the mask (Hadamard product) operation does not consume energy, the number of additions required by the  $\mathrm{SUM}_{\mathrm{c}}(Q_S \otimes K_S)$  is less than 0.02ND times. The operation between the vector output by  $g(Q_S, K_S)$  and  $V_{S}$  is still a column mask that does not consume energy. Consequently, in the whole self-attention part, the energy consumption of spike-driven self-attention can be lower than  $87.2 \times$  of ANN self-attention (see Table 4).
Experimental results on CIFAR-10/100, CIFAR10-DVS, and DVS128 Gesture are conducted in Table 5. These four datasets are relatively small compared to ImageNet. CIFAR10/100 are static image classification datasets. Gesture and CIFAR10-DVS are neuromorphic action classification datasets, which need to convert the event stream into frame sequences
Table 6: Studies on Spiking Transformer-2-512.
Model	CIFAR-10	CIFAR-100
Baseline [20]	93.12	73.17
+ SDSA	93.09 (-0.03)	72.83 (-0.34)
+ MS	93.93 (+0.81)	74.63 (+1.46)
This work	93.82 (+0.73)	74.41 (+1.24)
before processing. DVS128 Gesture is a gesture recognition dataset. CIFAR10-DVS is a neuromorphic dataset converted from CIFAR-10 by shifting image samples to be captured by the DVS camera. We basically keep the experimental setup in [20], including the network structure, training settings, etc., and details are given in the supplementary material. As shown in Table 5, we achieve SOTA results on Gesture (99.3%) and CIFAR-10 (95.6%), and comparable results to SOTA on other datasets.
Ablation study. To implement the spike-driven paradigm in Transformer, we design a new SDSA module and reposition the residual connections in the entire network based on the SpikFormer [20]. We organize ablation studies on CIFAR10/100 to analyze their impact. Results are given in Table 5. We adopt spiking Transformer-2-512 as the baseline structure. It can be observed that SDSA incurs a slight performance
loss. As discussed in Section 3.3, SDSA actually masks some unimportant channels directly. In Fig. 3, we plot the attention maps (the detailed drawing method is given in the supplementary material), and we can observe: i) SDSA can optimize intermediate features, such as masking background information; ii) SDSA greatly reduces the spike firing rate of  $\hat{V}_S$ , thereby reducing energy cost. On the other hand, the membrane shortcut leads to significant accuracy improvements, consistent
Figure 3: Attention Map Based on Spike Firing Rate (SFR).  $V_{S}$  is the Value tensor.  $\hat{V}_{S}$  is the output of SDSA(·). The spike-driven self-attention mechanism masks unimportant channels in  $V_{S}$  to obtain  $\hat{V}_{S}$ . Each pixel on  $V_{S}$  and  $\hat{V}_{S}$  represents the SFR at a patch. The spatial resolution of each attention map is  $14 \times 14$  (196 patches). The redder the higher the SFR, the bluer the smaller the SFR.
with the experience of Conv-based MS-SNN [26, 25]. Comprehensively, the proposed Spike-driven Transformer simultaneously achieves better accuracy and lower energy consumption (Table 3).
We propose a Spike-driven Transformer that combines the low power of SNN and the excellent accuracy of the Transformer. There is only sparse addition in the proposed Spike-driven Transformer. To this end, we design a novel Spike-Driven Self-Attention (SDSA) module and rearrange the location of residual connections throughout the network. The complex and energy-intensive matrix multiplication, softmax, and scale in the vanilla self-attention are dropped. Instead, we employ mask, addition, and spike neuron layer to realize the function of the self-attention mechanism. Moreover, SDSA has linear complexity with both token and channel dimensions. Extensive experiments are conducted on static image and neuromorphic datasets, verifying the effectiveness and efficiency of the proposed method. We hope our investigations pave the way for further research on Transformer-based SNNs and inspire the design of next-generation neuromorphic chips.
This work was supported by Beijing Natural Science Foundation for Distinguished Young Scholars (JQ21015), National Science Foundation for Distinguished Young Scholars (62325603), and National Natural Science Foundation of China (62236009, U22A20103).
[1] Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural Networks, 10(9):1659-1671, 1997.
[2] Paul A Merolla, John V Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science, 345(6197):668-673, 2014.
[3] Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82-99, 2018.
[4] Jing Pei, Lei Deng, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):106-111, 2019.
[5] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607-617, 2019.
[6] Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence, 3(10):905-913, 2021.
[7] Catherine D Schuman, Shruti R Kulkarni, Maryam Parsa, J Parker Mitchell, Bill Kay, et al. Opportunities for neuromorphic computing algorithms and applications. Nature Computational Science, 2(1):10-19, 2022.
[8] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008, 2017.
[9] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE transactions on Pattern Analysis and Machine Intelligence, 45(1):87-110, 2022.
[10] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1-41, 2022.
[92] Souvik Kundu, Massoud Pedram, and Peter A Beerel. Hire-snn: Harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5209-5218, 2021.
[93] Priyadarshini Panda, Sai Aparna Aketi, and Kaushik Roy. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience, 14:653, 2020.
[94] Mark Horowitz. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pages 10-14. IEEE, 2014.
[95] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J. Davison, Jörg Conradt, Kostas Daniilidis, and Davide Scaramuzza. Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):154–180, 2022.
[96] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618-626, 2017.