Spiking Token Mixer:  An event-driven friendly Former structure for spiking neural networks
Authors: Shikuang Deng
Yuhang Wu
Kangrui Du
Shi Gu
Venue: NeurIPS 2024 poster (2024)
Keywords: spiking neural network; event-driven friendly; low energy consumption
Primary area: deep_learning_architectures
TL;DR: We proposed the STMixer architecture, consisting exclusively of convolutional, fully connected layers, and residual path, is more advantageous for deployment in asynchronous scenarios.

Abstract: Spiking neural networks (SNNs), inspired by biological processes, use spike signals for inter-layer communication, presenting an energy-efficient alternative to traditional neural networks. To realize the theoretical advantages of SNNs in energy efficiency, it is essential to deploy them onto neuromorphic chips. On clock-driven synchronous chips, employing shorter time steps can enhance energy efficiency but reduce SNN performance. Compared to the clock-driven synchronous chip, the event-driven asynchronous chip achieves much lower energy consumption but only supports some specific network operations. Recently, a series of SNN projects have achieved tremendous success, significantly improving the SNN's performance. However, event-driven asynchronous chips do not support some of the proposed structures, making it impossible to integrate these SNNs into asynchronous hardware. In response to these problems, we propose the Spiking Token Mixer (STMixer) architecture, which consists exclusively of operations supported by asynchronous scenarios, including convolutional, fully connected layers and residual paths. Our series of experiments also demonstrates that STMixer achieves performance on par with spiking transformers in synchronous scenarios with very low timesteps. This indicates its ability to achieve the same level of performance with lower power consumption in synchronous scenarios. The codes are available at \url{https://github.com/brain-intelligence-lab/STMixer_demo}.

Spiking Token Mixer: An Event-Driven Friendly Former Structure for Spiking Neural Networks
Shikuang Deng" Yuhang Wu' Kangrui Du Shi Gu 2,3
1 University of Electronic Science and Technology of China College of Computing. Georgia Institute of Technology 3 Shenzhen Institute for Advanced Study, UESTC dengsk@uestc. edu. en, gus@uestc. edu. on
Abstract
Spiking neural networks (SNNs), inspired by biological processes, use spike signals for inter-layer communication, presenting an energy-efficient alternative traditional neural networks. To realize theoretical advantages of SNNs energy efficiency, it is essential to deploy them onto neuromorphic chips. On clock-driven synchronous chips, employing shorter time steps can enhance energy efficiency but reduce SNN performance. Compared to the clock-driven synchronous chip, the event-driven asynchronous chip achieves much lower energy consumption but only supports some specific network operations. Recently, series of SNN projects have achieved tremendous success, significantly improving the SNN's performance. However, event-driven asynchronous chips do not support the proposed structures, making it impossible to integrate these SNNs into asynchronous hardware. In response these problems, we propose the Spiking Token Mixer (STMixer) architecture, which consists exclusively of operations supported by asynchronous scenarios, including convolutional, fully connected layers and residual paths. Our series of experiments also demonstrates that STMixer achieves performance on par with spiking transformers synchronous scenarios with very low timesteps. This indicates its ability to achieve the same level of performance with lower power consumption in synchronous scenarios. The codes are available at https: //g thub. com/brat bwt//brainn--eee demo.
-aaicceeeaaa//aaaaee Jab/STMixer.
Introduction
Spiking neural networks (SNNs) are bio-inspired neural networks that use spike signals carrier for inter-layer transmission. Because of the spiking mechanism, SNNs have the potential to turn high-precision multiplication into addition, which makes them better energy-save -aving infrastructure [Maass, 1997, Roy et al., 2019]. However, such an advantage in efficiency is not guaranteed for typical hardware [Kim al., 2019] but for neuromorphic chips [Akopyan al., 2015, Davies et al., 2018b, Pei et al., 2019]. The computational energy of the synchronous neuromorphic hardware is affected by the amount of calculation, the spike frequency, and the SNN time steps [Rathi and Roy, 2020, Li et al., 2021]. And synchronous hardware provides support for nearly all matrix operations, even allowing for the execution of mixed networks that combine the ANN and SNN. In order to achieve low power consumption on synchronous hardware, SNNs must consider using
Equal contribution
ES Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

shorter time steps. But too few time steps may potentially undermine the performance of SNNs. On the other hand, in theory, event-driven asynchronous hardware serves as an ideal implementation platform for such mechanism. On the event-driven hardware, each neuron is configured as an independent entity driven by events rather than the high-frequency clock. Compared to synchronous scenarios, asynchronous scenarios impose more stringent constraints on network architecture. Hence, in asynchronous neuromorphic hardware, SNN usually utilizes the soft-reset Integrate-and-Fae (IF) model as the fundamental neuron model. The supported network operations in asynchronous scenarios adhere to the rule that matrix multiplication should take place between a spike matrix and a fixed float weight matrix. Consequently, due to the accumulation of membrane potential in the IF neuron, spike arrival timing errors do not significantly impact subsequent layers.
Over the past years, there have been lot of efforts dedicated to advancing the performance of SNNs across various datasets. These efforts have explored multitude of perspectives, aiming to optimize the training pipeline [Neftci et al., 2019, Wu et al., 201 8, Li et al., 2021, Deng et al., 2023], refine the optimization objectives [Deng et al., 2021, Guo et al., 2022b], and design the network structure [Fang et al., 2021a, Zhou et al., 2023b.a]. Recently, Spikformer [Zhou et al., 2023b,a] has achieved remarkable results by incorporating the Transformer into SNN, resulting in unprecedented performance gains across wide range of datasets. Spikformer has designed new attention module, SSA, which aligns Spikformer with the forward rules of SNN during the inference phase. Such progression in Spikformer-lit design theoretically eliminates the need for high-precishe matrix multiplication. Spikformer-like models have indeed demonstrated significant performance improvements in synchronous hardware or GPU simulation. However, an easily overlooked point is their feasibility in implementing asynchronous hardware. In event-driven scenarios, the spike arrival times may not be as precise as in synchronous scenarios with hardware clocks. Calculating two spike matrix multiplications or spike matrix passing through the max-pooling layer could result in significant differences in the output.
To address these issues, we proposed a new architecture called Spiking Token Mixer (STMixer), which only uses asynchronous neuromorphic comppaabe operations, including convolutional, fully connected layers and residual paths. STMixer still adopts the holistic framework of spikformer, which consists of spiking patch splitting (SPS), encoders, and classification heads (Fig. 1 (A)). The encoder module in STMixer consists of two components: the token mixing module and the channel module. In the token mixer component of STMixer, the value matrix undergoes matrix multiplication with trainable attention map weight matrix (Fig. 2 (A)). This new architecture eliminates the requirement of performing matrix multiplication between two spike matrices during the forward pass, thereby conforming to the event-driven situation. We also analyzed and enhanced the SPS module, ensuring its compliance with the requirements of an asynchronous environment while reducing information loss. Furthermore, we employed the proposed mixer architecture to optimize the components of the surrogate module [Deng et al., 2023] and used the surrogate module learning to assist the SNN training. The following summarizes our main contributions:
We examine problems with the SSA module in Spikformer in asynchronous scenarios and suggest new module. the Spiking Token Mixing (STM) module, which consists solely of network components that cater to asynchronous environments.
We investigate the SPS module's 's asynchronous support and the information loss problem. To address these concerns, we proposed the information protection spiking patch splitting (IPSPS) module, which removes the maxpool layer and adds a residual path to protect information.
Through the network architecture search, we verified that the STMixer structure excels among all possible configurations in terms of performance and parameter quantity. Addition- ally, we have validated that STMixer achieves performance on par with or even surpasses existing Spikformer-lic like works in synchronous scenarios by using only single time step across almost all datasets. This suggests that STMixer has the potential to deliver high performance with lower power consumption in synchronous scenarios, too.

---

A
Encoder Block X
Spiking Patch Splitting (SPS) x4
Token Mix
Channel Mix
Canda
Vantee
Call
Qui Ted
B
Map
T-1
T+2
Figure I: The architecture of Spikformer-like networks and the issue of spiking matrix multiplication and max pooling layers (A) The overall architecture of STMixer and Spikformer. (B) The issue of the Max Pooling layer when there is a spike delay asynchronous hardware. (C) The spike delay issue of spiking matrix multiplication. When both matrices involved in matrix multiplication are composed of spikes, the imprecise timing of spike arrival can result in inaccuracies in the computed product.
Related Work
2.1 Transformer-like architectures in artificial neural networks
Convolutional networks, such as ResNet and VGG, have historically been the most popular archi- tectures in the realm of computer vision [Zheng et al., 2021, Rathi and Roy, 2020]. Some works have tried to add attention mechanisms to visual tasks (Yao et al., 2023, 7hu et al., 2024] after being inspired by the amazing results of transformers. in NL.P [Vaswani et al., 2017, Devlin et al., 2019, Floridi and Chiriatti, 2020]. Especially, the Vision Transformer (VIT) [Dosovitskiy al., 2020] has demonstrated that transformer architectures can achieve remarkable performance in computer vision tasks. VIT uses patch embedding as an input and multiple stacked transformer encoders as the backbone. Since then, a lot of research has been done to improve performance and efficiency by making changes to how images are stored [Parmar et al., 2018, Chen et al., 2020, Liu et al., 2021], how attention is calculated [Shen et al., 2021, Bolya et al., 2022], and other methods ['Tourron et al., 2021, Zhang and Sabuncu, 2020]. Meanwhile, some works have revealed that just employing MLPs the token mixer also lead good performance| anrformmnceeTTlssiihhin Tolstikhin al., 2021, Touvron al., 2022, Guo et al., 2022a]. Recently, a fresh perspective has been proposed by Metaformeri Yu et al., 2022], suggesting that the success of Transformer in visual tasks stems from the holistic encoder architecture, where token mixer and channel mixer interact. Interestingly, Metaformer even achieved competitive performance by employing a simple pooling layer the token mixer. Their findings offer us novel insights into how to modify Spikformer.
2.2 Spiking neural networks
The forward process of SNN encompasses the following rule: all high-precision matrix multiplications should be capable of degrading to matrix additions, eliminating the need for high-precision matrix multiplication operations after implanting hardware [DeBole al., 2019, Davies et al., 2018b) 20]8b.. However, this rule restricts the flexibility of SNNs to incorporate various operations, limiting them to few substructures such the convolution layer, the fully connected layer, the pooling layer, and so on. The majority of SNN architectural designs are indeed based on these substructures, such as the advancements of ResNet structures like SEW-ResNet [Fang et al., 2021a] and MS-ResNet [Hu et al., 2024], as well the structure search efforts like AutoSNN [Na et al., 2022] and SNASNet [Kim al., 2022]. Recently, Spikformer [Zhou et al., 2023b] has achieved remarkable success by integrating the transformer architecture into SNNs, attaining SOTA performance across various datasets. The authors proposed SSA module to ingeniously realize attention computation in SNN while adhering to the aforementioned forward rule. At present, Spikformer and its subsequent series [Zhou al., 2023a, Yao et al., 2024] of improvements are steadily pushing the boundaries of SNN performance. Nevertheless, though SSA exhibits notable performance in clock-driven hardware,
3

---

may encounter significant discrepancies in event-driven hardware due to the fact that the spikes may not arrive the same time.
3 Preliminaries
3.1 Spiking neural model
The spike neuron serves as the fundamental unit of SNN, wherein their somatics harbor membrane potential that integrates information from the synaptic according to prescribed model principles. The leaky integrate-and-fire (LIF) model [Lapieque, 1907] is currently the most prevalent SNN neuron model, which describes the membrane potential update rule as follows:
H[ = 1 V - 1] X 7
(I)
S|G] = θ (M - Vth ), vld HT(1(1 S(( Vreset S(1),
where [[t] is the membrane potential, r is the membrane time constant, and S(t) is the output in time t, it would be equal or 0, which means if there is spike or not. ( means the Heaviside step function, which equals I when H [t] exceeds the threshold value Vin Furthermore, we employ the hard reset mechanism, which means the membrane potential will reset to the reset potential when the neuron fires spike.
3.2 Spikformer
We use the Spikingformer [Zhou et al., 2023a] architecture as the foundational framework of our network. Spikingformer addresses the issue of residual connections in Spikformer by using membrane potential residual instead of spiking residual. The overall structure of Spikingformer, as depicted in Fig. (A), comprises three main components: Spiking Patch Splitting (SPS), encoder module, and a classification head. The SPS consists of five convolutional layers, where the first four layers are followed by a max pooling layer. The function of SPS is to perform convolution processing and downsampling on input images, resulting in an output matrix with the shape D X w. Eventually, this matrix is flattened into N (N h x w) patches, each with D-dimensional vector. The classification head consists of an adaptive average pooling layer, which is used for downsampling, and fully connected layer, responsible for obtaining the network's final output. The classification head will introduce floating-pon Because the calculation amount of the classification head is very small, it will not cause much energy overhead or inference delay but will greatly improve SNN performance [Zhou et al., 2023a].
matrix multiplication, so it needs to be deployed on the CPU.
The most important component of Spikformer is its encoder module, which consists of two cells and residual connections (Fig. (A)). Cell corresponds to the Feed Forward Network (FFN) in VIT. It consists of two fully connected layers designed for channel mixing. Cell refers to the Spiking Self-Attention (SSA) module proposed by the Spikformer. SSA has been demonstrated to be more suitable for SNN compared to vanilla self-attention (VSA). In SSA, when the input is X E I Nx the query (Q E R Nx),) P), key (K E ND),, and value (V E IR NxDD) are computed using the following equations:
= SNQ QSSNq (BN (XW2)) SNK (XWx)) V - SNV (BN (XWv))
(2)
where Wo, Kr Wy E RDD are learnable weight matrices, SN means spiking neural (LIF model) activation, and BN denotes the batch normalization layer. The authors of Spikformer argue that in ANN, the presence outliers or negative values in the query and key makes the attention map require softmax function. However, since the output of eqn. is binary matrix that only consists of 0 and 1, there are no outliers or negative values. Therefore, they directly perform matrix multiplication on the three matrices to obtain the output of SSA:
SSA K, V) = SN (QKTV** *
(3)
where 0.125 is constant factor.

---

4 Methodology
4.1 Event-driven Scenarios
In ideal asynchronous scenarios, each neuron is an autonomous entity solely focused on updating its membrane potential upon receiving spike signals, then firing spike once the membrane potential exceeds the threshold. Since the model in asynchronous scenarios lacks the specific hardware clock to control the spike firing and arriving time, it is inevitable to encounter temporal errors in the spike arrival time. In the extreme case, it is conceivable to assume that each neuron fires spikes at an individual time (the SNN time step T approaches infinity, and each spike fires at individual time step). When there are errors in the arrival time of spikes (1 = + σ), the output sum of asynchronous scenarios supported operations like convolution, fully connected layer, and average pooling will not change. However, some operations that are commonly observed in synchronous scenarios may have output errors due to imput spike arrival timing error.
Maxpool layer. As shown in Fig. (B), a spike specific location arrives one time step later in the max pooling layer, resulting in a noticeable change in the output. This output error will further affect the result of the subsequent convolution layer, as well as the membrane potential accumulation of neurons and their output spikes. In the extreme case where each spike has an independent arrival time, all the spikes will pass through the max pooling layer. Therefore, the max pooling layer needs a clock to control the spike arrival time and cannot be simply deployed in event-driven scenarios. To address this problem, we can simply remove the max pooling layer and change the stride the convolutional layer following the max pooling layer to 2 for downsampling. After this modification, when the arrival spikes have temporal discrepancies, the cumulative membrane potential of the post-convolution spike neurons will remain relatively stable.
Spike matrix multiplication. On the synchronization and driven hardware, the query, key, and value spike matrices can arrive on time. This guarantees that the forward SSA results on this hardware correspond to the GPU simulation results. However, on event-driven hardware, the absence of clock constraints often makes it challenging for spikes to arrive at precise timings. If there is any spike loss or similar occurrence during matrix multiplication SSA, a significant error in the SSA output will occur. Here is a simple example that shows how the computed attention map matrix changes significantly SSA when there a spike delay at certain point in both the query and key matrices (Fig. 1 (C)). This example shows how hard it is to make sure that the forward process with two spike matrix multiplications runs smoothly on event-driven neuromorphic hardware.
4.2 Clock-driven Scenarios
According Rathi and Roy [2020], Hu et al. [2024], Zhou et al. [2023a], the theoretical computational energy consumption of SÑNs on clock-driven scenarios can be defined as:
ESNN EAC X
SOPc + EMAC x (FLOP + FLOP FLOPL
Lineer);
(4)
where EAC 0.9pJ and EMAC = 4.6p/ are the assumed energy consumption of accumulate and multply-and--ccumulle
operations, respectively, of 45mm CMOS hardware [Horowitz, 2014], 4],, SOP denotes the number of spike-based accumulate operations, and F FLOP! Conr means the computation of the first layer that encodes static RGB images into spike matrix [Zhou et al., 2023a], FLOP means the computation of the classification head. And SOP = FLOP! frr". i' affected by the SNN fire rate fr' and time step T. However, recent works [Dampfhoffer et al., 2022, Bhattacharjee et al., 2024] have pointed out that Eqn. 4.2 neglects the overhead of memory scheduling Essem. which is much larger than EMAA-
On each time step of SNN forward inference, hardware will deal with memory scheduling, so EMAC also affected by the time step T. And SNNs with a large number of time steps may have large energy overhead in synchronous scenarios, even potentially surpassing that of ANNs under the same structure. Therefore, we must closely examine the performance of the new SNN architecture at extremely low time steps to ensure its adaptability to low-power scenarios.
5

---

Spiking Token Mixing (STA))
Call Toker Mix
Information Spiking Patch Splitting DPSPS)
Attention Weight
Camall
Value
Lino IN
Creadl
Figure 2: The architecture of (A) Spiking token mixing (STM) module and (B) Information protect
Spiking Patch Splitting (IPSPS) module.
4.3 STMixer
To address the aforementioned requirements in the two scenarios, we present the STMixer network architecture. STMixer solely consists of asynchronously supported operations and performs well at extremely low time steps (T = 1). STMixer has the same overall structure as Spikformer but uses STM the token mixing module and IPSPS replace the SPS module (Fig. (A)). STMixer controlled by three syper-parameters: L. D, and H. which respectively control the number of encoders, the encoding dimension, and the number of multi-heads of STM. We use the naming convention "STMixer-L-D-HH" to represent the three parameters, which is similar to Spikformer-like works. Next, will go into detail about the important modules in STMixer.
4.3.1 Spiking Token Mixing (STM)
As shown in 2 (A), we use the STM module to realize the token mixing function. This module utilizes a weight matrix (WSTM E NRNNN to mix the token dimension information. Mathematically, its forward process can be expressed as follows:
STM(X) (WTTMV) = WOWWSTSSN(B((XWW)
(5)
Where X E NxD the input spiking matrix of STM, V E RNDD is the value matrix, and Wy DDD is the projection matrix acquire the value matrix V. When we simplify the operations of the SSA module by removing the activations for query and key, the formula for SSA can be simplified follows:
SSA (,Q,KKVV) V) > WN(XXWWXX. *
(6)
= -XNXXXX'V 8).
After this simplification, Wi and WK can converge into a single matrix, and the attention map is reduced to X"" So the objective of WSTM is to gradually fit the attention map during the training process. And some studies also suggest that linear token mixing can capture external attention between samples in the dataset [Guo et al., 2022a].
Furthermore, we evenly divide the value matrix V E IR NxD into H groups based on dimension D, allocating a weight matrix Whh E N to each group. This method employs different attention weights for various patch encoding groups Vh E NxD/H It facilitates a finer granularity in token mixing, leading to enhanced information mixing and integration among tokens. In addition, due to the relatively modest number of tokens in the vision task, implementing multi-head in this manner would not significantly increase the parameters and maintain the same level of computational complexity.
4.3.2 Information Loss in Spiking Patch Splitting
In a neural network, as the input X undergoes successive forward propagation layers, the inherent information gradually dissipates. Mathematically this phenomenon can be described
x(x,)) 2 ((X)((X)) 2 1(X,
I(X,Y),
where / means the mutual information, f and y are the operations of network layers, and Y is the network output. An SNN layer may lose more information than an ANN layer because the layer output is a spike matrix, which inherently contains significantly less information than floating-poin

---

Input Image
SPS
T=1
SPS
T T=1 IPSPS(UF) To4
Figure 3: Visualization results of random initial weight output feature maps of the SPS module. We present the feature maps of the 100th channel and the average of all channels, and in SNN, the feature map is the mean feature map across all time steps. When using the LIF neuron model the activation function for SPS, it obvious that there is significant information loss in the output feature map compared to the input image. Employing the IPSPS can alleviate this information loss.
matrix. The Spiking Patch Splitting (SPS) module comprises five layers of convolutional operations and spiking neural activation. After passing through total of five convolutional layers and spiking neural, the feature map may have already lost significant amount of information contained in the input images (Fig. 3).
Since Spikformer-like SNNs rely on convolutional layers in the SPS module for information extraction, reducing the number of layers in the SPS module will compromise the SNN's ultimate performance. To address this information loss problem, we add an additional path that directly uses a convolutional layer for input image encoding (Fig. 2 (B)). In our configuration, the encoding outputs from the auxiliary pathway account for 1/8 of the entire SPS output feature map. This new encoding module called information protected spiking patch splitting (IPSPS).
There are two advantages IPSPS: (1) IPSPS does not reduce the convolutional layers in the original pathway; it only decreases the number of feature map channels by 1/8 for the last convolutional layer. Theoretically, this approach should not significantly impair the SNN performance. Simultancously, the new pathway enriching the SPS output's feature representation. (2) The output of the additional pathway in IPSPS be regarded as linear transformation applied to the input image. The input feature map for the first encoder from the additional pathway essentially the linearly transformed input image passed through SNN neural layer, which maximally preserves the input image's information.
provide feature maps with varying convolutional depths for subsequent encoders,
4.4 STM on surrogate module learning
Even if we reduce the information loss of the encoder's input, the accumulation of gradient errors caused by surrogate gradient may still prevent SNN from achieving better performance with low time steps. As result, we use the surrogate module learning (SML) method [Deng et al., 2023]. The SML an exceptionally effective approach proposed to alleviate the issue of gradient errors in SNN by adding auxiliary pathway during the training phase. However, the use of large kernel convolutions results in significant computational overhead during the training phase. this paper, we employ the STMixer encoder replace the computationally intensive convolutional encoder, thereby greatly reducing the computational overhead of SML. The STM the surrogate module uses the GeLU activation function and employs the same number of heads as the STM in the main structure. Finally, we use classification head for the output. Starting from index O, we place a surrogate module after every two encoders for surrogate module learning. And the number of encoders in the surrogate module is half of the count of encoders after the place point in the main pathway.
5 Experiments
5.1 Comparison to exiting works
CIFAR. In IPSPS, we only set the stride to for the last two convolutional layers, similar to Spikformer. This implies that IPSPS downsamples the input images to resolution of 8 8 and
7

---

subsequently obtains 64 patches through flattening, Here, STM in STMixer uses 32 heads. And we use the same training script as Spikeformer, the learning rate set to 0.005, the batch size 128. and training total 300 epochs. As shown in Tab. 1, STMixer achieves state-of-the-art performance both on the CIFAR-100 100 dataset and considering models with comparable parameter numbers and flops. However, on the CIFAR-10 10 dataset, the SML method slightly outperforms the results reported by us. This could be attributed to the larger parameter count the structure they used or the structure being more suitable for the CIFAR-10 10 dataset. These results demonstrate the potential of STMixer to achieve excellent performance for SNN on simple static datasets, even with only one time step. also suggests that STMixer serve competitive alternative to Spikformer- works.
ImageNet. The IPSPS module of STMixer downsamples the input images of x24 224 resolution to 16x 16 through four successive convolutional layers, ultimately obtaining 196 patches. We employ the same training script as Spikeformer, set the batch size to 256, and set the training total epochs to 300. All the STMMxxer use 16 heads for STM in ImageNet experiments. We set the time step to 1, aiming to verify its performance under low energy consumption. The energy consumption of synchronous hardware estimation is computed by Eqn. 4.2. As shown in Tab. 1, we even use much lower time step (much lower energy consumption), our STMixer-8- 8-8---8-----1-- performs better than Spikeformer and achieves SOTA accuracy on ImageNet. This observation highlights the substantial potential of STMixer in synchronous and low energy scenarios.
Table 1: Performance comparison of our method with existing methods on CIFAR10/100 and ImageNet. Param means the number of parameters.
Datasel
Methods
Architecture
Faram (M)
Time Steps
Energy (mJ)
Accuracy (%)
IBN [2heng ot 2021]
19
12.57
92.92
TET Deng at. 2021]
ResNet-19
12,57
04.44:0007
AutoSNN (Na
2023]
AutoSNN (C-128)
93.15
SNASNa [Kim et 2023]
SNASNcIBWW
94, 12:1222022
CIFAR-10
SpikeDHS [Che
2022]
SpikeD
14
95, 00.36-0001
SML [Deng et al., 2023]
2023]
ResNet-18
11.22
96.04::0
Spikformer [Zhoa et al., 2023b]
Spikformer-45 4-384
9.32
95.19
Spikingfomeer [Zhou
2023g]
Spikingformer---35
9.32
95.61
95.49:0013
ours
STMixa-4-384-32
8.29
96.01
IdBN LdBNTZheng 2021
ResNc-19
12.57
70.36
TET al, 2021]
ResNet-19
12.57
74.47=0022
AutSNNN [Na al., 2022]
2022]
AutoSNN (C=128)
21
69.16
SNASNer (Kim et 2022]
SNWSNet-BW
73.04:0033
CIFAR-100 100
SpikeDHS [Che al., 2022]
SpikeDISS (n3si)
76.25
SML [Deng et al., 2023]
ResNet- 18
11.22
79.49=0.11
Spikformer (Zhoa et al., 2023b]
Spikformer mikkformer-4-38e 4-384
9.32
77,86
Spikingtomeer
Spikingformer-44 384
ours
STMina-4-386-33
8.29
81. $1.87=0.11 16
IBBN LZhnng 2021]
ResNcE31
21.79
63.72
ResNet [Fang et 2022]
SEW.ResNec14
TET [Deng et al,, 2021]
NeWeRe
21.79
68.00
SpikeDHS [Che
2022]
SpikeDHS-CLA-large
58M
68.64
SML [Deng et 2023]
ResNet-34
21.79
68. 25
Spitoorme---3112
23.37
9.41
72.46
ImageNet
Spikformer [Zhou al,, 2023b]
Spikforme--8-
29.68
11.57
73.38
Spikformer-
Spökingformer [Zhou et al., 2023a]
Spikingformer & 768
Spikingfommer- 8-312
29.68
66.34
13.68
7.46
75.85
74.79
STMixer-6 511--16
23.63
1.96
73.31
STMixer-8.512-t
owrs
STMixer- 10-512-16
36.61
2.48
74.91
STMises-8 768-16
61.16
4.45
76.68
5.2 Exploration of the architecture of Encoders
In this section, we endeavor validate the effect of different encoder structures (the structures of Cell I and Cell 2 in Fig. 1). Additionally, we also want to figure out whether the metaformer-lin like structure a superior architecture in SNN. As depicted in Fig. each cell consists of three positions, with each position having five possible structures (STM and SSA only use single head). Disregarding any duplicates generated by the identity structure, there are a total of 5 different substructures. Conducting a comprehensive validation of all these cases is impractical. Inspired by the concept of a one-shot NAS, we train supernet that encompasses all possible substructures. The supernet composed of three main components: the SPS, the classification head, and four identical encoders. Fach encoder consists of two cells that are used for network search. This supernet enables us to efficiently evaluate the performance of each substructure. The CIFAR training datasct is divided
8

---

20
24
34
Figure 4: Neural architecture search space and search results. Left: Search space of each Cell. Mid: random sample result from supernet on CIFAR-100. Right: random sample result from supernet on CIFAR-10.
Table Effect of STMixer componets and trainig methods
Table 3: Effect of number of STM heads
H
Accuracy (%)
Modification
Accuracy (%)
79.02
STMixer
79.55
2
79.07
IPSPS SPS
78.97
4
79.39
STM-SSS
78.33
SMLLSTM0-SMLLCCmy
79.07
8
79.51
SMI.-SDDT
77.17
16
79.55
32
80.15
into training and validation sets in a ratio of 8.5:1.5. During each forward pass, the supernet will randomly select path and train on the training set for 600 epochs. After the training completed, we randomly sample 4000 distinct subnetworks from the supernet and evaluate their performance on the validation set. Additionally, we calculate the forward FL.OPs for each of these subnetworks. Note that the FLOPs mentioned here refer to the estimated floating point operations on the training platform. The actual theoretical synaptic operations (SOPs) rely on FLOPs, the simulation time step, and spike frequency. We have categorized the subnetwork into four classes: (I) only convolutional layers: (2) combining SSA and convolutional layers; (3) incorporating STMixer and convolutional the mean and layers; and (4) simultaneously having STMixer and SSA. To draw ellipses, selection spaces. variance of accuracy and FLOPs, showcasing the relative merits of four different Furthermore, have marked the positions of STMixer and Spikingformer. As shown in Fig. 4, considering the computational complexity and network performance, as well as the suitability of the components used in asynchronous scenarios, STMixer appears to be the most promising architecture (STMixer is located at the top-left corner of the result figure).
5.3 Ablation studies
To understand the effect of various components and training methods in STMixer, we conducted ablation experiments on CIFAR-100 with 4TMMxer--3844-6 (T 1). The summarized results are presented in Table 2. As we can see, SML is the most significant improvement method. Considering that STMixer-4-384- 16 is indeed 25-layer network, the training process affects the nonnegligible gradient error accumulation. SML is an effective approach to mitigating the gradient error problem. And the employment of the modified SML algorithm with STM modules enables facilitation of training SNNs (+0.48%). Then. under the SML training method, the performance the STM module surpasses that of the SSA module (+1.22%). Finally, by preserving more information from the input images by IPSPS, the SNN achieves better performance (+).58%)
efficient
Another parameter that significantly influences the STMixer performance is H, the number of heads in STM. Here, we increment the value of H from 1 to 32 and evaluate the performance of STMixer- 4-384 on the CIFAR-100 100 dataset. As the number of heads (H) increases, the STMixer accuracy on the test set has improved from 79.02% to 80.15% 5.%. This observation highlights the necessity of performing token mixing at finer granularity.
5.4 Benchmark on fully event-driven asynchronous scenario
Besides the superior performance demonstrated on the aforementioned static datasets, our proposed STMixer model also exhibits significant adaptability for event-driven scenarios. In these scenarios,

---

SNNs are deployed on asynchronous chips to receive and process event streams from DVS cameras, thereby forming system that integrates sensing and computation. This integration leverages the sparse nature of spike activity, allowing SNNs to achieve ultra-low energy consumption compared to their ANN counterparts. To validate this advantage, we conducted additional experiments on fully asynchronous frameworks. Since the current commercial asynchronous chips (e.g.. Speck of the Synsense [Richter et al., 2024] and Loihi [Davies et al., 2018a, Lines et al., 2018] of Intel) are limited in capacities and lack support for various operators such shortcuts, we tested our models on home-made C++ simulator that reproduces the computational principles of the Speck DevKits (see Synsense Sinabs).
Table 4: STMixer vs Spikformer under large T and event-driven scenario on CIFARIO-DVSS
Model
w/ bias
T=25
T-40
T-80
T=160
T=320
event-driven
STMixer(T=23)
66.73
64.31
65.32
63.91
61.79
58.37
50.70
Spikformen
64.21
56.05
53.83
34.58
22.78
19.15
16.60
Model
w/ bias
T=40
T=80
T=160
T=320
T=640
event-driven
STMixer(T=
75.91
71.77
71.57
70.77
67.74
64.62
54.10
Spikformer(T=40)
67.74
10.08
10.08
10.08
10.08
10.08
10.00
We compared STMixer to a structurally identical SpikFormer, created by replacing the STM compo- nent in STMixer with SSA on CIFARIO-DVS. Due the prohibition of clock-driven bias in a fully event-driven scenario, we implemented two-phase training pipeline: initially training a standard SNN, then absorbing BN layers and eliminating all bias terms through additional fine-tuning, while the side effect is a slight performance degradation (see the w/ bias column and T=25(40) column in Table. 4). The training of SpikFormer becomes extremely unstable during the bias-removal phase when T=40 and fails to converge even with the learning rate reduced to 10-5.
We first deployed the two models on our event-driven simulator and evaluated their test accuracy. As shown in Table. 4) event-driven column, Spikformer undergoes significant performance degradation after deployment because events fail meet each other and end in SSA, whereas STMixer maintains high performance and demonstrates strong compatibility with the event-driven framework. We further assessed their event-driven adaptability by inference performance at large T. When T is considerably large, events rarely coincide within the same frame, effectively approximating an event-driven scenario. According to Table. 4), as T increases, Spikformer's 's performance deteriorates sharply, whereas STMixer exhibits only a modest accuracy decrease. These results are consistent with our analysis in Section 4.1. Further details are provided in the appendix.
6 Conclusion
The advantage of spiking neural networks over artificial neural networks is their efficient and low- power inferencing capabilities once implanted in neuromorphic hardware. Within neuromorphic hardware, synchronous hardware uses a hardware clock to control when the spikes fire and arrive. The power consumption of such hardware is influenced by the time step length (T), whereby a smaller T typically results in lower power consumption. Asynchronous hardware has lower energy consumption potential than synchronous hardware, but has limitations on the network structure and only supports some specific matrix operations due to the lack of the hardware clock. For the aforementioned reasons, SNN must use asynchronously supported operators to build new architectures (for asynchronous hardware) or complete the performance challenge with an extremely low time step (for synchronous hardware). In this paper, we propose a new architecture named STMixer that satisfies the above challenges. STMixer is completely composed of asynchronous scenario supported components, including convolutional layers, fully connected layers, and residual paths, and has excellent performance when the time step T = 1. Our work will provide ideas for the SNN architecture, which works on both event-driven friendly and clock- efficient SNNs.
7 Acknowledgment
This project is supported by NSFC Key Program 62236009, Shenzhen Fundamental Research Program (General Program) JCYJ 202103244408070199 NSFC General Program 61876032, and Key Laboratory of Data Intelligence and Cognitive Computing, Longhua District, Shenzhen.
10

---

References
Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, et al. Trucnorth: Design and tool flow of a mw million neuron programmable neurosynaptic chip. IEEE transactions on computer-aided design of integrated circuits and systems, 34( 34(10):1577-155, 2015.
Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, et al. low power, fully event-based gesture recognition system. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2017.
Abhiroop Bhattacharjee, Ruokai Yin, Abhishek Moitra, and Priyadarshini Panda. Are truly energy- efficient? a hardware perspective. ICASSP 2024-2024 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), pages 13311-13315. IEEE, 2024.
Daniel Bolya, Cheng-Yane Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient attention with many heads. In European Conference Computer Vision, pages 35-49. Springer, 2022.
Kaiwei Che, Luziwei Leng, Kaixuan Zhang. Jianguo Zhang, Qinghu Meng, Jie Cheng, Qinghai Guo, and Jianxing Liao. Differentiable hierarchical and surrogate gradient search for spiking neural networks. Advances in Neural Information Processing Systems, 35:24975 24990, 2022.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703, PMLR, 2020.
Manon Dampfhoffer, Thomas Mesquida. Alexandre Valentian, and Lorena Anghel. Are smns really more energy-efficient than anns? an in-depth hardware-aware study. IEEE Transactions on Emerging Topics in Computational Intelligence, 7(3):731-744) 2022.
Mike Davies, Narayan Srinivasa, Tsung-Har Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, Yuyun Liao, Chit-Kwan Lin, Andrew Lines, Ruokun Liu, Deepak Mathaikutty, Steven McCoy, Arnab Paul, Jonathan Tse, Guruguhanathan Venkataramanan, Yi- Hsin Weng, Andreas Wild, Yoonseok Yang. and Hong Wang. Loihi: A neuromorphic manycore processor with on chip learning. IEEE Micro, 38(1):82- 99, 2018a. doi: 10. 1090MMM.00 12213335,,
Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi. Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. leee Micro, 38( (3(1):82999) 99, 2018b.
Michael V DeBole, Brian Taba, Arnon Amir, Filipp Akopyan, Alexander Andreopoulos, William P Risk, Jeff Kusnitz, Carlos Ortega Otero, Tapan K Nayak, Rathinakumar Appuswamy, et al. Truenorth: Accelerating from zero to 64 million neurons in 10 years. Computer, 52(5):20 29, 2019.
Shikuang Deng. Yuhang Li. Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. In International Conference on Learning Representations. 2021.
Shikuang Deng. Hao Lin, Yuhang Li, and Shi Gu. Surrogate module learning: Reduce the gradient emror accumulation in training spiking neural networks. In International Conference on Muchine Learning, pages 7645-7657. PMLR, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 41 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov. Dirk Weissenbom. Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthiss Minderer, Georg Hcigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In Intermational Conference on Learning Representations, 2020.
Shuman Fang, Jie Li, Xianming Lin, and Rongrong Ji. Learning to learn transferable attack. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 571 579, 2022.
Wei Fang, Zhaofei Yu. Yanqi Chen, Tiejun Huang. Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34:21056-211 21069, 202

---

Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2661-2671, 2021b.
Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines. 30:681-694 2020.
Meng- Hao Guo, Zheng-Niug Liu, Tai-Jiang Mu, and Shi-Min Hu. Beyond acte-antention: External attention using two linear layers for visual tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 (5):5436-544) 5447, 2022a.
Yufci Guo, Yuampei Chen, Liwen Zhang, YingLei Wang, Xiaode Liu, Xinyi Tong, Yuanyuan Ou, Xuhui Huang, and Zhe Ma. Reducing information loss for spiking neural networks. In European Conference on Computer Vision, pages 36-52 Springer, 2022b.
Mark Horowitz. 1.1 computing's s energy problem (and what we can do about it). 2014 IEFF international solid-state circuits conference digest of technical papers (ISSCC), pages 10-14, IEFF, 2014.
Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks toward deep residual learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Seijoon Kim, Scongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for 2019.
energy-efficient object detection. arXiv preprint
Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, and Priyadarshini Panda. Neural architecture search for spiking neural networks. In European conference on computer vision, pages 36-56. Springer. 2022.
LM Lapicque. Recherches quantitatives sur l'excitation electrique des nerfs. J Physiol Paris, 9:620-635, 1907.
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Andrew Lines, Prasad Joshi, Ruokun Liu, Steve McCoy, Jonathan Tse, Yi-Hsin Weng, and Mike Davies. Loihi asynchronous neuromorphic research chip. In 2018 24th IEEE International Symposium on Asynchronous doi: 10.1109/ASYNC.2018.0000
Circuits and Systems (ASYNC), pages 32 33-33, 2018.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2002210022, 2021.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 1659)16591167,, 1997.
Byunggook Na, Jisoo Mok, Seongsik Park, Dongjin Lee, Hyeokjun Choe, and Sungroh Yoon. Autosnn: Towards energy-efficient spiking neural networks. In International Conference on Machine Learning, pages 16253-16269. PMIR, 2022.
Emre Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- optimization spiking neural networks. JEEE Signal Processing Magazine, 36(6):51-63, 2019.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pages 4055-4064. PMLR, 2018.
Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572 (7767):106-11) 11100061111000111111111111111111 2019.
Nitin Rathi and Kaushik Roy. Diet-smn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint urXiv:2008.0366 2020.
Ole Richter, Yannan Xing, Michele De Marchi, Carsten Nielsen, Merkourios Katsimpris, Roberto Cattaneo, Yudi Ren. Yalun Hu. Qian Liu, Sadique Sheik, et al. Speck: A smart event-based vision sensor with a low lateney 327k neuron convolutional neuronal network processing pipeline. 2024.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784)6007-6177 2019.
12

---

Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedingy of the IEEE/CVF winter conference on applications of computer vision, pages 3531-3539, 2021.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung. Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et Mip-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:24261-24222 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347 10357. PMLR, 2021.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbock, et al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(4):5314-55 5321, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz. Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-pee ceah-peeffraanceeeeee spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Zhenzhi Wu, Hchui Zhang, Yihan Lin, Guoqi Li, Meng Wang, and Ye Tang. Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin. Zhaoxu Yang. and Guoqi Li. Temporal- wise attention spiking neural networks for event streams classification. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10221 2021.
Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng. Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. IEEE transactions on pattern analysis and machine intelligence, 45(8):9393-9410, 2023.
Man Yao, Jiakui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer. Advonces in neural information processing systems, 36, 2024.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 108 19-9990829, 2022.
Zhilu Zhang and Mert Sabuncu. Self-distillation instancesspecite label smoothing. Advances in Neural Information Processing Systems, 33:2184-2195, 2020.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with firectly-trainee larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages
11062-11070, 2021.
Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Han Zhang, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual learning for transformer-haset spiking neural network. arXiv preprint arXiv:2304. 11954, 2023a.
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang. Shuicheng YAN, Yonghong Tian, and Yuan. Spikformer: When spiking neural network meets transformer. In The Eleventh International Conference on Learning Representations, 2023b. URL httttpp://opereeiieee
h.
Rui-Jie Zhu, Malu Zhang, Qihang Zhao, Haoyu Deng, Yule Duan, and Liang-Jian Deng. Teja-snn: Temporal- channel joint attention for spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2024.
13

---

Figure 5: The STM attention weights of different encoders. Each row illustrates the 16 attention weights for each encoder.
A Attention Weight Visualization
Here show the token mixing attention weights of the encoders from the well-trained STMixer-8 8- 768-16 on the ImageNet dataset. These weights provide insights into the blending process of tokens, revealing how they are mixing information. As shown in Fig. 5, most of the weight matrices exhibit pronounced regularity: there are some parallel lines near the symmetry axis. This observation implies two insights. While the attention weight of the STM enables the extraction of global information, it still tends to favor the extraction of local information. In visual tasks, the correlation between each patch and its neighboring patches is maximized. Moreover, it can be observed that the 16 attention weight matrices in each row are different, indicating that each head of the multi-head STM is capable of extracting distinct token mixing patterns.
B Experiments
B.1 Experiments settings
Training for comparison with existing works All experiments on CIFAR were conducted on two 4090 GPUs, while experiments on ImageNet were performed on four 4090 GPUs. We employed the same training script as Spikingformer [Zhou et al., 2023a] and employed identical data augmentation techniques. In the CIFAR experiments, we modify the learning rate to be 5e-3 and the minimum learning rate to 1e-7. In the ImageNet experiments, we modify the learning rate to be 3e-4 and the minimum learning rate to le-5.
Training for event-driven experiments Models for event-driven experiments are all trained by our two-phase training pipeline. Specifically,, we first train an SNN with bias and BN layers. Before further fine-tuning, we absorb BN layers and set all bias terms to O. Finally, we fine-tune the model for more epochs with bias frozen. For the first phase, we use the AdamW optimizer and set the learning rate to 0.001 and weight decay to 0.02. For the second phase, we use the AdamW optimizer and set the learning rate 0.001 and the weight decay to 0.005, For Spikformer and STMixer with T=25, a time step of T=10 is used for the first phase over 300 epochs. T is then set to 25 for the bias-removal phase, during which models are fine-tuned for an additional 30 epochs. For Spikformer and STMixer with T=40, consistent T=40 is used throughout both phases. Models are trained for 100 epochs during the first phase, followed by epochs of fine-tuning during the bias-removal phase. Notably, we found that MultiSpike IF neurons with soft reset (see Synsense Sinabs) are optimal for training models intended for event-driven deployment, and this configuration was applied in all event-driven experiments.
14

---

Backbone for event-driven experiments We employed the modified STMixer-2-255 2-21xer-2--5688 as the backbone for all event-driven experiments, incorporating several adjustments to enhance its suitability for fully event-driven deployment. First, the non-spiking membrane shortcut was replaced with standard spiking shortcut [Zheng al., 2021] for event-driven implementation. Second, the first two convolution layers of the SPS module are replaced with single 4 x 4 Avgpool layer. Building upon this foundation, we replace the STM module with SSA to construct the Spikformer model.
B.2 Compare with existing works on neuromorphic datasets in synchronous scenarios
Table 5: Performance comparison of our method with existing methods on CIFARIO-DVS and DVS128 Gesture.
CIFARIO-DVS
DVSI28 Gesture
Method
Time step
Acc
Time step
Acc
LIAF-Net
[2021]
70.4
60
97.6
TA-SM SNN Yao et al. [2021]
10
72.0
60
98.6
tdBN Zheng et al. [2021]
10
67.8
40
96.9
PLIF Fang al. [2021b]
10
74.8
20
97.6
Spikformer Zhou [2023h]
10
78.6
10
95.8
Spikingformer Zhou [2023a]
79.9
10
96.2
ours
10
82.67
10
97.19
CIFARIO-DVS is neuromorphic dataset that is obtained from the CIFAR-10 dataset through DVS camera. There are 10k images in CIFARIO-DVS, and we split them into 9k training images and Ik test images. We downsample the image resolution from 128 x 128 to 48 x48 and input them to STMixer-2-5i
structure SNN. We don't 't use the IPSPS module here because we find it does not enhance the performance the neuromorphic dataset too much. For the training setting, employ the AdamW optimizer with a learning rate of 0.01 and weight decay of 0.02. The training is conducted for 300 epochs. Meanwhile, we employ the same data augmentation methods previous works Li et al. [2021], Deng et al. [2021].
As depicted in Table. 5, architecture achieves the best performance on CIFARIO-DVS, demon- strating the suitability of STMixer for this neuromorphic dataset.
DVS128 Gesture Amir et al. [2017] is gesture recognition dataset that consists of 11 hand gesture classes performed by 29 individuals under different lighting conditions. We employ the network architecture STMixer-2-5122 2--8 and apply the same augmentation techniques from Spikingformer except cutmix. We don't 't use the IPSPS module here because we find it does not enhance the performance on neuromorphic dataset too much. We train the network for 200 epochs with TET loss Deng et al. [2021] and the SML method. STMixer also achieves the best performance on this dataset.
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We have stated the abstract and introduction that the model structure proposed can support asynchronous hardware and has low energy consumption synchronous hardware.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
16

---

Justification: In the 4.2 Clock-driven Scenarios section, we mentioned that the current SNN energy cost calculation equation incorrect and ignores the cost of memory calls. Furthermore, the real energy cost is still proportional to the time step, so SNN needs to use a lower time step to reduce potential energy consumption. Constructing an accurate energy consumption calculation equation is beyond the scope of this paper. In addition, due to current asynchronous hardware scale limitations (e.g., Speck 2e only supports 9-layer SNNs), cannot implant large-scale SNNs into the hardware for testing. Therefore, we can only theoretically build models using the asynchronous hardware supported operators.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide the explanation of why the multi-head STM module is effective from the perspective of fitting the attention map, and explaion the effectiveness of IPSPS from the perspective informantion loss.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
17

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the experiments in this paper the open-source dataset. And we have provided the code that reproduces the results of the paper in the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open tions to faithfully reproduce the main experimental results, as described in supplemental material?
the data and code, with sufficient instruc-
Answer: [Yes]
Justification: we have provided the code that reproduces the results of the paper in the supplementary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. co/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
18

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //niss.. /ttpss.ccppublllee/ wwwpiiic/ggi/////eeeeeeeeeeeeeee /ttegiidee////eduuiiiii (oodeSubmissiiie misnooiicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We use the same training script the open source work and provide repro- ducible training code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In comparing the results with existing work, in addition to the Imagenet results, we provide the mean and standard deviation of the results of three different seed experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
19

---

8. Experiments Compute Resourees
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide the above information in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https h/neurrrips.eee httpp///pptttttt.hccccceeeeeeeee
Answer: [Yes]
Justification:
The research with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed it in the conclusion part.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
20

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- usal-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cite the original paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
21

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "zIn5FM1TlX", "reviewer": "Reviewer_2hoo", "summary": "This article examines problems with the SSA module in Spikformer in asynchronous scenarios and suggest a new module, the Spiking Token Mixing (STM) module, which consists solely of network components that cater to asynchronous environments. Besides, This article proposed the information protection spiking patch splitting (IPSPS) module to reduce information loss.", "questions": "1. The training cost (memory consumption and training time) of STM, SML, and STMixer should be reported compared with previous work.\n2. The energy consumption of the SML block needs discussion. \n3. A complete comparison of STM and SSA need to be carried out (report the results of replacing SSA in Spikingformer or Spikformer by STM block.).", "limitations": "See weaknesses and questions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This article examines problems with the SSA module in Spikformer in asynchronous scenarios, which is a good observation. And authors intend to solve it by proposing STM and IPSPS.", "weaknesses": "1.From your code and Table 2 (SML->SDT). The performance improvement mainly comes from several SML blocks, which consist of several ANN layers. When considering adding residual connection and several SML blocks, the STMixer can even be seen as an ANN model (floats input is connected to the last layer through intermediate hidden ANN ). In my opinion, this is a very fatal question.\n2. From the perspective of solving the asynchronous scenarios problem, the residual connection is also a problem, residual connections in the pre-activation shortcut are floats point element-wise addition, which also suffers the same problem in maxpooling and spiking matrix multiplication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment 1:\n\nFrom your code and Table 2 (SML->SDT). The performance improvement mainly comes from several SML blocks, which consist of several ANN layers. When considering adding residual connection and several SML blocks, the STMixer can even be seen as an ANN model (floats input is connected to the last layer through intermediate hidden ANN ). In my opinion, this is a very fatal question. \n\nResponse to comment 1:\n\nWe apologize for the confusion. The STMixer does not involve the ANN module during the inference stage. The primary function of the SML block [1] is to transmit effective gradients to the SNN's intermediate layers during back-propagation, and it does not contribute additional information to the backbone SNN during forward propagation. Consequently, the SML block does not impact the forward propagation of the backbone SNN during the inference phase, as a result, SML blocks can be removed after the completion of training. The reported accuracy in all experiments pertains to the backbone SNN, not the SML block. We have added relevant instructions to the updated manuscript.\n\n\nComment 2:\n\nFrom the perspective of solving the asynchronous scenarios problem, the residual connection is also a problem, residual connections in the pre-activation shortcut are floats point element-wise addition, which also suffers the same problem in maxpooling and spiking matrix multiplication.\n\nResponse to comment 2:\n\nIn asynchronous scenarios, a pre-activation residual connection is not impossible to realize. Despite being termed as 'membrane shortcut' [2], the variable employed for the shortcut is the increment of the membrane potential, not its cumulative value (residual membrane potential). The hardware needs a circuit that supports floating-points values transmit to realize the membrane shortcut. Although spike arrival delay may slightly alter the output of the convolutional or fully connected layer, the output cumulative value remains constant. As a result, the sum of the increments in the shortcut from the current layer to the subsequent neuronal membrane potential will not change significantly. In other words, the spike arrival delay can affect the sequence of spike arrivals, but the sequence does not affect the summation, which has little impact on the shortcut.\n\nComment 3:\n\nThe training cost (memory consumption and training time) of STM, SML, and STMixer should be reported compared with previous work.\n\nResponse to comment 3:\n\nWe compared the training time and memory consumption of STMixer (T = 4) on CIFAR-100 when using either STM or SSA as the token mixer. And we also provide the training consumption of STMixer with SML. The table below summarizes the results. The training overhead for the three situations does not differ significantly, with the SSA case having the greatest memory consumption and the STM+SML case taking the longest training time. We hope this information is helpful. Please let us know if there are any other aspects you would like us to address.\n   | Case    | training time per epoch | memory consumption |\n   | ------- | ----------------------- | ------------------ |\n   | STM     | 33.27 s                  | 11,774 MB            |\n   | STM+SML | 40.04 s                  | 12,514 MB             |\n   | SSA     | 34.37 s                  | 13,994 MB            |\n\nComment 4:\n\nThe energy consumption of the SML block needs discussion.\n\nResponse to comment 4:\n\nWe apologize if there was any confusion caused previously. The SML block is primarily involved in the training phase of the Spiking Neural Network (SNN), but it does not participate in the inference phase. As such, while it does have an impact on the training energy consumption of the system during the training phase, it does not contribute to the energy consumption during the inference phase of the SNN. We hope this clarifies the role of the SML block in the energy consumption.\n\nComment 5:\n\nA complete comparison of STM and SSA need to be carried out (report the results of replacing SSA in Spikingformer or Spikformer by STM block.).\n\nResponse to comment 5:\n\nWe use the CIFAR-100 training script provided by the Spikingformer study and merely replaced its SSA module with STM. Consequently, the accuracy of the SNN dropped from 76.25% to 75.86%. This outcome aligns with the network search results displayed in Figure 4, suggesting that while the SSA module entails higher computational load, its performance is slightly superior to that of STM. Although STM's performance is inferior to SSA's, it possesses two unique advantages—lower computational energy on synchronous hardware and suitability for asynchronous hardware. Moreover, as shown in Table 2, STM benefits more from the SML training algorithm, which enables STM to achieve a performance of 79.55%, significantly higher than SSA's 78.33%. This indicates that STM holds the potential to rival the performance of the SSA module.\n\n[1] Deng S, Lin H, Li Y, et al. Surrogate module learning: Reduce the gradient error accumulation in training spiking neural networks[C]//International Conference on Machine Learning. PMLR, 2023: 7645-7657.\n[2] Hu Y, Deng L, Wu Y, et al. Advancing spiking neural networks toward deep residual learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q0SoR7TblE", "reviewer": "Reviewer_ESkb", "summary": "Most of the spiking neural network architectures can not truely show the superiority on the neuromporphic hardware, since in event-driven scenarios, the spike arrival times are not  precise and could result in significant differences in the output, like there is a max pooling layer. This paper propose the Spiking Token Mixer, which can well handle the problem. The authors also have validated that STMixer could achieve performance on par with or even surpasses existing Spikformer-like works in synchronous scenarios.", "questions": "1.  Can you further  explain the meaning of the sentence in first paragraph of Introduction that \"due to the accumulation of membrane potential in the IF neuron, spike arrival timing errors do not significantly impact subsequent layers.\".\n2. Could you provided a detailed explanation for the method?", "limitations": "None", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-organized and written in a clear manner, making it accessible to readers interested in the topic.\n\n2. The paper solves an important problem in the SNN filed and the proposed method is interesting and new.\n\n3. The paper povide various experiments and ablations to  show the effectiveness of the proposed method.", "weaknesses": "1. Does the proposed method increase the energy consumption? The authors could provided detailed explanations.\n\n2. The ablation experiments could include different types of datasets and models.\n\n3. The comparison methods in the experiments contrasting with other approaches on CIFAR-10 may not be the most up-to-date.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment 1:\n\nDoes the proposed method increase the energy consumption? The authors could provided detailed explanations.\n\nResponse to comment 1:\n\nTheSML method does not increase the energy consumption during the inference stage. After the training phase end, the SML method eliminates the added blocks, ensuring that the energy consumption of SNN inference remains unaffected. As demonstrated in Table 1, the STMixer does not introduce a significant additional energy cost compared to the Spikingformer. For instance, the energy consumption of STMixer-8-768-16 at T=1 is 4.45mJ, whereas the Spikingformer-8-768 at T=4 consumes 13.68mJ. Considering that energy consumption is proportional to the time step T, we can infer that an STMixer, with the same scale and time step, would have energy consumption comparable to that of a Spikingformer.\n\nComment 2:\n\nThe ablation experiments could include different types of datasets and models.\n\nResponse to comment 2:\n\nWe have conducted an ablation study on the CIFAR-10 dataset. The experiment results validate the effectiveness of all the designed components. The results are summarized in the table below.\n | Modification | STMixer | IPSPS -> SPS | STM -> SSA | SML(STM)->SML(Conv) | SML->SDT |\n   | ------------ | ------- | ------------ | ---------- | ------------------- | -------- |\n   | Accuracy(%)  | 95.65   | 95.49        | 94.45      | 95.40               | 95.13    |\n\nComment 3:\n\nThe comparison methods in the experiments contrasting with other approaches on CIFAR-10 may not be the most up-to-date.\n\nResponse to comment 3:\n\nThank you for your valuable feedback. In the original manuscript, we indeed primarily compared our method with classical structure works and related studies. Recognizing the importance of keeping our work current and relevant, we have now updated our comparison in the revised manuscript to include more recent approaches. Specifically, these updates have been incorporated into Table 1. We believe that these changes will provide a more comprehensive and up-to-date comparison, addressing your concern. \n\nComment 4:\n\nCan you further explain the meaning of the sentence in first paragraph of Introduction that \"due to the accumulation of membrane potential in the IF neuron, spike arrival timing errors do not significantly impact subsequent layers.\".\n\nResponse to comment 4:\n\nDuring the SNN training phase, we convert the DVS stream in a time window into an image frame. For instance, if a position (x, y) of the DVS stream has 4 spikes in the time window, the value at position (x, y) in the image frame would be 4. This value is then input into the SNN for training. Suppose an Integrate-and-Fire (IF) neuron (threshold is 1.0 and employing soft reset) only receives input from the point (x, y), and the weight is 0.3. After receiving the input, the neuron would generate a spike by the end of this frame, leaving the residual membrane potential of 0.2. In asynchronous scenarios,  spikes in the DVS data stream enter the neuron one by one, not in frame format. Each arriving spike increases the membrane potential by 0.3, ultimately generating a spike and leaving a residual membrane potential of 0.2.  If there is a certain error in the arrival time of the spikes, as long as the number of spikes is constant, the neuron will still generate a spike and leave a residual membrane potential of 0.2 after operation.  Of course, the above scenario is the simplest example. In reality, there may be errors due to uneven spike generation, which can have some effects on subsequent layers.\nStarting from this simple example, we can infer that traditional operators such as convolution and full connection can alleviate the problem of pulse arrival error to a certain extent, as long as there is an IF neuron layer subsequently.\n\nComment 5:\n\nCould you provided a detailed explanation for the method?\n\nResponse to comment 5:\n\nOur approach is based on modifications to the Spikingformer model, specifically its SPS and SSA components. The modifications were made with two objectives in mind: 1) to adapt the model for asynchronous scenarios, and 2) to enhance the model's performance. To achieve the first objective, we eliminated the max pooling layer in the SPS and performed downsampling by modifying the stride of the convolutional layer. Concurrently, we proposed the STM module to replace the SSA module. The STM module only contains fully connected components and does not introduce multi-branch spike matrix operations.  The STM divides the input $X \\in \\mathbb{R}^{T\\times C \\times N}$ after the fully connected layer into H parts, each part $X^h  \\in  \\mathbb{R}^{T\\times C/H\\times N}$ is mixed at the token dimension through a fully convolutional matrix $W^h\\in \\mathbb{R}^{N\\times N}$. We found that under the same training script, the performance of STM is not inferior to that of SSA. To achieve the second objective, we proposed the IPSPS method to replace the SPS module. A small portion of IPSPS's output tensor comes from the input directly transformed by a convolutional layer, thereby preserving sufficient input information. We also modified the surrogate module used by the SML method, using the STM module to construct the SML block, which significantly reduces the additional training overhead brought by the SML method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fLTYHOZ8Mv", "reviewer": "Reviewer_DoAS", "summary": "To address the issue where certain operators (e.g., spiking self-attention, SSA) in existing spiking Transformers cannot be executed on asynchronous neuromorphic chips, this work designs the Spiking Token Mixer (STMixer) architecture, which consists exclusively of operations supported by asynchronous scenarios including convolutional, fully connected layers, and residual paths.", "questions": "1. Some references are incorrect and cannot be accessed. LInes 91-92, 255\n\n2. The mathematical descriptions in Section 4.3 are not rigorous and omit some crucial information. Is the matrix X composed of continuous values or is it a spiking matrix? All the dimensional changes are ignored, which is unprofessional. You cannot expect every reader to read all your references.\n\n3. Could the authors explain the proposed STM from the perspective of linear attention? Since the LIF following Q and K functions as a kernel, why can the spiking neuron layer be directly removed in this context?\n\n4. SML training seems crucial, but other baseline works do not appear to use this method. Therefore, is the comparison in Table 1 unfair? Why is there no comparison with Spike-driven Transformer v1/v2 in Table 1?\n\n5. What is SDT in Table 2? It is mentioned only once in the entire text and is not explained.\n\n6. I suggest the authors carefully check the references. If the paper has been accepted, please cite the official version instead of the arXiv version, as this is an academic convention.", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This work raises a significant issue: the current spiking self-attention operators are indeed difficult to implement on asynchronous chips. Generally, designers of SNN algorithms do not often consider their hardware execution. I find it reasonable that the authors have designed algorithms with hardware limitations in mind.", "weaknesses": "1. The technical contribution is limited; there is no in-depth analysis of the proposed operators, and some existing work in the field is overlooked. \n2. Some key points are not clearly explained, especially the section on operators.\n3. The work does not follow the latest results in the field, e.g., meta-spikeformer [1], and the performance is not SOTA.\n\n[1] Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. In ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment 1:\n\nThe technical contribution is limited...\n\nResponse to comment 1:\n\nI appreciate the reviewer's feedback but respectfully disagree that the technical contribution of this paper is limited. This paper does not merely propose a structure (implant ANN former structure into SNN) to achieve the SNN SOTA performance on GPU. Instead, it designs a model suitable for asynchronous scenarios, inspiring future research to refocus on the running environments of SNNs. As mentioned in the General Response, constructing a high-performance network using operators supported by current asynchronous hardware is significantly more challenging than using those supported by synchronous hardware. Currently, advanced SNN models use multibranch spike matrix multiplication, Hadamard product operations, and others, which allow for straightforward access to global information  but difficult to execute in event-driven asynchronous settings. The STM module proposed in this article employs simple weight matrices to mix global information across token dimensions, this technique has already proven effective in works such as ANN's MLP-Mixer[1] and External Attention[2]. Additionally, STM adjusts the number of parameters and performance by employing multiple heads (i.e., multiple mixing weight matrices). \n\nComment 2:\n\nSome key points are not clearly explained, especially the section on operators.\n\nResponse to comment 2:\n\nWe appreciate your feedback and have revised Section 4.3 to provide a more comprehensive explanation of the STM operator. \nPlease refer to \"Response to comment 5\" for reviewer ESkb.\n\nComment 3:\n\nThe work does not follow the latest results in the field, e.g., meta-spikeformer [1]...\n\nResponse to comment 3:\n\nThe architecture of STMixer is not at odds with the meta-transformer structure; rather, it is composed of a token mixer module and a channel mixer part. We have incorporated the spike-driven transformer into Table 1 for a comparative analysis.  Although STMixer's performance on ImageNet may not surpass that of the Spike-driven transformer v2, it's important to note that due to STMixer's ultra-lightweight structure, it only consumes one-third of the energy that the Spike-driven transformer v2 does under the same parameter scale (in a synchronous scenario). As an example, Meta-SpikeFormer (T=1) consumes 7.8mJ of energy with 31.3M parameters and achieves 75.4% accuracy on ImageNet. In contrast, STmixer-8-512-16 (T=1) with 30.12M parameters achieves 73.82% ImageNet accuracy but only consumes 2.2mJ of energy. \n\nComment 4:\n\nSome references are incorrect and cannot be accessed. LInes 91-92, 255\n\nResponse to comment 4:\n\nWe apologize for the oversight. We have thoroughly reviewed the references and have corrected any inaccuracies.\n\nComment 5:\n\nThe mathematical descriptions in Section 4.3 are not rigorous and omit some crucial information...\n\nResponse to comment 5:\n\nThank you for your comment. We apologize for any confusion caused by our initial presentation. To clarify, the matrix X indeed represents a spike matrix, serving as the input for Cell 1 as shown in Fig. 1A. We acknowledge the importance of providing comprehensive dimensional information and regret the omission in our previous version. To address your concerns, we have revised Section 4.3 in the manuscript to include detailed dimensional changes and additional necessary information for each variable. \n\nComment 6:\n\nCould the authors explain the proposed STM from the perspective..\n\nResponse to comment 6:\n\nOur proposed STM is analogous to the workings of the MLP-Mixer, which uses a weight matrix to mix token dimension information. Its mixing weight matrix $W_\\text{STM}^h\\in\\mathbb{R}^{N\\times N}$ can be viewed as an approximation of the attention matrix to a certain extent.  To better illustrate this point, we remove the LIF layers from Q and K. We have found that this operation does not compromise the performance of SNN (79.81% to 80%).\n\nComment 7:\n\nSML training seems crucial, but other baseline works do not appear to use this method. Therefore, is the comparison in Table 1 unfair? Why is there no comparison with Spike-driven Transformer v1/v2 in Table 1?\n\nResponse to comment 7:\n\nThank you for your insightful comment. Our primary intention is to illustrate the potential of the SML algorithm when optimized by the STM module, which results in a significant performance boost for STMixer. We believe this indicates the potential for SNNs to achieve outstanding performance. To address your concern about fairness in comparison, we have now included the results of STMixer without SML training in Table 1. Even without SML training, STMixer still outperforms the comparison work with an accuracy $79.79\\% \\pm 0.19$ accuracy on CIFAR-100 and $95.96% \\pm 0.04$ on CIFAR-10 for STMixer-4-384-32 (T=4). In response to your second query, we have now also included the results of Spike-driven Transformer v1/v2 in Table 1 for a more comprehensive comparison.\n\nComment 8:\n\nWhat is SDT in Table 2..\n\nResponse to comment 8:\n\nWe apologize for the oversight. SDT in Table 2 stands for \"Standard Direct Training\". This refers to our baseline training pipeline, which is conducted without the use of the SMLmethod.\n\nComment 9:\n\nI suggest the authors carefully check the references..\n\nResponse to comment 9:\n\nThank you for your suggestion. We have carefully reviewed all the references and replaced the citations of arXiv versions with the official versions of the papers where they have been officially published. We appreciate your attention to detail and adherence to academic convention.\n\n\n[1] Tolstikhin I O, Houlsby N, Kolesnikov A, et al. Mlp-mixer: An all-mlp architecture for vision[J]. Advances in neural information processing systems, 2021, 34: 24261-24272.\n\n[2] Guo M H, Liu Z N, Mu T J, et al. Beyond self-attention: External attention using two linear layers for visual tasks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(5): 5436-5447.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General Response**

We appreciate all of the reviewers' comments and reviews. Here, we would like to provide a general response to reemphasize the motivation of this paper and its contribution to the SNN field. 

The main goal of this work is to design a well-performing SNN model that is friendly to asynchronous environments. Currently, SNN hardware bifurcates into synchronous and asynchronous types. Asynchronous hardware, devoid of a hardware clock and entirely event-driven, emerges as the ideal choice for SNN due to its substantially lower energy consumption. Nevertheless, it only supports very few network operations compared to synchronous hardware. 

On the other hand, current advanced SNN models have adopted the design philosophy of Transformer, enhancing their performance significantly beyond previous spiking CNN models on GPU simulation. However, except for matrix addition, many of these SOTA SNN models necessitate other operations (e.g., matrix multiplication) when merging multi-branch spike matrices, which require precise timings for spike arrivals. In asynchronous environments, it is very challenging to ensure the simultaneous arrival of two spikes. Related works often overlook discussions about the SNN running environment. Our study aims to provoke researchers to rethink the adaptability of SNN models to different running environments. 

Given the limited network operations supported by asynchronous hardware, the design of asynchronous friendly SNN is constrained, making the task of achieving superior performance with new models far more challenging than with existing advanced models. Consequently, the goal of this research is not to outperform the current SOTA SNN models, but to find an SNN model that is both compatible with asynchronous environments and capable of excellent performance in synchronous settings. With these considerations in mind, we propose the STMixer, which is composed of convolution, fully connected, and residual structures. It does not involve operations other than addition when merging multiple branches; thus, theoretically, STMixer supports asynchronous environments. Concurrently, our experiments demonstrate that STMixer can achieve outstanding performance even at extremely low time steps, suggesting its potential as an alternative SNN model for synchronous hardware as well.  

STMixer can provide guidance for the design of asynchronous-friendly SNN models. And we hope that  this paper will guide future SNN structural design efforts to pay more attention to implanting environment issues, further yielding a greater number of excellent asynchronous-friendly SNN models and training algorithms.

Meta-review: The authors argue that certain operators used in Transformer implementations are not very suitable for asynchronous spiking based neuromorphic chips. The authors make an attempt to provide ways so that the self-attention module can be implemented in a way suitable for asynchronous chips. The reviewers agree that that the authors are focusing on a real problem that limits the applicability of transformers on asynchronous chips.

Among the weaknesses listed are presentation issues (missing or incorrect references and notational issues making some parts of the paper feel a bit unprofessional). Another weakness is that the authors seem to make claims on the suitability of the algorithm for implementation on neuromorphic hardware, yet they do not seem to take into account issues that may hinder the adoption of such an algorithm on actual hardware. These can include issues that are specific to each hardware implementation, such as the fan-out supported by the hardware's neurons or the spike rate supported. 

Nevertheless there is consensus among the reviewers that the merits of the paper outweigh the drawbacks, and the authors seem to have answered the reviewer questions to their satisfaction. The authors should fix all the presentation issues and the references issues mentioned by the reviewers before publication.
Decision: Accept (poster)