Spiking Token Mixer:  A event-driven friendly Former structure for spiking neural networks

Authors: shikuang  deng
yuhang  wu
kangrui  du
shi  gu
encoder  block
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 04:37:23

Abstract: Spiking neural networks (SNNs), inspired by biological processes, use spike signals for inter-layer communication, presenting an energy-efficient alternative to traditional neural networks. To realize the theoretical advantages of SNNs in energy efficiency, it is essential to deploy them onto neuromorphic chips. On clock-driven synchronous chips, employing shorter time steps can enhance energy efficiency but reduce SNN performance. Compared to the clock-driven synchronous chip, the event-driven asynchronous chip achieves much lower energy consumption but only supports some specific network operations. Recently, a series of SNN projects have achieved tremendous success, significantly improving the SNN's performance. However, event-driven asynchronous chips do not support some of the proposed structures, making it impossible to integrate these SNNs into asynchronous hardware. In response to these problems, we propose the Spiking Token Mixer (STMixer) architecture, which consists exclusively of operations supported by asynchronous scenarios including convolutional, fully connected layers, and residual paths. Our series of experiments also demonstrate that STMixer achieves performance on par with spiking transformers in synchronous scenarios with very low timesteps. This indicates its ability to achieve the same level of performance with lower power consumption in synchronous scenarios. Codes are available at \url{https://github.com/brain-intelligence-lab/STMixer_demo}.

Main paper: 
 introduction: 
 Spiking neural networks (SNNs) are bio-inspired neural networks that use spike signals as a carrier for inter-layer transmission. Because of the spiking mechanism, SNNs have the potential to turn high-precision multiplication into addition, which makes them a better energy-saving infrastructure [Maass, 1997 , Roy et al., 2019] . However, such an advantage in efficiency is not guaranteed for typical hardware [Kim et al., 2019] but for neuromporphic chips [Akopyan et al., 2015 , Davies et al., 2018b , Pei et al., 2019] . The computation energy of synchronous neuromporphic hardware is affected by the calculation quantity, spike frequency, and SNN time steps [Rathi and Roy, 2020, Li et al., 2021] . And synchronous hardware provides support for nearly all matrix operations, even allowing for the execution of mixed networks that combine the ANN and SNN. In order to achieve low power consumption on synchronous hardware, SNNs must consider using shorter time steps. But too few time steps may potentially undermine the performance of SNNs. On the other hand, in theory, eventdriven asynchronous hardware serves as an ideal implementation platform for such a mechanism. On the event-driven hardware, each neuron is configured as an independent entity driven by events rather than the high-frequency clock. Compared to synchronous scenarios, asynchronous scenarios impose more stringent constraints on network architecture. Hence, in asynchronous neuromporphic hardware, SNN usually utilizes the soft-reset Integrate-and-Fire (IF) model as the fundamental neuron model. The supported network operations in asynchronous scenarios adhere to the rule that matrix multiplication should take place between a spike matrix and a fixed float weight matrix. Consequently, due to the accumulation of membrane potential in the IF neuron, spike arrival timing errors do not significantly impact subsequent layers.Over the past years, there have been a lot of efforts dedicated to advancing the performance of SNNs across various datasets. These efforts have explored a multitude of perspectives, aiming to optimize the training pipeline [Neftci et al., 2019 , Wu et al., 2018 , Li et al., 2021 , Deng et al., 2023] , refine the optimization objectives [Deng et al., 2021 , Guo et al., 2022b] , and design the network structure [Fang et al., 2021a , Zhou et al., 2023b,a] . Recently, Spikformer [Zhou et al., 2023b,a] has achieved remarkable results by incorporating the Transformer into SNN, resulting in unprecedented performance gains across a wide range of datasets. Spikformer has designed a new attention module, SSA, which aligns Spikformer with the forward rules of SNN during the inference phase. Such progression in a Spikformer-like design theoretically eliminates the need for high-precision matrix multiplication. Spikformer-like models have indeed demonstrated significant performance improvements on synchronous hardware or GPU simulation. However, an easily overlooked point is their feasibility in implementing asynchronous hardware. In event-driven scenarios, the spike arrival times may not be as precise as in synchronous scenarios with hardware clocks. Calculating two spike matrix multiplications or a spike matrix pass through the max pooling layer could result in significant differences in the output.To address these issues, we proposed a new architecture called Spiking Token Mixer (STMixer), which only uses asynchronous neuromporphic hardware compatible operations, including convolutional, fully connected layers, and residual paths. STMixer still adopts the holistic framework of spikformer, which consists of spiking patch splitting (SPS), encoders, and classification heads (Fig. 1 (A) ). The encoder module in STMixer consists of two components: the token mixing module and the channel module. In the token mixer component of STMixer, the value matrix undergoes matrix multiplication with a trainable attention map weight matrix (Fig. 2 (A) ). This new architecture eliminates the requirement of performing matrix multiplication between two spike matrices during the forward pass, thereby conforming to the event-driven situation. We also analyzed and enhanced the SPS module, ensuring its compliance with the requirements of an asynchronous environment while reducing information loss. Furthermore, we employed the proposed mixer architecture to optimize the components of the surrogate module [Deng et al., 2023 ] and used the surrogate module learning to assist the SNN training. The following summarizes our main contributions:• We examine problems with the SSA module in Spikformer in asynchronous scenarios and suggest a new module, the Spiking Token Mixing (STM) module, which consists solely of network components that cater to asynchronous environments.• We investigate the SPS module's asynchronous support and the information loss problem.To address these concerns, we proposed the information protection spiking patch splitting (IPSPS) module, which removes the maxpool layer and adds a residual path to protect information.• Through the network architecture search, we verified that the STMixer structure excels among all possible configurations in terms of performance and parameter quantity. Additionally, we have validated that STMixer achieves performance on par with or even surpasses existing Spikformer-like works in synchronous scenarios by using only a single time step across almost all datasets. This suggests that STMixer has the potential to deliver high performance with lower power consumption in synchronous scenarios, too.Spiking Patch Splitting (SPS) 2 Related Work× = × = Q K T Attn Map 1 1 1 1 1 1 1 1 2 × = × = Q K T Attn Map 
transformer-like architectures in artificial neural network: 
 Convolutional networks, such as ResNet and VGG, have historically been the most popular architectures in the realm of computer vision [Zheng et al., 2021, Rathi and Roy, 2020] . Some works have tried to add attention mechanisms to visual tasks [Yao et al., 2023 , Zhu et al., 2024] after being inspired by the amazing results of transformers in NLP [Vaswani et al., 2017 , Devlin et al., 2019 , Floridi and Chiriatti, 2020] . Especially, the Vision Transformer (VIT) [Dosovitskiy et al., 2020] has demonstrated that transformer architectures can achieve remarkable performance in computer vision tasks. VIT uses patch embedding as an input and multiple stacked transformer encoders as the backbone. Since then, a lot of research has been done to improve performance and efficiency by making changes to how images are stored [Parmar et al., 2018 , Chen et al., 2020 , Liu et al., 2021] , how attention is calculated [Shen et al., 2021 , Bolya et al., 2022] , and other methods [Touvron et al., 2021, Zhang and Sabuncu, 2020] . Meanwhile, some works have revealed that just employing MLPs as the token mixer can also lead to good performance [Tolstikhin et al., 2021 , Touvron et al., 2022 , Guo et al., 2022a] . Recently, a fresh perspective has been proposed by Metaformer [Yu et al., 2022] , suggesting that the success of Transformer in visual tasks stems from the holistic encoder architecture, where token mixer and channel mixer interact. Interestingly, Metaformer even achieved competitive performance by employing a simple pooling layer as the token mixer. Their findings offer us novel insights into how to modify Spikformer.The forward process of SNN encompasses the following rule: all high-precision matrix multiplications should be capable of degrading to matrix additions, eliminating the need for high-precision matrix multiplication operations after implanting in hardware [DeBole et al., 2019 , Davies et al., 2018b] . However, this rule restricts the flexibility of SNNs to incorporate various operations, limiting them to a few substructures such as the convolution layer, the fully connected layer, the pooling layer, and so on. The majority of SNN architectural designs are indeed based on these substructures, such as the advancements of ResNet structures like SEW-ResNet [Fang et al., 2021a] and MS-ResNet [Hu et al., 2024] , as well as the structure search efforts like AutoSNN [Na et al., 2022] and SNASNet [Kim et al., 2022] . Recently, Spikformer [Zhou et al., 2023b] has achieved remarkable success by integrating the transformer architecture into SNNs, attaining SOTA performance across various datasets. The authors proposed the SSA module to ingeniously realize attention computation in SNN while adhering to the aforementioned forward rule. At present, Spikformer and its subsequent series [Zhou et al., 2023a , Yao et al., 2024] of improvements are steadily pushing the boundaries of SNN performance. Nevertheless, though SSA exhibits notable performance in clock-driven hardware, it may encounter significant discrepancies in event-driven hardware due to the fact that the fact that the spikes may not arrive at the same time. 
spiking neural model: 
 The spike neuron serves as the fundamental unit of SNN, wherein their somatics harbor membrane potential that integrates information from the synaptic according to prescribed model principles. The Leaky integrate-and-fire (LIF) model [Lapicque, 1907] currently is the most prevalent SNN neuron model, which describes the membrane potential update rule as follows:EQUATIONwhere V [t] is the membrane potential, τ is the membrane time constant, S[t] is the output in time t, it would be equal to 1 or 0, which meas if there is a spike or not. Θ(•) means the Heaviside step function, which equals to 1 when H[t] exceed the threshold value V th . Furthermore, we employ the hard reset mechanism, which means the membrane potential will reset to the reset potential when the neuron fire a spike.We use the Spikingformer [Zhou et al., 2023a] architecture as the foundational framework of our network. Spikingformer addresses the issue of residual connections in Spikformer by using membrane potential residual instead of spiking residual. The overall structure of Spikingformer, as depicted in Fig. 1 (A), comprises three main components: Spiking Patch Splitting (SPS), an encoder module, and a classification head. The SPS consists of five convolutional layers, where the first four layers are followed by a max pooling layer. The function of SPS is to perform convolution processing and downsampling on input images, resulting in an output matrix with the shape D × h × w. Eventually, this matrix is flattened into N (N = h × w) patches, each with a D-dimensional vector. The classification head consists of an adaptive average pooling layer, which is used for downsampling, and a fully connected layer, responsible for obtaining the network's final output. The classification head will introduce floating-point matrix multiplication, so it needs to be deployed on the CPU.Because the calculation amount of the classification head is very small, it will not cause much energy overhead or inference delay but will greatly improve SNN performance [Zhou et al., 2023a] .The most important component of Spikformer is its encoder module, which consists of two cells and residual connections (Fig. 1 (A) ). Cell 2 corresponds to the Feed Forward Network (FFN) in VIT. It consists of two fully connected layers designed for channel mixing. Cell 1 refers to the Spiking Self-Attention (SSA) module proposed by the Spikformer. SSA has been demonstrated to be more suitable for SNN compared to vanilla self-attention (VSA). In SSA , when the input isX ∈ R N ×D , the query (Q ∈ R N ×D ), key (K ∈ R N ×D), and value (V ∈ R N ×D ) are computed using the following equations:EQUATIONwhere W Q , W K , W V ∈ R D×D are learnable weight matrices, SN means spiking neural (LIF model) activation, and BN denotes the batch normalization layer. The authors of Spikformer argue that in ANN, the presence of outliers or negative values in the query and key makes the attention map require a softmax function. However, since the output of eqn. 2 is a binary matrix that only consists of 0 and 1, there are no outliers or negative values. Therefore, they directly perform matrix multiplication on the three matrices to obtain the output of SSA:EQUATIONwhere s = 0.125 is a constant factor.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: We provide the explanation of why the multi-head STM module is effective from the perspective of fitting the attention map, and explaion the effectiveness of IPSPS from the perspective of informantion loss. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
event-driven scenarios: 
 In ideal asynchronous scenarios, each neuron is an autonomous entity solely focused on updating its membrane potential upon receiving spike signals, then firing a spike once the membrane potential exceeds the threshold. Since the model in asynchronous scenarios lacks the specific hardware clock to control the spike firing and arriving time, it is inevitable to encounter temporal errors in the spike arrival time. In the extreme case, it is conceivable to assume that each neuron fires spikes at an individual time (the SNN time step T approaches infinity, and each spike fires at an individual time step). When there are errors in the arrival time of spikes (t = t + σ), the output sum of asynchronous scenarios supported operations like convolution, fully connected layer, and average pooling will not change. However, some operations that are commonly observed in synchronous scenarios may have output errors due to input spike arrival timing error.Maxpool layer. As shown in Fig. 1 (B), a spike at a specific location arrives one time step later in the max pooling layer, resulting in a noticeable change in the output. This output error will further affect the result of the subsequent convolution layer, as well as the membrane potential accumulation of neurons and their output spikes. In the extreme case where each spike has an independent arrival time, all the spikes will pass through the max pooling layer. Therefore, max pooling layer needs a clock to control the spike arrival time and cannot be simply deployed in event-driven scenarios. To address this problem, we can simply remove the max pooling layer and change the stride of the convolutional layer following the max pooling layer to 2 for downsampling. After this modification, when the arrival spikes have temporal discrepancies, the cumulative membrane potential of the post-convolution spike neurons will remain relatively stable.Spike matrix multiplication. On the synchronization and clock-driven hardware, the query, key, and value spike matrices can arrive on time. This guarantees that the forward SSA results on this hardware correspond to the GPU simulation results. However, on event-driven hardware, the absence of clock constraints often makes it challenging for spikes to arrive at precise timings. If there is any spike loss or similar occurrences during matrix multiplication in SSA, a significant error in the SSA output will occur. Here is a simple example that shows how the computed attention map matrix changes significantly in SSA when there is a spike delay at a certain point in both the query and key matrices (Fig. 1 (C)). This example shows how hard it is to make sure that the forward process with two spike matrix multiplications runs smoothly on event-driven neuromorphic hardware.According to Rathi and Roy [2020] , Hu et al. [2024] , Zhou et al. [2023a] , The theoretical computational energy consumption of SNNs on Clock-driven Scenarios can be defined as:EQUATIONwhere E AC = 0.9pJ and E M AC = 4.6pJ are the assumed energy consumption of accumulate and multiply-and-accumulate operations respectively of 45nm CMOS hardware [Horowitz, 2014] , SOP l Conv denotes the number of spike-based accmulate operations, F LOP 1 Conv means the computation of first layer that encoding static RGB images into spike matrix [Zhou et al., 2023a] , F LOP L Linear means the computation of the classification head. And SOP l Conv = F LOP l Conv • f r l • T is effected by the SNN fire rate f r l and time step T . However, recent works [Dampfhoffer et al., 2022 , Bhattacharjee et al., 2024] have pointed out that Eqn. 4.2 neglects the overhead of memory scheduling E M em , which is much lager than E M AC . On each time step of SNN forward inference, hardware will deal with memory scheduling, so E M AC is also affected by the time step T . And SNNs with a large number of time steps may have a large energy overhead in synchronous scenarios, even potentially surpassing that of ANNs under the same structure. Therefore, we must closely examine the performance of the new SNN architecture at extremely low time steps to ensure its adaptability to low-power scenarios. To address the aforementioned requirements in the two scenarios, we present the STMixer network architecture. STMixer solely consists of asynchronously supported operations and performs well at extremely low time steps (T = 1). STMixer has the same overall structure as Spikformer but uses STM as the token mixing module and IPSPS to replace the SPS module (Fig. 1 (A) ). STMixer is controlled by three hyper-parameters: L, D, and H, which respectively control the number of encoders, the encoding dimension, and the number of multi-heads of STM. We use the naming convention "STMixer-L-D-H" to represent the three parameters, which is similar to Spikformer-like works. Next, we will go into detail about the important modules in STMixer.As shown in Fig. 2 (A), we use the STM module to realize the token mixing function. This module utilizes a weight matrix (W STM ∈ R N ×N ) to mix the token dimention information. Mathematically, its forward process can be expressed as follows:EQUATIONWhere X ∈ R N ×D is the input spiking matrix of STM, V ∈ R N ×D is the value matrix, and W V ∈ R D×D is the projection matrix to aquire value matrix V . When we simplify the operations of the SSA module by removing the activations for query and key, the formula for SSA can be simplified as follows:EQUATIONAfter this simplification, W Q and W K can converge into a single matrix, and the attention map is reduced to XW X T . So the objective of W STM is to gradually fit the attention map during the training process. And some studies also suggest that linear token mixing can capture external attention between samples in the dataset [Guo et al., 2022a] .Furthermore, we evenly divide the value matrix V ∈ R N ×D into H groups based on dimension D, allocating a weight matrix W h STM ∈ R N ×N to each group. This method employs different attention weights for various patch encoding groups V h ∈ R N ×D/H . It facilitates a finer granularity in token mixing, leading to enhanced information mixing and integration among tokens. In addition, due to the relatively modest number of tokens in the vision task, implementing multi-head in this manner would not significantly increase the parameters and maintain the same level of computational complexity.In a neural network, as the input X undergoes successive forward propagation layers, the inherent information gradually dissipates. mathematically this phenomenon can be described as:EQUATIONwhere I means the mutual information, f and g is the operation of network layers, and Y is the network output. A SNN layer may lose more information than an ANN layer because the layer output is a spike matrix, which inherently contains significantly less information than a floating-point matrix.The Spiking Patch Splitting (SPS) module comprises five layers of convolutional operations and We present the feature maps of the 100th channel and the average of all channels, and in SNN, the feature map is the mean feature map across all time steps. When using the LIF neuron model as the activation function for SPS, it is obvious that there is a significant information loss in the output feature map compared to the input image. Employing the IPSPS can alleviate this information loss.spiking neural activation. After passing through the total of five convolutional layers and spiking neural, the feature map may have already lost a significant amount of information contained in the input images (Fig. 3 ).Since Spikformer-like SNNs rely on convolutional layers in the SPS module for information extraction, reducing the number of layers in the SPS module will compromise the SNN's ultimate performance.To address this information loss problem, we add an additional path that directly uses a convolutional layer for input image encoding (Fig. 2 (B)). In our configuration, the encoding outputs from the auxiliary pathway account for 1/8 of the entire SPS output feature map. This new encoding module is called information protected spiking patch splitting (IPSPS).There are two advantages to IPSPS: (1) IPSPS does not reduce the convolutional layers in the original pathway, it only decreases the number of feature map channels by 1/8 for the last convolutional layer. Theoretically,this approach should not significantly impair the SNN performance. Simultaneously, the new pathway can provide feature maps with varying convolutional depths for subsequent encoders, enriching the SPS output's feature representation.(2) The output of the additional pathway in IPSPS can be regarded as a linear transformation applied to the input image. The input feature map for the first encoder from the additional pathway is essentially the linearly transformed input image passed through a SNN neural layer, which maximally preserves the input image's information.Even if we reduce the information loss of the encoder's input, the accumulation of gradient errors caused by surrogate gradient may still prevent SNN from achieving better performance with low time steps. As a result, we use the surrogate module learning (SML) method [Deng et al., 2023] . The SML is an exceptionally effective approach proposed to alleviate the issue of gradient errors in SNN by adding an auxiliary pathway during the training phase. However, the use of large kernel convolutions results in significant computational overhead during the training phase. In this paper, we employ the STMixer encoder to replace the computationally intensive convolutional encoder, thereby greatly reducing the computational overhead of SML. The STM in the surrogate module uses the GeLU activation function and employs the same number of heads as the STM in the main structure. Finally, we use a classification head for the output. Starting from index 0, we place a surrogate module after every two encoders for surrogate module learning. And the number of encoders in the surrogate module is half of the count of encoders after the place point in the main pathway.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: All the experiments in this paper use the open-source dataset. And we have provided the code that reproduces the results of the paper in the supplementary.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
comparison to exiting works: 
 CIFAR. In IPSPS, we only set the stride to 2 for the last two convolutional layers, similar to Spikformer. This implies that IPSPS downsamples the input images to a resolution of 8×8 and subsequently obtains 64 patches through flattening. Here, STM in STMixer uses 32 heads. And we use the same training script as Spikeformer, the learning rate is set to 0.005, the batch size is 128, and training total 300 epochs. As shown in Tab. 1, STMixer achieves state-of-the-art performance both on the CIFAR-100 dataset and considering models with comparable parameter numbers and flops. However, on the CIFAR-10 dataset, the SML method slightly outperforms the results reported by us. This could be attributed to the larger parameter count of the structure they used or the structure being more suitable for the CIFAR-10 dataset. These results demonstrate the potential of STMixer to achieve excellent performance for SNN on simple static datasets, even with only one time step. It also suggests that STMixer can serve as a competitive alternative to Spikformer-like works.ImageNet In this section, we endeavor to validate the effect of different encoder structures (the structures of Cell 1 and Cell 2 in Fig. 1 ). Additionally, we also want to figure out whether the metaformer-like structure is a superior architecture in SNN. As depicted in Fig. 4 , each cell consists of three positions, with each position having five possible structures (STM and SSA only use a single head). Disregarding any duplicates generated by the identity structure, there are a total of 5 6 different substructures.Conducting a comprehensive validation of all these cases is impractical. Inspired by the concept of a one-shot NAS, we train a supernet that encompasses all possible substructures. The supernet is composed of three main components: the SPS, the classification head, and four identical encoders. into training and validation sets in a ratio of 8.5:1.5. During each forward pass, the supernet will randomly select a path and train on the training set for 600 epochs. After the training is completed, we randomly sample 4000 distinct subnetworks from the supernet and evaluate their performance on the validation set. Additionally, we calculate the forward FLOPs for each of these subnetworks. Note that the FLOPs mentioned here refer to the estimated floating point operations on the training platform. The actual oretical synaptic operations (SOPs) rely on FLOPs, the simulation time step, and spike frequency. We have categorized the subnetwork into four classes: (1) only convolutional layers;(2) combining SSA and convolutional layers; (3) incorporating STMixer and convolutional layers; and (4) simultaneously having STMixer and SSA. To draw ellipses, we use the mean and variance of accuracy and FLOPs, showcasing the relative merits of four different selection spaces. Furthermore, we have marked the positions of STMixer and Spikingformer. As shown in Fig. 4 , considering the computational complexity and network performance, as well as the suitability of the components used in asynchronous scenarios, STMixer appears to be the most promising architecture (STMixer is located at the top-left corner of the result figure).To understand the effect of various components and training methods in STMixer, we conducted ablation experiments on CIFAR-100 with STMixer-4-384-16 (T = 1). The summarized results are presented in Table 2 . As we can see, SML is the most significant improvement method. Considering that STMixer-4-384-16 is indeed a 25-layer network, the training process affects the nonnegligible gradient error accumulation. SML is an effective approach to mitigating the gradient error problem. And the employment of the modified SML algorithm with STM modules enables a more efficient facilitation of training SNNs (+0.48%). Then, under the SML training method, the performance of the STM module surpasses that of the SSA module (+1.22%). Finally, by preserving more information from the input images by IPSPS, the SNN achieves better performance (+0.58%).Another parameter that significantly influences the STMixer performance is H, the number of heads in STM. Here, we increment the value of H from 1 to 32 and evaluate the performance of STMixer-4-384 on the CIFAR-100 dataset. As the number of heads (H) increases, the STMixer accuracy on the test set has improved from 79.02% to 80.15%. This observation highlights the necessity of performing token mixing at a finer granularity.Besides the superior performance demonstrated on the aforementioned static datasets, our proposed STMixer model also exhibits significant adaptability for event-driven scenarios. In these scenarios, SNNs are deployed on asynchronous chips to receive and process event streams from DVS cameras, thereby forming a system that integrates sensing and computation. This integration leverages the sparse nature of spike activity, allowing SNNs to achieve ultra-low energy consumption compared to their ANN counterparts. To validate this advantage, we conducted additional experiments on fully asynchronous frameworks. Since the current commercial asynchronous chips (e.g., Speck of the Synsense [Richter et al., 2024] and Loihi [Davies et al., 2018a , Lines et al., 2018] of the Intel) are limited in capacities and lack support for various operators such as shortcuts, we tested our models on a home-made C++ simulator that reproduces the computational principles of the Speck DevKits (see Synsense Sinabs). We compared STMixer to a structurally identical SpikFormer, created by replacing the STM component in STMixer with SSA on CIFAR10-DVS. Due to the prohibition of clock-driven bias in a fully event-driven scenario, we implemented a two-phase training pipeline: initially training a standard SNN, then absorbing BN layers and eliminating all bias terms through additional fine-tuning, while the side effect is a slightly performance degradation (see the w/ bias column and T=25(40) column in Table . 4). The training of SpikFormer becomes extremely unstable during the bias-removal phase when T=40 and fails to converge even with the learning rate reduced to 10 -5 .We first deployed the two models on our event-driven simulator and evaluated their test accuracy. As shown in Table . 4) event-driven column, SpikFormer undergoes significant performance degradation after deployment because events fail to meet each other and end in SSA, whereas STMixer maintains high performance and demonstrates strong compatibility with event-driven framework. We further assessed their event-driven adaptability by inference performance at large T. When T is considerably large, events rarely coincide within the same frame, effectively approximating an event-driven scenario. According to Table. 4), as T increases, SpikFormer's performance deteriorates sharply, whereas STMixer exhibits only a modest accuracy decrease. These results are consistent with our analysis in Section 4.1. Further details are provided in the appendix.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: we have provided the code that reproduces the results of the paper in the supplementary.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 The advantage of spiking neural networks over artificial neural networks is their efficient and lowpower inferencing capabilities once implanted in neuromorphic hardware. Within neuromorphic hardware, synchronous hardware uses a hardware clock to control when the spikes fire and arrive.The power consumption of such hardware is influenced by the time step length (T), whereby a smaller T typically results in lower power consumption. Asynchronous hardware has a lower energy consumption potential than synchronous hardware, but it has limitations on the network structure and only supports some specific matrix operations due to the lack of the hardware clock. For the aforementioned reasons, SNN must use asynchronously supported operators to build new architectures (for asynchronous hardware) or complete the performance challenge with an extremely low time step (for synchronous hardware). In this paper, we propose a new architecture named STMixer that satisfies the above challenges. STMixer is completely composed of asynchronous scenario supported components, including convolutional layers, fully connected layers, and residual paths, and has excellent performance when the time step T = 1. Our work will provide ideas for the SNN architecture, which works on both event-driven friendly and clock-driven efficient SNN.  
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: In comparing the results with existing work, in addition to the Imagenet results, we provide the mean and standard deviation of the results of three different seed experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide the above information in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The research with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We have discussed it in the conclusion part.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Here we show the token mixing attention weights of the encoders from the well-trained STMixer-8-768-16 on the ImageNet dataset. These weights provide insights into the blending process of tokens, revealing how they are mixing information. As shown in Fig. 5 , most of the weight matrices exhibit a pronounced regularity: there are some parallel lines near the symmetry axis. This observation implies two insights. While the attention weight of the STM enables the extraction of global information, it still tends to favor the extraction of local information. In visual tasks, the correlation between each patch and its neighboring patches is maximized. Moreover, it can be observed that the 16 attention weight matrices in each row are different, indicating that each head of the multi-head STM is capable of extracting distinct token mixing patterns.Training for comparison with existing works All experiments on CIFAR were conducted on two 4090 GPUs, while experiments on ImageNet were performed on four 4090 GPUs. We employed the same training script as Spikingformer [Zhou et al., 2023a] and employed identical data augmentation techniques. In the CIFAR experiments, we modify the learninig rate to be 5e-3 and the minimum learning rate of 1e-7. In the ImageNet experiments, we modify the learninig rate to be 3e-4 and the minimum learning rate of 1e-5.Training for event-driven experiments Models for event-driven experiments are all trained by our two-phase training pipeline. Specifically, we first train a SNN with bias and BN layers. Before further fine-tuning, we absorb BN layers and set all bias terms to 0. Finally, we fine-tune the model for more epochs with bias frozen. For the first phase, we use AdamW optimizer and set learning rate to 0.001, weight decay to 0.02. For the second phase, we use AdamW optimizer and set learning rate to 0.001, weight decay to 0.005. For SpikFormer and STMixer with T=25, a time step of T=10 is used for the first phase over 300 epochs. T is then set to 25 for the bias-removal phase, during which models are fine-tuned for an additional 30 epochs. For SpikFormer and STMixer with T=40, a consistent T=40 is used throughout both phases. Models are trained for 100 epochs during the first phase, followed by 30 epochs of fine-tuning during the bias-removal phase. Notably, we found that MultiSpike IF neurons with soft reset (see Synsense Sinabs) are optimal for training models intended for event-driven deployment, and this configuration was applied in all event-driven experiments.Backbone for event-driven experiments We employed the modified STMixer-2-256-8 as the backbone for all event-driven experiments, incorporating several adjustments to enhance its suitability for fully event-driven deployment. First, the non-spiking membrane shortcut was replaced with a standard spiking shortcut [Zheng et al., 2021] for event-driven implementation. Second, the first two convolution layers of SPS module are replaced with a single 4 × 4 Avgpool layer. Building upon this foundation, we replace the STM module with SSA to construct the SpikFormer model.B.2 Compare with existing works on neuromorphic datasets in synchronous scenarios CIFAR10-DVS is a neuromporphic dataset that is obtained from the CIFAR-10 dataset through a DVS camera. There are 10k images in CIFAR10-DVS, and we split them into 9k training images and 1k test images. We downsample the image resolution from 128×128 to 48×48 and input them to STMixer-2-512-8 structure SNN. We don't use IPSPS module here, because we find it not enhance the performance on neuromporphic dataset too much. For the training setting, we employ the AdamW optimizer with a learning rate of 0.01 and a weight decay of 0.02. The training is conducted for 300 epochs. Meanwhile, we employ the same data augmentation methods as previous works Li et al.[2021], Deng et al. [2021] As depicted in Table . 5, our architecture achieves the best performance on CIFAR10-DVS, demonstrating the suitability of STMixer for this neuromorphic dataset.DVS128 Gesture Amir et al. [2017] is a gesture recognition dataset that consists of 11 hand gesture classes performed by 29 individuals under 3 different lighting conditions. We employ the network architecture STMixer-2-512-8 and apply the same augmentation techniques from Spikingformer except cutmix. We don't use IPSPS module here, because we find it not enhance the performance on neuromporphic dataset too much. We train the network for 200 epochs with TET loss Deng et al.[2021] and the SML method. STMixer also achieves the best performance on this dataset.Justification: In the 4.2 Clock-driven Scenarios section, we mentioned that the current SNN energy cost calculation equation is incorrect and ignores the cost of memory calls. Furthermore, the real energy cost is still proportional to the time step, so SNN needs to use a lower time step to reduce potential energy consumption. Constructing an accurate energy consumption calculation equation is beyond the scope of this paper. In addition, due to current asynchronous hardware scale limitations (e.g., Speck 2e only supports 9-layer SNNs), we cannot implant large-scale SNNs into the hardware for testing. Therefore, we can only theoretically build models using the asynchronous hardware supported operators. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: The authors argue that certain operators used in Transformer implementations are not very suitable for asynchronous spiking based neuromorphic chips. The authors make an attempt to provide ways so that the self-attention module can be implemented in a way suitable for asynchronous chips. The reviewers agree that that the authors are focusing on a real problem that limits the applicability of transformers on asynchronous chips.

Among the weaknesses listed are presentation issues (missing or incorrect references and notational issues making some parts of the paper feel a bit unprofessional). Another weakness is that the authors seem to make claims on the suitability of the algorithm for implementation on neuromorphic hardware, yet they do not seem to take into account issues that may hinder the adoption of such an algorithm on actual hardware. These can include issues that are specific to each hardware implementation, such as the fan-out supported by the hardware's neurons or the spike rate supported. 

Nevertheless there is consensus among the reviewers that the merits of the paper outweigh the drawbacks, and the authors seem to have answered the reviewer questions to their satisfaction. The authors should fix all the presentation issues and the references issues mentioned by the reviewers before publication.

Reviews: review: paper_summary: paper_summary: This article examines problems with the SSA module in Spikformer in asynchronous scenarios and suggest a new module, the Spiking Token Mixing (STM) module, which consists solely of network components that cater to asynchronous environments. Besides, This article proposed the information protection spiking patch splitting (IPSPS) module to reduce information loss.
strength_weakness: strength_weakness: 1.From your code and Table 2 (SML->SDT). The performance improvement mainly comes from several SML blocks, which consist of several ANN layers. When considering adding residual connection and several SML blocks, the STMixer can even be seen as an ANN model (floats input is connected to the last layer through intermediate hidden ANN ). In my opinion, this is a very fatal question.
2. From the perspective of solving the asynchronous scenarios problem, the residual connection is also a problem, residual connections in the pre-activation shortcut are floats point element-wise addition, which also suffers the same problem in maxpooling and spiking matrix multiplication.
questions: questions: 1. The training cost (memory consumption and training time) of STM, SML, and STMixer should be reported compared with previous work.
2. The energy consumption of the SML block needs discussion. 
3. A complete comparison of STM and SSA need to be carried out (report the results of replacing SSA in Spikingformer or Spikformer by STM block.).
limitations: limitations: See weaknesses and questions.
paper_summary: paper_summary: Most of the spiking neural network architectures can not truely show the superiority on the neuromporphic hardware, since in event-driven scenarios, the spike arrival times are not  precise and could result in significant differences in the output, like there is a max pooling layer. This paper propose the Spiking Token Mixer, which can well handle the problem. The authors also have validated that STMixer could achieve performance on par with or even surpasses existing Spikformer-like works in synchronous scenarios.
strength_weakness: strength_weakness: 1. Does the proposed method increase the energy consumption? The authors could provided detailed explanations.

2. The ablation experiments could include different types of datasets and models.

3. The comparison methods in the experiments contrasting with other approaches on CIFAR-10 may not be the most up-to-date.
questions: questions: 1.  Can you further  explain the meaning of the sentence in first paragraph of Introduction that "due to the accumulation of membrane potential in the IF neuron, spike arrival timing errors do not significantly impact subsequent layers.".
2. Could you provided a detailed explanation for the method?
limitations: limitations: None
paper_summary: paper_summary: To address the issue where certain operators (e.g., spiking self-attention, SSA) in existing spiking Transformers cannot be executed on asynchronous neuromorphic chips, this work designs the Spiking Token Mixer (STMixer) architecture, which consists exclusively of operations supported by asynchronous scenarios including convolutional, fully connected layers, and residual paths.
strength_weakness: strength_weakness: 1. The technical contribution is limited; there is no in-depth analysis of the proposed operators, and some existing work in the field is overlooked. 
2. Some key points are not clearly explained, especially the section on operators.
3. The work does not follow the latest results in the field, e.g., meta-spikeformer [1], and the performance is not SOTA.

[1] Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. In ICLR 2024.
questions: questions: 1. Some references are incorrect and cannot be accessed. LInes 91-92, 255

2. The mathematical descriptions in Section 4.3 are not rigorous and omit some crucial information. Is the matrix X composed of continuous values or is it a spiking matrix? All the dimensional changes are ignored, which is unprofessional. You cannot expect every reader to read all your references.

3. Could the authors explain the proposed STM from the perspective of linear attention? Since the LIF following Q and K functions as a kernel, why can the spiking neuron layer be directly removed in this context?

4. SML training seems crucial, but other baseline works do not appear to use this method. Therefore, is the comparison in Table 1 unfair? Why is there no comparison with Spike-driven Transformer v1/v2 in Table 1?

5. What is SDT in Table 2? It is mentioned only once in the entire text and is not explained.

6. I suggest the authors carefully check the references. If the paper has been accepted, please cite the official version instead of the arXiv version, as this is an academic convention.
limitations: limitations: N/A

Comments: title: Good rebuttal
comment: I would like to express our sincere gratitude for your prompt response, for taking the time to review our rebuttal, and for increasing the score.
Thank you for addressing my concerns. I have updated the score.
Thank you for your insightful comments and your appreciation of our work.
Thank you for your constructive feedback and for the updated score.
Dear Reviewer,

I hope this message finds you well. We greatly appreciate the time and effort you’ve already invested in reviewing our work, and we truly value your comments. We are writing to kindly follow up on the rebuttal we submitted and sincerely hope you could review it and update the scores if your concerns have been resolved. If there are any additional points you would like us to address, we would be more than happy to provide further clarification.

Thank you once again for your time and consideration.

Best regards,

Authors
Thank you for your response. The rebuttal addresses my concerns. "What kind of SNN can be supported by neuromorphic chips" is actually always ignored by people in the field. The authors' questions and solutions are insightful. I will raise my score.
Thanks for your response. My concerns have been well addressed.