PSSD-Transformer: Powerful Sparse Spike-Driven Transformer for Image Semantic Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-20 09:05:02

Abstract: Spiking Neural Networks (SNNs) have indeed shown remarkable promise in the field of computer vision, emerging as a low-energy alternative to traditional Artificial Neural Networks (ANNs). However, SNNs also face several challenges: \romannumeral1) Existing SNNs are not purely additive and involve a substantial amount of floating-point computations, which contradicts the original design intention of adapting to neuromorphic chips; \romannumeral2) The incorrect positioning of convolutional and pooling layers relative to spiking layers leads to reduced accuracy; \romannumeral3) Leaky Integrate-and-Fire (LIF) neurons have limited capability in representing local information, which is disadvantageous for downstream visual tasks like semantic segmentation. \par
To address the challenges in SNNs, \romannumeral1) we introduce Pure Sparse Self Attention (PSSA) and Dynamic Spiking Membrane Shortcut (DSMS), combining them to tackle the issue of floating-point computations; \romannumeral2) the Spiking Precise Gradient downsampling (SPG-down) method is proposed for accurate gradient transmission; \romannumeral3) the Group-LIF neuron concept is introduced to ensure LIF neurons' capability in representing local information both horizontally and vertically, enhancing their applicability in semantic segmentation tasks. Ultimately, these three solutions are integrated into the Powerful Sparse-Spike-Driven Transformer (PSSD-Transformer), effectively handling semantic segmentation tasks and addressing the challenges inherent in Spiking Neural Networks. The experimental results demonstrate that our model outperforms previous results on standard classification datasets and also shows commendable performance on semantic segmentation datasets. The code will be made publicly available after the paper is accepted for publication.

Main paper: 
 applications.: 
 Implementing above techniques has led to significant results in various fields. For instance, object detection can be achieved through Spiking-Yolo [20] and ems-yolo [37] . In terms of large language models, SpikingGPT [49] and SpikingBert [2] are preferable, and for generative models, SpikingGAN [22] is the way to go. SpikingGCN [50] and SpikingGAT [40] are recommended for Graph Neural Networks. Furthermore, neuromorphic chips such as TrueNorth [28] , Loihi [10] , and Tianjic [31] are now available, making it increasingly likely that SNNs will be widely used in the near future.2.2.1 Traditional semantic segmentation. This section primarily introduces semantic segmentation algorithms based on deep neural networks, categorizing them into four main representative types: i) Fully Convolutional Network (FCN) [26] , by replacing fully connected layers with convolutional layers and adopting an encoderdecoder architecture, facilitates the extraction of deep discriminative features for subsequent instance localization and segmentation tasks. However, the computational demands on high-resolution images may result in challenges such as insufficient memory and slower processing speed, while exhibiting a relatively weaker capability in handling detailed boundary information. ii) SegNet [1] employs a pooling index mechanism for upsampling its lowerresolution input feature maps, aiming to accurately reconstruct object contours during the decoding process. However, when dealing with large-sized images, it may encounter significant computational and memory overhead. iii) DeepLab [7] utilizes Dilated Convolution to expand the receptive field, enhancing segmentation performance by capturing contextual information more effectively. iv) strategies based on Visual Transformer (ViT) [13] : SETR [47] achieves pixel-level segmentation through a multi-level feature aggregation module; Segmenter [36] obtains class labels and predicts segmentation masks through a point-wise linear decoder or a Mask Transformer decoder; Segformer [43] employs a simple yet efficient MLP decoder to aggregate information from different layers, cleverly combining local attention and global attention to present powerful representations; PVT overcomes the low-resolution output features of a single scale by introducing a progressively contracting pyramid network backbone.Unlike traditional counterpart, sparse networks commonly employ lightweight architectures to minimize parameter count and computational load, a critical consideration for deploying semantic segmentation models on resourceconstrained devices. ERFNet [32] is a lightweight network designed for real-time semantic segmentation, employing residual connections and decomposed convolutions to reduce computational load, suitable for embedded systems. SGCPNet [17] proposes a strategy for spatial detail-guided context propagation, achieving sparsity by avoiding the maintenance of high-resolution features throughout the network. Other strategies like sparse self-attention [18] , which decomposing the dense affinity matrix into the product of two sparse matrices. Researchers also employ SNNs to achieve sparsity.SpikeCalib [25] utilizes FCN for ANN to SNN conversion in semantic segmentation, introducing Burst-Spikes neuron model and proposing Lateral Inhibition Pooling (LIPooling) to address errors from max-pooling during the conversion process. SpikeSEG [21] is the first algorithm for semantic segmentation that directly trains SNNs using spike events with the STDP method, but it is effective primarily on simpler datasets. Our work addresses the gap in SNNs' capability for semantic segmentation in complex scenarios. 
preliminaries: 
 In this section, we first introduce Leaky Integrate-and-Fire (LIF) neurons, widely applied within the SNN domain. Following this, we explore traditional attention mechanisms and those already present in SNNs. Then, we introduce existing residual connections in SNNs, analyzing their advantages and limitations. Finally, we discuss the mainstream downsampling operations in SNNs and analyze their shortcomings.In SNNs, spike neurons control the release of spikes based on a threshold. In this paper, we use LIF [12] neurons, which work in the following way:EQUATIONEQUATIONEQUATIONwhere EQUATIONhere, 𝑊 𝑄 ,𝑊 𝐾 ,𝑊 𝑉 are learnable weight matrices.Next, attention weights are computed. The attention weight between positions 𝑖 and 𝑗 is calculated using dot product: These components respectively yield a decoded semantic segmentation feature matrix and a category feature embedding. By performing an inner product of these two feature matrices, the model generates K mask sequences for semantic segmentation, where K represents the number of semantic categories. Finally, a bilinear interpolation upsampling is applied to these sequences to match the original image size, resulting in the semantic segmentation output.EQUATIONEach output element, 𝑧 𝑖 , is computed as weighted sum of a linearly transformed input elements:EQUATION3.2.2 Spiking Self-Attention in Spikformer. In Spiking Self-Attention (SSA) mechanism, the input undergoes linear transformations and neuron operations to be transformed into spike-form queries (Q), keys (K), and values (V) containing only 0 and 1. The computed attention map has natural non-negativity, disregarding irrelevant features. Therefore, using floating-point representations for Q, K, V, and the softmax function is redundant for modeling such spike sequences.EQUATIONEQUATIONEQUATION𝑆𝑆𝐴(𝑄, 𝐾, 𝑉 ) = 𝑆𝑁 (EQUATION)where 𝑆𝑁 𝑄 , 𝑆𝑁 𝐾 , 𝑆𝑁 𝑉 and 𝑊 𝑄 ,𝑊 𝐾 ,𝑊 𝑉 represent Spiking Neurons and weights of the Q, K, V branches, respectively. 𝑄, 𝐾, 𝑉 ∈ R 𝑇 ×𝑁 ×𝑑 , in this context, T denotes the time-step, N denotes the length of the sequence, and d denotes the dimension. BN denotes the Batch Normalization layer, and X is the input of the SSA. s represents the scale factor.The Membrane Shortcut (MS) has been demonstrated to satisfy the dynamic isometry theory. It adds a shortcut connection to the membrane potential of each spiking neuron, achieving an identity mapping. This ensures that the tensor of spike transmission always contains binary pulse signals.EQUATION𝐹 𝑟𝑒𝑠 (𝑋 𝐿-1 ) = 𝐶𝑜𝑛𝑣𝐵𝑁 (𝑆𝑁 (𝑋 𝐿-1 ))where 𝑋 𝐿 represents the output of the layer L, 𝐹 𝑟𝑒𝑠 (•) denotes the residual path, 𝐹 𝑠ℎ𝑜𝑟𝑡 (•) denotes the shortcut path, 𝐶𝑜𝑛𝑣𝐵𝑁 (•) represents the operation combining Convolution and Batch Normalization. However, the MS residual connection method still has some limitations. When changing the channel number to match the feature channel number on the residual path and the shortcut path, it may easily overlook non-sparse and non-spike-driven operations in this part. Non-sparse spike operations can lead to significant energy consumption in this module.Input undergoes a combination of convolution and batch normalization operations (ConvBN), followed by activation of spiking neurons over multiple time steps. Neurons accumulate membrane potential over time based on the received current, generating a spike signal when the membrane potential exceeds a threshold. Finally, max-pooling is performed for downsampling.The backward propagation stage of the above process can be expressed by Eq.13-15:EQUATIONEQUATION)EQUATIONwhereℎ ∈ R 𝐻 ×𝑊 , 𝑥 ∈ R 𝐻 ×𝑊 , 𝑦 ∈ R 𝐻 𝑠 × 𝑊𝑠 denote the output of the SN, ConvBN and Max Pooling layer, respectively, ℎ 𝑢𝑣 , 𝑥 𝑢𝑣 , 𝑦 𝑖 𝑗 denote the represent the feature information at a specific position for feature maps ℎ, 𝑥, 𝑦. 𝑠 is the pooling step. 𝐿 represents the loss function. 𝛼 is a parameter designed to ensure the integration of the gradient equals 1 and determines the steepness of the curve. 
method: 
 In Sec.3, we refine our methods to better address semantic segmentation challenges. In this section, we start with the introduction of Group-LIF, enhancing local information processing in LIF neurons. We then present PSSA, a sparser alternative to SSA, ideal for asynchronous neural chips, with parameter integration into thresholds to minimize float count. DSMS adapts to the dynamic channel variations in semantic segmentation, extracting features from varying feature map dimensions and ensuring MS pulse-driven functionality. Additionally, SPGD is introduced to ensure accurate pulse-driven gradient backpropagation essential in downsampling operations within semantic segmentation tasks.Global LIF neurons introduce weak long-range correlations due to the larger time-steps introduced. In semantic segmentation tasks, it is crucial to maintain spatial consistency, including the interaction of contextual information and local details in the image.To address the aforementioned issues, we proposes Group-LIF, based on a grouping mechanism. It divides the patch sequence of the feature map into several groups and applies Group-LIF neurons to facilitate information interaction within each group. This enables a more compact pattern of local context information exchange.As shown in Fig. 2 (b), G-LIF attempts to transmit information from both horizontal and vertical directions, that is, by learning local contexts and spatial information in different orientations on the image through horizontal LIF and vertical LIF for different communication directions of image blocks. Ablation study for G-LIF is presented in Sec.5. Due to space limitations, details will be provided in the supplementary materials.The SSA still involves non-spiking computations, making it challenging to deploy the model on neuromorphic chips. To avoid nonspiking computations and energy-consuming operations like multiplication and addition, it is essential to leverage energy-efficient operations such as logical AND, addition, masking, which require minimal energy consumption.Lemma 1. Transforming matrix dot product using element-wise multiplication and column summation operations incurs almost negligible energy cost.EQUATION)EQUATIONwhere 𝐴, 𝐵 ∈ R 𝑁 ×𝑑 , 𝑎 = 𝐴 𝑖1 and 𝑏 = 𝐵 𝑖2 , (•) 𝑇 denotes the matrix transformation, ⊙ represents the matrix multiplication, ⊗ is the hadamard product.As shown in Eq.16, when performing the dot product of the transposed binary column vectors 𝑎 and 𝑏 with dimensions 𝑁 × 1, the resulting dimension is 1 × 1. This is equivalent to taking the Hadamard product of column vectors 𝑎 and 𝑏 to obtain an 𝑁 × 1 tensor, followed by column-wise summation to get a 1 × 1 vector.As shown in Eq.17, generalizing to the dot product of all columns of matrices A and B, it can be easily transformed using element-wise multiplication and column summation operations. Moreover, the spiking-driven Hadamard product can be considered as a logical bitwise AND operation with almost negligible energy cost, resulting in a 1 × 𝑑 dimensional matrix.Extending this theoretical discovery to the dot product calculation of three binary matrices in the spiking self-attention mechanism, this process can be equivalently substituted by spiking-driven computations for the 𝑄, 𝐾, 𝑉 calculation steps in the SSA mechanism.𝑄 𝑠 ⊙ 𝐾 𝑇 𝑠 ⊙ 𝑉 𝑠 = 𝑆𝑁 ( 𝑁 ,𝑑 ∑︁ 𝑖=1,𝑗=1 𝑄 𝑠 ⊗ 𝐾 𝑠 ) ⊙ 𝑉 (18) = 𝑆𝑁 (𝑆𝑈 𝑀 𝑟 (𝑄 𝑠 ⊗ 𝐾 𝑠 )) ⊕ 𝑉 𝑠where ⊕ denotes the mask operation, 𝑄 𝑠 , 𝐾 𝑠 , 𝑉 𝑠 ∈ R 𝑁 ×𝑑 , 𝑆𝑈 𝑀 𝑟 (•) is the row summary. Lemma 2. The scaling factor (s) can be incorporated into the threshold 𝑉 𝑡ℎ .EQUATION)where 𝑥 and 𝒐 (𝑡 ) are the input and the output of the Spiking Neuron, respectively.As shown in Eq.19, we update the 𝑉 𝑡ℎ to 𝑉 𝑛𝑒𝑤 𝑡ℎEQUATIONThus, in PSSA, the computation process is pure sparse, focusing attention on relevant features to reduce computational overhead. It is also spike-driven, introducing nearly energy-free fully binary spiking computation, significantly reducing power resources and energy costs, aligning with the computational rules of neuromorphic chips.DSMS retains the concept of MS connections in the main residual path. In the shortcut path, when the feature channel number needs to change, an additional LIF neuron model is added before ConvBN to transform information into sparse spiking signals.In the context of semantic segmentation, DSMS adapts to backbone networks that extract features through continuous downsampling using Maxpooling to reduce feature map size and parameters. When the channel number remains the same or decreases, DSMS employs a single-branch concatenation of LIF neurons and ConvBN after Maxpooling. In cases requiring an increase in channel number, DSMS introduces a Concat operation for comprehensive spiking feature reuse. This dynamic control of channels ensures consistency with the residual main branch, allowing DSMS to efficiently extract target features from diverse dimensions and channel numbers. The above process can be described by the following equations.EQUATIONEQUATIONwhere 𝑋 ′ is the computation result of the maxpool branch. The quantity and channel width of DSMS can be dynamically adjusted based on specific tasks. In semantic segmentation tasks, optimizing DSMS improves information flow, and the output flow on the shortcut path conceptually approximates the sum of synaptic inputs to the neuron membrane. This representation of complete spiking features effectively reflects the energy-efficient characteristics of the network, ensuring that all modules in the network are spike-driven.Unlike the sequence of operations from the order discussed in3.4, the input in SPG-Down undergoes the ConvBN operation to obtain the feature map 𝑥. Following this, it undergoes max-pooling to obtain a downsized feature map ℎ. Finally, the pulse neuron activation produces the feature map 𝑦.The backward propagation stage of the above process can be expressed by the Eq.23-25 :EQUATIONEQUATIONEQUATIONwhere 𝑥 denotes the output of "ConvBN". The output of the LIF neuron is its membrane potential ℎ ∈ R 𝐻 ×𝑊 , with the maximization goal being the output vector 𝑦 ∈ R 𝑆 ×𝑆 , making the gradients 𝜕𝑦 𝜕ℎ and 𝜕𝐿 𝜕𝑦 non-zero. The activation function and the non-differentiable membrane potential update equation pose key challenges for gradient propagation. During backpropagation, the non-zero nature of 𝜕𝐿 𝜕𝑦 𝑖 𝑗 and𝜕𝑦 𝑖 𝑗 𝜕ℎ 𝑢𝑣results in the gradient 𝜕𝐿 𝜕ℎ 𝑢𝑣 becoming zero, where 𝑥 𝑢𝑣 , ℎ 𝑢𝑣 , 𝑦 𝑖 𝑗 represent the respective gradients from 𝜕ℎ 𝑢𝑣 𝜕𝑥 𝑢𝑣 , then all gradients would vanish during propagation. Each term of the gradient is the product of the derivative of the previous term and the current term, where 𝛼 is an amplification factor to apply this amplification to all gradient calculations, with 𝑚𝑎𝑥 (ℎ) being amplified to a sufficiently large value.Evidently, the maximum element after max-pooling in the feature map x is selected as the maximum value, representing the position with the most feature information in x. During back propagation, this position retains the gradient, preserving the maximum feature information in each local region before max-pooling (aggregation). Additionally, since max-pooling is applied to feature map x first, the feature map h has a reduced size, saving energy and memory resources. 
experiment: 
 In this work, the datasets used include ImageNet, CIFAR-10, and ADE20K. The introductions to these datasets and basic experimental setup are provided in the supplementary materials. In this section, ablation studies are conducted on the various modules proposed in this paper; followed by an analysis of the performance of the proposed PSSD-TFSeg on semantic segmentation datasets. It's worth noting that, as we are the first to apply an SNN model on largescale semantic segmentation datasets, there are no other models for comparison. The chapter concludes with the presentation of Grad-Cam images and semantic segmentation result images from different layers of the model.To investigate the characteristics and effectiveness of several modules proposed in this chapter, including the spiking computing approach (PSSA self-attention, MS and DSMS residual connections), SPG-Down precise gradient downsampling, and neuron mechanisms (Group Horizontal G-HLIF and Group Vertical G-VLIF), a series of ablation experiments were conducted. These experiments further assessed the contributions of the proposed algorithms.Experiments were conducted on the ImageNet1K dataset to assess the accuracy and performance of the PSSA self-attention mechanism and MS/DSMS residual connections. This involved ablation studies on SSA and PSSA mechanisms, SEW versus MS/DSMS, and the combined effect of these two modules.Tab.1 reveals that although replacing SSA with PSSA incurs a slight loss in accuracy, the energy consumption is less than 50% of that of the SSA-based Spikformer. Essentially, the pulse-driven selfattention mechanism utilizes binary self-attention scores to mask non-essential channels in sparse pulse value tensors, converting all multiplications into sparse computational additions. While this leads to a minor accuracy reduction, the PSSA operator consumes almost no energy. This not only achieves sparse pulse driving but also aligns with low-power computing principles, bringing the deployment of neuromorphic chips closer to reality. As shown in Tab.2, this setting demonstrated the effectiveness and rationality of the optimized precise downsampling method.In this section, comparative quantitative experiments were conducted using the ADE20K dataset (with a resolution of 512 × 512) and three evaluation metrics. The semantic segmentation backbone network employed was PSSD-Transformer, where residual connections utilized both MS and DSMS. Additionally, the model incorporated the pulse downsampling method SPG-Down and neuron Clearly shown in Fig. 3 , the Grad-CAM [34] of the PSSD-Transformer effectively segments the image into distinct semantic regions, aligning with the semantic categories of each area. Within each encoder block of the model, the focus is gradually refined to emphasize key object areas in the actual data. This precise targeting significantly bolsters the model's ability to learn various semantic categories, thereby enhancing its capability to more accurately assign class labels to individual pixels in later segmentation tasks. Fig. 4 presents a glimpse of the PSSD-TFSeg model's semantic segmentation capabilities on the ADE20K dataset. This model features an MLP-based semantic segmentation head and MS for residual connections within its encoder-decoder framework. Excelling in a variety of scenes, the model adeptly focuses on key objects for segmentation. This includes intricate indoor settings with numerous people and objects, as well as outdoor landscapes populated with cars, buildings, and animals. 
discussion 6.1 energy consumption estimation: 
 The essence of SNN algorithms is to reconcile the trade-off between precision and power usage. This segment delves into how spike-based models achieve efficiency, utilizing FLOPS and SOPs as performance indicators. SOPs, indicating spikes per second, mirror an SNN's complexity; fewer SOPs mean lower power draw. PSSD-Transformer's efficiency stems from Self-Attention, ConvBN, and MLP modules, which skip computations when inputs are inactive. Energy use for ConvBN and MLP is calculable via operator energy, ANN FLOPS, SFR, and timestep count. The energy consumption of each component is reflected in Tab.1 and Tab.3. Above process can be described by Eq.26-28.EQUATION)EQUATION𝐸 sum = 𝐸 ConvBn + 𝐸 MLP + 𝐸 Attn (28) Here, FL represents the known FLOPS of the corresponding architecture ANN, reflecting the spiking rate, 𝐸 𝐴𝐶 represents the energy consumption of a single addition operation in the operators. Generally, in ANN models, 𝐸 𝑀𝐴𝐶 = 4.6pJ, 𝐸 𝐴𝐶 = 0.9pJ, T represents the simulation time step, and 𝑁 𝐿𝐼 𝐹 represents the number of spiking neurons.The application of our work on other datasets has not yet been completed, but it has already demonstrated strong performance on the ADE20K dataset. Future work will focus on both depth and breadth: depth refers to deeper and more powerful networks, while breadth refers to more fields and datasets. Another point is that, similar to other works in this field, energy consumption estimation is still in a preliminary phase. Experiments are conducted on GPUs and have not yet been actually deployed on neuromorphic chips. We believe that in the near future, neuromorphic chips will support a wider range of networks and find extensive applications across various fields, shining brightly. By then, PSSD will become even more meaningful. 
conclusion: 
 In this work, we introduce the PSSD-Transformer, a novel approach that effectively marries SNNs with the demanding task of semantic segmentation. By addressing the challenges faced by SNNs in computer vision, such as their intensive floating-point computations, inaccurate layer positioning, and limited local information representation, we have paved the way for the first large-scale application of SNNs in semantic segmentation. Our proposed solutions, including PSSA, DSMS, SPG-down, and Group-LIF neurons, collectively enhance the model's efficiency, accuracy, and applicability. Demonstrating significant improvements on various datasets, the PSSD-Transformer not only surpasses existing models in performance but also highlights the untapped potential of SNNs in energy-efficient, high-precision visual tasks.  
: 
 convolutional and pooling layers relative to spiking layers leads to reduced accuracy; iii) Leaky Integrate-and-Fire (LIF) neurons have limited capability in representing local information, which is disadvantageous for downstream visual tasks like semantic segmentation.To address the challenges in SNNs, i) we introduce Pure Sparse Self Attention (PSSA) and Dynamic Spiking Membrane Shortcut (DSMS), combining them to tackle the issue of floating-point computations; ii) the Spiking Precise Gradient downsampling (SPGdown) method is proposed for accurate gradient transmission; iii) the Group-LIF neuron concept is introduced to ensure LIF neurons' capability in representing local information both horizontally and vertically, enhancing their applicability in semantic segmentation tasks. Ultimately, these three solutions are integrated into the Powerful Sparse-Spike-Driven Transformer (PSSD-Transformer), effectively handling semantic segmentation tasks and addressing the challenges inherent in SNNs. The experimental results demonstrate that our model outperforms previous results on standard 1 INTRODUCTION Introduction of SNNs. Spiking Neural Networks (SNNs) are a type of neural network that more closely mimic biological neural networks. Unlike traditional Artificial Neural Networks (ANNs) that use continuous values, SNNs operate using discrete events or "spikes." These spikes are binary events that occur at points in time, resembling the way biological neurons transmit information. SNNs are known for their energy efficiency and are considered more biologically realistic. They have potential applications in areas where low power consumption [15, 27, 33, 39] is crucial, such as edge computing and neuromorphic hardware [3, 9, 11, 35] .Introduction of semantic segmentation. Semantic segmentation [16, 29, 41, 46] is a computer vision task where the goal is to assign a label to each pixel in an image such that pixels with the same label share certain characteristics. This process essentially divides the image into meaningful parts with semantic interpretations, like identifying objects, boundaries, or regions. Semantic segmentation is widely used in various applications, including autonomous vehicles [5, 19, 23, 45] , medical imaging [14, 38] , and scene understanding [8, 42] . The challenge lies in accurately classifying each pixel while maintaining the integrity of object shapes and contextual relationships within the image.Why combining the spike mechanisms with segmantic segmentation tasks? Semantic segmentation, a task of assigning accurate semantic categories to each pixel in an image, plays a crucial role in image understanding by identifying and dividing different semantic regions in a scene. This vision-intensive computation task has significant application potential in various realworld scenarios like autonomous driving, medical surgery robots, and wearable devices. In these contexts, neuromorphic chips with sparse spike-driven properties and low-power computational rules are ideal due to their efficiency and suitability for these applications. The deployment of SNNs on neuromorphic chips is considered a key approach for implementing vision models, such as image segmentation models, marking a promising direction in the future development of artificial intelligence.How to combine the spike mechanisms with semantic segmentation tasks? To answer this question, we consider two perspectives: the contradiction between semantic segmentation's computational density and the low-power nature of spike-driven computation, and the need for spatial consistency and semantic accuracy in semantic segmentation. i) To address the core issue of the contradiction between the computationally dense nature of semantic segmentation and the low-power spike-driven computation rules, on one hand, the introduction of Pure Sparse Self Attention (PSSA) allows the semantic segmentation model to selectively focus on the most important areas for the task, ignoring irrelevant features and fully utilizing the sparsity of model parameters. On the other hand, the introduction of Dynamic Spiking Membrane Shortcut (DSMS) reduces the information loss caused by the use of sparse spike-driven attention, achieving sparse spikedriven computation, significantly reducing energy consumption and easing the application of image semantic segmentation models on neuromorphic chips. ii) Regarding the need for spatial consistency and semantic accuracy in semantic segmentation, the spiking Transformer inherits the global semantic information modeling capabilities of the Transformer for long-distance dependencies. The core issue then is how to enhance spatial consistency and focus on local information. On one hand, the introduction of a locality-based Spiking Precise Gradient downsampling (SPG-down) mechanism preserves gradients at the points of maximum feature information during backpropagation, and uses DSMS to better preserve spatial information. On the other hand, by migrating Group-LIF neurons from the temporal domain to the spatial domain in the feedforward network, considering the neighborhood relationships of pixels in different directions based on feature grouping, local feature contexts are obtained, addressing the problem of weak local information relevance caused by long distances, thereby enhancing the spatial consistency and correlation of features.Our contributions can be summarized as follows:• Group-LIF for Local Information Control: Developing Group-LIF neurons to enhance control over local information representation. with more recent studies [30, 44, 48] often utilizing surrogate functions for gradient propagation. 
 
 Appendix:  


Decision: The paper introduces the PSSD-Transformer, a novel model for improving Spiking Neural Networks (SNNs) in image semantic segmentation. Reviewers have noted the innovation and substantial contributions of components like Pure Sparse Self Attention (PSSA), Dynamic Spiking Membrane Shortcut (DSMS), and Spiking Precise Gradient downsampling (SPG-down). Despite some concerns regarding formula formatting, clarity of illustrations, and comparative analysis, the majority of reviewers have acknowledged the solid empirical results and the paper's significant contribution to the field. Consequently, I recommend accepting this submission.

Reviews: review: paper_summary: paper_summary: The paper introduces the PSSD-Transformer, a novel model designed to improve the performance of Spiking Neural Networks (SNNs) in the domain of image semantic segmentation. This model addresses several challenges faced by existing SNNs, including the reliance on floating-point computations, poor positioning of convolutional and pooling layers, and the limited local information representation of Leaky Integrate-and-Fire (LIF) neurons. The authors propose three key components: Pure Sparse Self Attention (PSSA), Dynamic Spiking Membrane Shortcut (DSMS), and Spiking Precise Gradient downsampling (SPG-down). These innovations are combined in the PSSD-Transformer architecture, which has demonstrated superior performance on standard classification and semantic segmentation datasets.
strength_weakness: strength_weakness: 1) Innovative Components: Introduction of PSSA, DSMS, and SPG-down addresses specific limitations of traditional SNNs, improving overall performance.
2) Experiment: The ablation experiment is well set up.
3) The workload of the paper is substantial.
limitations: limitations: 1) Many details in the article require careful adjustment by the author. Here just briefly list some. In terms of formulas: the problem of alignment between (21) and (22), the problem of switching formula (25), and A_{ij=1} in formula (17). There are also lack of symbol descriptions behind the formulas. For example, the R and MSFR symbols in formulas 26-28 need to be explained after the formulas; section references: such as in section 4.4. "Unlike the sequence of operations from the order discussed in3.4, “ this could be written as “Unlike the sequence of operations from the order discussed in sec 3.4,”; citation: Some citations need to be reconfirmed, such as “[2] Malyaban Bal and Abhronil Sengupta. 2023. SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation. arXiv preprint arXiv:2308.10873 (2023).” But as far as I know, this article has been published at AAAI2024. Please pay attention again to the issue of citation format. At the same time, the article lacks citations, such as "Generally, in ANN models, E_{MAC}= 4.6pJ, E_{AC} = 0.9pJ," this data needs to be cited; Please readjust the abstract format.
2) Lack of comparative in Experiments. In your paper, “ It’s worth noting that, as we are the first to apply an SNN model on largescale semantic segmentation datasets, there are no other models for comparison. ” I hope to see you guys applying your model on small scale semantic segmentation datasets to compare with previous SNN models and show your model’s ability in small datasets.
3) Neuromorphic Hardware. If this paper cannot apply real Hardware experiments, please mention less hardware advantages in the article. Low computational effort is enough. If you really wants to mention the hardward. Please add the specific hardware requirements to achieve the energy efficiency benefits described.
4) Training Efficiency: How does the training time of the PSSD-Transformer compare to traditional ANN-based models for semantic segmentation?
5) Group-LIF: Could you please tell me the specific formula and derivation of Group-LIF?
paper_summary: paper_summary: mentation.
To address the challenges in SNNs, i) we introduce Pure Sparse
Self Attention (PSSA) and Dynamic Spiking Membrane Shortcut
(DSMS), combining them to tackle the issue of floating-point computations; ii) the Spiking Precise Gradient downsampling (SPGdown) method is proposed for accurate gradient transmission; iii)
the Group-LIF neuron concept is introduced to ensure LIF neurons’ capability in representing local information both horizontally
and vertically, enhancing their applicability in semantic segmentation tasks. Ultimately, these three solutions are integrated into the
Powerful Sparse-Spike-Driven Transformer (PSSD-Transformer),
effectively handling semantic segmentation tasks and addressing
the challenges inherent in SNNs. The experimental results demonstrate that our model outperforms previous results on standard
strength_weakness: strength_weakness: • Group-LIF for Local Information Control: Developing
Group-LIF neurons to enhance control over local information
representation.
• PSSA + DSMS for Non-floating Point Operations: Implementing Pure Sparse Self Attention (PSSA) combined with
Dynamic Spiking Membrane Shortcut (DSMS) to achieve
spike-based processing without floating-point computations.
• SPG-down for Precise Gradient Transmission: Introducing Spiking Precise Gradient downsampling (SPG-down) to
ensure more accurate gradient propagation.
• PSSD-Transformer Architecture: Proposing the Powerful Sparse-Spike-Driven Transformer (PSSD-Transformer)
architecture and applying it to image classification and semantic segmentation tasks, marking the first application of
a spiking Transformer architecture to large-scale semantic
segmentation datasets.
limitations: limitations: it's hard to train, and how to get snn information
paper_summary: paper_summary: Proposed the Pure Sparse Self Attention (PSSA) and Dynamic Spiking Membrane Shortcut (DSMS) to address the issues of floating-point computations and inaccurate layer positioning in SNNs. Introduced the Spiking Precise Gradient downsampling (SPG-down) method to enable accurate gradient backpropagation for downsampling operations in semantic segmentation. Developed the Group-LIF neurons to enhance the local information representation capability of LIF neurons.
strength_weakness: strength_weakness: 1. A pulse-based self-attention mechanism (PSSA) and residual connection (MS and DSMS) are proposed, which can significantly reduce energy consumption with only a slight loss of accuracy.12
2. An optimized accurate downsampling method (SPG-Down) is proposed, which can improve the spatial consistency and thus improve the performance of semantic segmentation.
limitations: limitations: 1. The formatting of the formulas is inconsistent; some are centered, some are aligned to the left, and others to the right.
2. The clarity of Figure 1(B) is insufficient.
paper_summary: paper_summary: This paper proposes Powerful Sparse-Spike-Driven Transformer (PSSD-Transformer), a spiking Transformer for image semantic segmentation. It employs four techniques: Group-LIF neuron, PSSA mechanism, DSMS shortcut, and SPG-down. Ablation experiments demonstrate the effectiveness of the proposed techniques.
strength_weakness: strength_weakness: + This paper discusses in detail the design of residual shortcuts for spiking neural networks, which is commonly ignored in SNN works.
limitations: limitations: - Lines 118-120 state that the PSSD is the first model in the SNN field to perform semantic segmentation on large datasets. However, Spike-driven Transformer v2 [1] has performed semantic segmentation on the ADE20K dataset and achieved better performance (up to 35.3% MIoU). Therefore, this claim is misleading.
- The proposed PSSA mechanism lacks novelty. It is similar to the SDSA mechanism proposed in Spike-driven Transformer [2]. Please detail the novelty of PSSA compared to SDSA.
- The performance of PSSD lags behind the state-of-the-art SNN model Spike-driven Transformer v2.
- Figure 1 is not well drawn. the three sub-figures A, B, and C are not stylized uniformly. In addition, these sub-figures are not vector graphics, making the text inside difficult to recognize.



[1] Yao, M., Hu, J., Hu, T., Xu, Y., Zhou, Z., Tian, Y., ... & Li, G. (2024). Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. In *The Twelfth International Conference on Learning Representations*.

[2] Yao, M., Hu, J., Zhou, Z., Yuan, L., Tian, Y., Xu, B., & Li, G. (2024). Spike-driven transformer. *Advances in Neural Information Processing Systems*, *36*.