Towards Low-latency Event-based Visual Recognition with Hybrid Step-wise Distillation Spiking Neural Networks

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-18 07:38:19

Abstract: Spiking neural networks (SNNs) have garnered significant attention for their low power consumption and high biological interpretability. Their rich spatio-temporal information processing capability and event-driven nature make them ideally well-suited for neuromorphic datasets. However, current SNNs struggle to balance accuracy and latency in classifying these datasets. In this paper, we propose Hybrid Step-wise Distillation (HSD) method, tailored for neuromorphic datasets, to mitigate the notable decline in performance at lower time steps. Our work disentangles the dependency between the number of event frames and the time steps of SNNs, utilizing more event frames during the training stage to improve performance, while using fewer event frames during the inference stage to reduce latency. Nevertheless, the average output of SNNs across all time steps is susceptible to individual time step with abnormal outputs, particularly at extremely low time steps. To tackle this issue, we implement Step-wise Knowledge Distillation (SKD) module that considers variations in the output distribution of SNNs at each time step. Empirical evidence demonstrates that our method yields competitive performance in classification tasks on neuromorphic datasets, especially at lower time steps.

Main paper: 
 introduction: 
 In recent years, spiking neural networks (SNNs) [26] , heralded as the next generation neural network paradigm, have garnered considerable attention due to their rich spatio-temporal characteristics and event-driven communication [31] . Notably, compared to conventional neural networks, SNNs exhibit the distinctive advantage of low power consumption.Indeed, SNN manifests a significant advantage in terms of low power consumption due to their transmission of information through binary 0 or 1 signals [6] . However, satisfactory results are achieved by accumulating information over multiple time steps, leading to heightened inference latency and increased power consumption [6] .To strike a balance between accuracy and latency, two primary methods for training SNNs have been proposed: the artificial neural network (ANN)-SNN conversion method [2, 3, 8, 9, 32] and the direct training SNN method [8, 12, 44, 46] . The former achieves high accuracy after converting weights from a pre-trained ANN but experiences noticeable latency during inference stage. The utilization of surrogate gradients during SNN training via back-propagation reduces latency but comes with the drawback of information transmission loss [15] . In particular, due to the temporal dimension inherent in the spiking neurons of SNNs, it renders them exceptionally well-suited for neuromorphic, event data [39] . The event data captured by event cameras record variations in light intensity at each pixel, resulting in a sequence of events that include the pixel's location, time, and polarity arising from changes in light intensity. On the one hand, the reductions in latency are primarily observable in traditional static data scenarios, e.g., CIFAR 10/100 and even the larger ImageNet dataset [3, 8] . And in other event-based vision tasks, such as dense prediction, recent works have demonstrated high accuracy can be achieved at extremely low time steps in the inference stage [4, 16] . This motivates us to consider a similar thought, which naturally raises the question: How to achieve a trade-off between high accuracy and low-latency in event-based visual recognition? 7LPHVWHS 7HVW$FFXUDF\ +6' 7(7676 7 (7' The existing methods require higher latency for SNNs to recognize event-based neuromorphic objects, e.g., CIFAR10-DVS [22] , N-CALTECH101 [28] , and DVS-GESTURE [1] , which still require 10 time steps. As illustrated in Fig. 1 , to reduce the latency in the inference stage, we believe that it can be divided into STS and DTS, where STS representing ensuring the consistency of time steps between the training and inference stages of SNNs, thereby reducing the overall time step during SNN training, DTS does not guarantee the consistency of time steps between the training and inference stages of SNNs, it only decreases the time step during the inference stage. And as shown in Fig. 2 , when the time step is reduced, especially when the time step is less than or equal to 5, there is a significant decrease in performance. This decrease in accuracy can be mainly attributed to the sparsity of event data and the limited accumulation of event frame information in lower time steps [13, 42, 43] . Furthermore, we observe that STS often outperforms DTS, particularly at lower time steps. Through analysis, we attribute the performance gap to the fact that using an equal number of event frames for both training and inference allows the SNN to capture the dynamic features and contextual information of the complete event data learned during training. In contrast, inconsistency results in information loss, leading to a significant performance drop.Hence, achieving higher performance often requires maintaining consistency in the time steps between the training and inference stages of SNNs, but this is closely associated with increased latency. We identify the sparse nature of event data as the primary cause of this issue, necessitating a larger number of event frames compared to static data to accumulate sufficient information. However, conventional practices in event data classification tasks often bind the time steps of SNN with the number of event frames, hindering the low-latency requirements for event data classification.To address this challenge, we propose disentangling the relation between the time steps of SNN and the number of event frames. Specifically, during the training stage, we ensure that SNN learns as much information from event frames as possible to enhance performance. Conversely, during the inference stage, we aim to minimize the utilization of event frames to reduce the time steps and consequently decrease latency in SNN.In this paper, we propose a novel Hybrid Step-wise Distillation (HSD) method for event data classification tasks. We advocate for the partition of the SNN training process into a pre-training phase and a fine-tuning phase. The pre-training phase aims to facilitate the acquisition of more event frame information by the SNN, thereby enhancing overall performance. In contrast, the fine-tuning phase is designed to ensure that the SNN meets the low-latency requirements during the inference stage. Consequently, we partition event data into two distinct parts.Given the limited information in event data, the first part leverages the robust feature extraction capabilities of ANN to learn the spatial features of event data. Subsequently, employing ANN-SNN conversion, these features are transmitted to the SNN during the pre-training phase. The second part involves fine-tuning the converted SNN to capture the spatio-temporal characteristics of event data under low time steps as effectively as possible. However, owing to the scarcity of information in event data compared to static data, the average output distribution of SNN across all moments can be more susceptible to the influence of individual time steps with abnormal output distributions, particularly at lower time steps.To address this challenge, we introduce the Step-wise Knowledge Distillation (SKD) module. This module facilitates the transfer of the "Soft Labels" learned by the ANN during the pre-training phase to the output distribution of each time step in the SNN. This mechanism ensures a more stable training process, mitigating the impact of individual abnormal time steps and enhancing the overall robustness of the network. Experimental results demonstrate that our method achieves a balance between accuracy and latency in classification tasks on neuromorphic datasets.Our contributions can be summarized fourfold:• New Benchmark. We present a novel HSD that disentangles the dependency between the number of event frames and the time steps of SNN, maximizing the utilization of rich event data information. To our knowledge, we are the first to utilize both ANN-SNN conversion and knowledge distillation in classification tasks on neuromorphic datasets. [12] proposed learning membrane parameters with spiking neurons to enhance the expressiveness of SNNs. LIAF-SNN [38] utilized analog values to represent neural activations instead of traditional binary values in leaky integrate-and-fire (LIF) SNNs. EICIL [34] expanded the representation space of spiking neurons. Some studies advocate for enhancing SNN architecture through attention mechanisms, such as [40, 42, 47, 48] . Li et al. [24] , Shen et al. [35] aimed to improve the generalization of SNNs by incorporating data augmentation techniques. In order to improve the generalizability of SNNs on event-based datasets, He et al. [17] used static images to assist SNN training on event data. Meng et al. [27] proposed SLTT method that can achieve high performance while greatly improving training efficiency compared with BPTT. Unlike ANN, SNN introduces a temporal dimension, prompting many studies to focus on this aspect. Threshold-dependent BatchNorm (tdBN) [46] extended temporal dimension information to conventional BatchNorm. Deng et al. [8] expanded common loss functions to the temporal dimension to enhance generalizability. Ding et al. [10] proposed SSNN alleviates the temporal redundancy of SNN and significantly reduces inference latency. And Li et al. [25] adjusted the number of time steps based on different samples to reduce the latency of SNN.Knowledge Distillation is initially proposed by [19] for model compression, subsequently utilized to enhance the performance of student models by transferring knowledge from teacher models. Romero et al. [30] first introduced the feature transfer into knowledge distillation (KD) and subsequently improved it through a series of works [5, 18] . Recent research in SNNs, such as [21] , employing large SNN teacher models to guide training of smaller SNN student models. Takuya et al. [36] introduced an ANN-teacher model to instruct SNN-student models. Xu et al. [41] utilized feature-based and logit-based information from ANNs to distill SNNs. Guo et al. [15] proposed a joint training framework of ANN and SNN, in which the ANN can guide the SNN's optimization. In contrast to compressing the temporal dimension information of SNNs as in their methods, we consider variations in the output distribution of SNNs at each time step. 
preliminaries 3.1 partition of neuromorphic datasets: 
 Neuromorphic datasets typically utilize the Address Event Representation (AER) format, represented as 𝐸 (𝑥 𝑖 , 𝑦 𝑖 , 𝑡 ′ 𝑖 , 𝑝 𝑖 ) where 𝑖 ranges from 0 to 𝑁 -1, to convey event location in the asynchronous stream, timestamp, and polarity. To manage the large volume of events, we aggregate them into frames for processing, following methodologies outlined in previous studies [12, 48] . Specifically, events are partitioned into 𝑇 slices, with events within each slice cumulatively accumulated. The integrated event at location (𝑥, 𝑦) in the 𝑗-th slice (0 ⩽ 𝑗 ⩽ 𝑇 -1) is denoted as:EQUATIONwhere the function L 𝑝,𝑥,𝑦 (𝑝 𝑖 , 𝑥 𝑖 , 𝑦 𝑖 ) serves as an indicator. The indices 𝑗 𝑙 and 𝑗 𝑟 correspond to the minimal and maximal timestamp indexes within the 𝑗-th slice. It is worth to note that 𝑇 signifies the total number of time steps in the training stage under our experimental setup.In HSD, the time steps 𝑇 also represents the total number of input event frames in the training stage. We partition it into two segments: the preceding segment consists of 𝑇 1 event frames, serving as input for pre-training phase, while the subsequent segment comprises 𝑇 2 event frames, utilized by SNN in the fine-tuning phase. Therefore, it can ensure that SNN learns complete event frame information during the training stage, improving accuracy, while the inference stage only uses partial event frames for inference, reducing latency. Specifically, 𝑇 = 𝑇 1 + 𝑇 2 .For ANNs, the input 𝒂 𝑙 -1 to layer 𝑙 undergoes a linear transformation using matrix 𝑾 𝑙 and a nonlinear activation function 𝑓 (•), where 𝑙 = {1, 2, 3, • • • , 𝐿},EQUATIONwhere 𝑓 (•) typically chosen as ReLU activation function.In SNNs, IF neuron model is commonly employed for converting ANNs to SNNs [2, 3, 7] . To minimize information loss during inference, our neurons perform a "reset-by-subtraction" mechanism [33] , where the firing threshold 𝜃 𝑙 is subtracted from the membrane potential upon firing. The fundamental kinetic equations of IF neuron can be expressed as:EQUATIONwhere 𝒗 𝑙 (𝑡) denotes the membrane potential of layer 𝑙 at the 𝑡-th time step. 𝑾 𝑙 is the synaptic weight between layer 𝑙-1 and layer 𝑙, and 𝜃 𝑙 is the spike firing threshold in the 𝑙-th layer. 𝒔 𝑙 (𝑡) represents whether the spike fires at time step 𝑡, which is defined as:EQUATIONwhere 𝒖 𝑙 (𝑡) = 𝒗 𝑙 (𝑡 -1) + 𝑾 𝑙 𝒔 𝑙 -1 (𝑡)𝜃 𝑙 -1 denotes the membrane potential of neurons before spiking at time step 𝑡, 𝐻 (•) represents the Heaviside step function. The output spike 𝒔 𝑙 (𝑡) is set to 1 if the membrane potential 𝒖 𝑙 (𝑡) exceeds the threshold 𝜃 𝑙 and 0 otherwise.  
ann temporal training: 
 Considering that the event stream generates discrete events every 1 𝜇s, and the information carried by individual events is limited, the conventional method is to aggregate the event stream into continuous event frames [12, 24, 37, 42, 46] . However, while the event stream contains temporal information at the millisecond level, the number of integrated event frames typically falls below 100, diminishing the significance of temporal information. Furthermore, for event data classification tasks, spatial information holds greater importance. Therefore, we opt not to employ a 3D network for processing event frames in the spatio-temporal training of ANN, but instead, we utilize a 2D network. This decision not only reduces computational overhead but also facilitates the migration of ANN-SNN conversion used for static datasets.For ANNs, we utilize consecutive event frames, treating each frame as independent. Therefore, for ANN training, we employ the average cross-entropy loss:EQUATIONwhere 𝒚 𝑡 denotes the model's prediction probability, 𝒚 true signifies the true labels, and𝑇 1 represents the number of event frames utilized for ANN training. L ACE resembles TET loss [8] , tailored to improve the model's generalization.To facilitate the conversion from ANN to SNN, aligning the firing rates (or postsynaptic potentials) of spiking neurons with ReLU activation outputs of artificial neurons is crucial. By integrating Eq. ( 3) from 𝑡 = 1 to 𝑇 AS and normalizing by 𝑇 AS , resulting:EQUATIONwhere 𝑇 AS denotes the total simulation cycle of ANN-SNN conversion. For simplicity, we substitute the term𝑇 AS 𝑡 =1 𝑠 𝑙 (𝑡 )𝜃 𝑙 𝑇 ASin Eq. ( 6) with the average postsynaptic potential:EQUATIONwhere mirrors the forward propagation Eq. ( 2) in ANNs when 𝜙 𝑙 (𝑇 AS ) ⩾ 0. This indicates that lossless ANN-SNN conversion can be attained as 𝑇 AS methods infinity. For high-performance SNNs under low-latency, Bu et al. [3] suggested substituting the ReLU activation function in source ANNs with QCFS function:EQUATIONwhere 𝐿 denotes ANN quantization step, and 𝜆 𝑙 signifies the trainable threshold of the outputs in ANN layer 𝑙, which corresponds to the threshold 𝜃 𝑙 in SNN layer 𝑙. Our work adheres to the conversion framework [3] , employing QCFS function. Consequently, the SNN inherits the rich spatial information acquired by the ANN, mitigating the risk of excessive information loss.Vanilla KD is categorized into feature-based distillation, logic-based distillation, and relation-based distillation [14] . We opt for logicbased distillation following [19] . Soft labels are preferred over hard labels as they allow student models to retain the predicted probability distribution obtained from teacher models. We employ Kullback-Leibler (KL) divergence to constrain the student's output to match the distribution of the teacher. Thus, the vanilla L KD is defined as:EQUATIONwhere 𝑝 𝑎 𝜏 and 𝑝 𝑠 𝜏 denote the predicted distributions for ANNs and SNNs, respectively, and 𝑁 represents the total number of samples. During SNN training, the standard method integrates information across all time steps to derive the final prediction, yielding:EQUATIONwhere 𝑇 2 denotes the time steps of fine-tuning phase, and 𝑝 𝑎 𝜏 resembles Eq. (10) . The output of the vanilla KD student model follows this pattern. However, since SNN outputs the probability distribution of each category at every time step, as 𝑇 2 increases, the final average probability distribution can roughly reflect the relation between each category. Nonetheless, when 𝑇 2 is small, the ability of the average probability distribution to represent the overall category probability distribution is limited. In contrast to SNNs on static datasets, the input at each time step for SNNs on neuromorphic datasets exhibits dynamic variability, leading to a significant difference in the distribution of the final output. To address this issue, SKD module is constructed to transfer the probability distribution learned by the ANN teacher model to each time step of SNN, facilitating a smoother distribution of output for SNN. Therefore L SKD is defined as:EQUATIONThe overall training algorithm is outlined in Algorithm 1. To address the non-differentiability inherent in training SNNs via backpropagation, we employ the surrogate gradient technique. We choose triangular-shaped surrogate gradients [8] , it can be described as:EQUATION)where 𝛾 is the constraint factor determining the sample range for activating the gradient, while 𝑉 th denotes the threshold of IF neuron. We set 𝛾 to 1 and 𝑉 th to 1. The final loss function is expressed as:EQUATIONwhere L CE denotes cross-entropy loss, while 𝜆 signifies a hyperparameter to adjust the proportion of L SKD .  
experimental settings: 
 Datasets. CIFAR10-DVS [22] dataset includes 10k dynamic vision sensor (DVS) images, adapted from the original CIFAR10 dataset. A training-validation split of 9:1 is used, resulting in 9k training images and 1k validation images. Initially, images are of 128 × 128 pixels but are resized to 48 × 48 for training, with the event data distributed across 10 frames per sample [8] .N-Caltech101 [28] dataset consists of 8,831 DVS images, sourced from the original Caltech101 dataset. The preprocessing method mirrors that of CIFAR10-DVS. DVS-Gesture [1] dataset obtained using a DVS128 camera, it encompasses recordings of 11 hand gestures performed by 29 subjects under varying lighting conditions. This dataset organizes each event data into 16 frames.Metric. Across all datasets, we utilize Top-1 accuracy to evaluate the model's performance [8] .Implementation Details. Our experimental setup leverages an NVIDIA Tesla V100 GPU, utilizing the PyTorch framework and the SpikingJelly package [11] . We employ the VGG-SNN [8] for CIFAR10-DVS and N-Caltech101, and SNN-5 [12] for DVS-Gesture. Data augmentation [24] are implemented consistently across all datasets. During the pre-training phase, the batch size is set to 128 for CIFAR10-DVS and N-Caltech101, and we utilize the stochastic gradient descent (SGD) [29] optimizer with a learning rate of 0.1. The batch size is set to 32 for DVS-Gesture, we employ the adaptive moment (Adam) estimation [20] optimizer with a learning rate of 1e-3. Training epochs is set to 300, adopting a cosine learning rate schedule and weight decay parameter of 5e-4.During the fine-tuning phase, we utilize the Adam optimizer with a learning rate of 1e-5, and training epochs is set to 100. Specific to our method, the initial 5 frames are dedicated to pretraining ANN-SNN, while the subsequent 5 frames are reserved for fine-tuning the converted SNN on CIFAR10-DVS. A similar partitioning strategy is applied to N-Caltech101. Notably, owing to the richer temporal information on DVS-Gesture, the initial 6 frames are utilized for pre-training, followed by 10 frames for fine-tuning. Here, 𝑇 signifies the number of event frames involved in inference process, serving as the time steps for SNN, denoted as 𝑇 2 within the method.In Tab. 1, we compare our experimental results with previous works. Concerning CIFAR10-DVS, our HSD has demonstrated superior accuracy compared to the majority of existing methods. At low time steps, our HSD achieves an accuracy of 81.10%, which is 7.47% higher than that of SSNN [10] . Although it may not attain the performance level of TET [8] , it is noteworthy that our time step is only half of theirs. Moreover, when both time steps are set to 5, HSD surpasses TET with an enhancement of 2.30%. Additionally, we also reproduce the results of TCJA [48] and SLTT [27] on CIFAR10-DVS. Specifically, in 𝑇 = 5, TCJA achieves an accuracy of 79.73%, whereas our HSD outperforms TCJA with an improvement of 1.37%. And compared to SLTT, the performance improves by 4.98%.Limited results are available for N-Caltech101. When the time step is 5, our HSD achieves an accuracy of 80.20%, exceeding SSNN by 2.23%. And, our HSD surpasses all baseline methods, even outperforming TET by 0.55% in 𝑇 = 5. Compared to TCJA and SLTT, our HSD demonstrates a significant performance improvement of 4.59% and 2.63%. On DVS-Gesture, we achieve competitive accuracy results of 97.92% with 10 time steps, surpassing the 97.60% accuracy of PLIF [12] and the 96.90% accuracy of SpikFormer [47] . Under the same experimental settings, our method shows an increase of 1.04% compared to TET. Furthermore, under the condition of 𝑇 = 10, our method outperforms TCJA and SLTT. The limited performance improvement observed on DVS-Gesture can be attributed to the insufficient data and the fact that temporal aggregation with 𝑇 = 10 has already accumulated a substantial amount of spatio-temporal information. And when the time step is set to 5, HSD shows a significant performance improvement compared to TET, TCJA, and SLTT, reaching 97.57%. The main reason is that our proposed partition of neuromorphic datasets idea solves the problem of reducing training samples at low time steps.In our HSD, the proposed hybrid training (HT) module and SKD module have significantly enhanced the classification performance at lower time steps. Therefore, in Tab. 2, we aim to validate the effectiveness of these two modules on CIFAR10-DVS, N-Caltech101, and DVS-Gesture. In Tab. 2, compared to vanilla KD, SKD module shows performance enhancements on all three datasets. Notably, for CIFAR10-DVS, we observe a 1.60% increase in accuracy. In addition, for N-Caltech101 and DVS-Gesture experience significant accuracy improvements of 0.50% and 0.70%, respectively.In our HSD, the hyper-parameter quantization step 𝐿 in pre-training phase significantly influences the accuracy of the converted SNN. To assess the impact of 𝐿 and identify the optimal value, we train VGG-SNN [8] and SNN-5 [12] using QCFS activation with different quantization steps 𝐿, specifically 4, 8, 16, and 32. Subsequently, we convert these models into corresponding SNNs.The experimental results are presented in Tab. 3. It is evident that employing an excessively small quantization step 𝐿 adversely impacts the model's capacity, leading to a decrease in the final classification accuracy. Conversely, when the quantization step 𝐿 is too large, it requires the use of lower time steps 𝑇 during the subsequent fine-tuning phase. Therefore, it leads to a decrease in performance. Thus, for CIFAR10-DVS, N-Caltech101, and DVS-Gesture, we select quantization steps 𝐿 of 16, 32, and 16, respectively.The selection of time steps in SNNs significantly impacts both inference latency and power consumption. Achieving lower time steps is a critical goal for SNNs. In this work, we evaluate the robustness of HSD under reduced time steps and compare it with TET [8] . We evaluate the performance of VGG-SNN model trained on synthetic datasets, e.g., CIFAR10-DVS and N-Caltech101. Our analysis indicate that when there is a discrepancy between training and inference time steps, the accuracy of SNNs declines rapidly. To underscore the superiority of our method, we maintain consistency in training and inference time steps within TET by ensuring they are equal, i.e., TET (STS).For CIFAR10-DVS, setting the time step to 1, TET (STS) achieves a performance of 74.82%, while our HSD attains a higher accuracy of 75.81%. As depicted in Fig. 2 (a), HSD's performance consistently exceeds TET (STS)'s at each time step. And in Fig. 2 (b), concerning N-Caltech101, HSD uniformly surpasses TET (STS) in accuracy across all time step, underscoring HSD's superior efficacy, even at extremely low time steps on synthetic datasets. However, for N-Caltech101, the improvement of accuracy from HSD is not as significant as CIFAR10-DVS, this is because N-Caltech101 dataset only has approximately 80 samples per class, so the improvement from HSD is limited.In the case of DVS-Gesture, there is a notable distinction: training and inference stages involve different time steps. Remarkably, when 𝑇 is greater than or equal to 2, its accuracy surpasses scenarios where training and inference stages share the same time steps, as depicted in Fig. 2(c ). This is primarily due to the relatively small size of DVS-Gesture, which comprises only 1, 342 samples. Compared to the other two datasets, this dataset is characterized by limited data, and using more training data during the training stage may assist the model in better generalizing to testing data, thereby enhancing overall performance. For consistency in comparison, the same time steps are maintained for both the training and inference stages on DVS-Gesture.To evaluate the performance of our HSD on real-world DVS-Gesture, we compare it with TET on SNN-5 [12] . While TET may not yield the best results on DVS-Gesture, it accounts for the variations at each time step and maintains the overall network architecture, aligning with our HSD in these respects. As depicted in Fig. 2(c ), in 𝑇 = 1, TET (STS) achieves an accuracy of 90.97%. This is mainly due to the relatively small size of DVS-Gesture. Our HSD also achieves promising results, reaching 91.32%, indicating a marginal improvement of 0.35% over TET (STS). With the increase in 𝑇 , both methods exhibit significant accuracy enhancements, with HSD consistently outperforming TET (STS) and TET (DTS). Overall, leveraging a larger number of event frames and integrating an ANN as a teacher model, our HSD surpasses TET (STS) and TET (DTS), particularly at extremely low time steps.To understand more clearly the feature extraction capability of HSD and the spatio-temporal characteristics of SNN, we visualize feature maps at different time steps 𝑇 in the first spiking neurons layer [12] on CIFAR10-DVS, specifically 𝑇 = 1 and 5. In Fig. 4(a)-(d) , it is evident that as 𝑇 increases, the texture constructed by firing rates becomes more distinct. Our method leverages a larger amount of event frame information and benefits from guidance provided by the teacher model ANN, resulting in feature maps with more pronounced characteristics compared to those of TET [8] . Furthermore, Fig. 4 (e) specifically presents the features of channel 37, revealing consistent patterns in its performance.To verify the generalization of the proposed HT module, we consider applying it to TET [8] and SLTT [27] . We train a VGG-SNN model on CIFAR10-DVS. And in the fine-tuning phase, TET and SLTT are used instead of SKD module. In Tab. 4, as TET does not specifically consider the relation between the time steps of SNN and the number of event frames in classification tasks on neuromorphic datasets, the performance improvement is limited at low time steps. However, when combined with our proposed HT module to learn more event frame information, we find that the performance improves by 3.22%, reaching 80.82%. However, it still falls short of HSD, mainly because HSD utilizes ANN as the teacher network and its powerful feature extraction ability. Furthermore, the performance improvement of SLTT on CIFAR10-DVS is similar to that of TET. And similar performance enhancements can be observed on N-Caltech101 and DVS-Gesture.In Tab. 1, we compare our HSD with other works and illustrate the advantages of our work. Furthermore, we consider that vanilla KD can improve the performance of student models. So we choose [41] and conduct the experiment on CIFAR10-DVS. Among them, ensuring that the training and inference time steps of SNN are consistent.As shown in Tab. 5, in response based KD, the soft labels retains the hidden information in the output of the teacher model compared to the hard labels, resulting in a 1.60% improvement compared to the baseline. Moreover, feature-based knowledge utilizes the hidden information from the intermediate layers of ANNs to actively guide the training process of SNNs. Hence, the performance improves by 2.00%, reaching 76.80%. When combined with these two KD methods, achieving an accuracy of 78.40% and improving by 3.60%. This is because it helps the student model SNN obtain richer intermediate layer features and output category probabilities from the teacher model ANN. But its performance is still lower than our HSD, because HSD can enable SNN to learn as much event frame information as possible during the training stage, while using fewer event frames to reduce the latency during the inference stage. 
conclusion: 
 In this paper, we address the issue of performance degradation in classification tasks on neuromorphic datasets when operating at lower time steps. We propose Hybrid Step-wise Distillation (HSD) method, which addresses the dependency between the number of event frames and the time steps of SNN. This method involves partitioning the SNN training process into pre-training and fine-tuning phases. Additionally, we introduce Step-wise Knowledge Distillation (SKD) module to stabilize the output distribution of the SNN at each time step, thereby achieving a balance between accuracy and latency in classification tasks on neuromorphic datasets.In future work, we aim to explore the potential of heterogeneous ANN and SNN architectures in classification tasks on neuromorphic datasets. Our goal is to achieve a better balance among accuracy, latency, and efficiency, ultimately realizing efficient event-based visual recognition. 
 
 Appendix:  


Decision: The paper proposes an innovative Hybrid Step-wise Distillation (HSD) method for low-latency event-driven visual recognition using Spiking Neural Networks (SNNs). The reviewers acknowledge the novelty and potential impact of the approach, highlighting its ability to balance accuracy and latency effectively. While there are some concerns about the explanation of technical terms, the rationale for using ANNs, and the fairness of the experimental setup, the overall consensus leans towards acceptance due to the solid empirical results and the significant contribution to the field. Therefore, I recommend accepting this submission.

Reviews: review: paper_summary: paper_summary: This paper proposes an innovative approach that uses fewer event frames during the inference stage to reduce latency. Additionally, it presents a new testing framework using partial event frames to balance accuracy and latency. Empirical evidence demonstrates that this method achieves competitive performance.
strength_weakness: strength_weakness: 1.	This paper proposes an innovative training strategy that integrates advancements from both ANNs and SNNs.
2.	It presents a new testing framework that supports future research in the SNNs field.
3.	The experiment results are solid and the paper is easy to read.
limitations: limitations: 1.	Some professional words lack explanation, such as DTS and STS.
2.	How to obtain soft labels? Why aren't soft labels applied during the pretraining stage? 
3.	In the experimental section, what is the ``response-based’’? It should be described in details.
paper_summary: paper_summary: This paper introduces a Hybrid Step-wise Distillation (HSD) method for low-latency event-driven visual recognition using Spiking Neural Networks (SNNs). The approach leverages the temporal dynamics of SNNs to efficiently process spike events, resulting in a significant reduction in recognition latency.
strength_weakness: strength_weakness: 1. By incorporating a Step-wise Knowledge Distillation (SKD) module, the issue of anomalous outputs at low temporal steps is addressed, enhancing the performance of the Spiking Neural Networks (SNNs).
2. The method achieves exceptional performance on neuromorphic dataset classification tasks, particularly at low temporal steps.
limitations: limitations: 1. The ANN-SNN conversion method achieves high accuracy but experiences noticeable latency during the inference stage.
2. The reductions in latency are primarily observable in traditional static data scenarios, but in event-based vision tasks, high accuracy has not been achieved at extremely low time steps
paper_summary: paper_summary: This paper proposes the HSD method to hybridize the ANN and SNN to improve the performance of SNN for event visual recognition. To improve the performance of the converted SNN, the authors used the ANN to distill the SNN to facilitate training. The experiments conducted by the authors on three datasets confirm the effectiveness of the proposed method.
strength_weakness: strength_weakness: This paper decouple the dependence of training event frames and inference timesteps by introducing ANNs and larger timesteps during training and using smaller timesteps during inference. Experiments show that this method is able to recognize event data with low latency, even when the timestep is 1.
limitations: limitations: 1. The motivation for using an ANN to guide an SNN for event data recognition should be elaborated. It is generally agreed that the ANN to SNN method is only able to match or exceed the performance of ANN on static datasets, while it is difficult to achieve satisfactory performance on event data because both ANNs and converted SNNs are less capable of temporal feature extraction. Using ANNs to recognize event data on a frame-by-frame basis and guide SNNs through conversion and distillation is hardly convincing.
2. In this paper, the first T1 event frames are used to train the ANN, while the last T2 event frames are used for fine-tuning the SNN and testing. Using the later T2 event frames for testing wastes the earlier T1 event frames because the test still has to process those T1 event frames, undermining the low-latency advantage of this method. In addition, why didn't the authors use the first T2 event frames for testing, so as not to cause additional test-time processing overhead.
3. The experimental results are not convincing. Table 2 shows that the CIFAR10-DVS baseline achieves 73.60% accuracy, which exceeds the accuracy of tdBN, TA-SNN, LIAF, DTSNN, and SSNN with more time steps. Given this, I think the performance of this paper comes mainly from small tricks in the experimental setup such as data sizing, data augmentation, etc., which has different practices in the SNN community. The authors should report the differences in their experimental setups when making comparisons and try to ensure a fair comparison.
paper_summary: paper_summary: Spiking neural networks are well suited for neuromorphic datasets due to their rich spatio-temporal information processing capabilities and event-driven properties. In this paper, the authors propose a hybrid stepwise distillation (HSD) approach to untangle the dependency between the number of event frames and the time step of SNNs, i.e., using more event frames in the training phase to improve the performance and fewer event frames in the inference phase to reduce the latency.
strength_weakness: strength_weakness: 1. I think the biggest contribution of this paper is to unravel the relationship between the time step of SNNs and the number of event frames. In my opinion, this idea is presented for the first time, and previous work has only focused on improving the performance of SNNs on neuromorphic datasets without considering the latency of the model in the inference phase.
2. how to achieve the trade-off between high accuracy and low latency in event-based visual recognition? This issue has been discussed in the paper of static data [1, 2], although this paper's application to a neuromorphic dataset is indeed the first of its kind.
3. the experimental part is adequate and convincing.


[1]: Li, C., Jones, E. G., & Furber, S. (2023). Unleashing the Potential of Spiking Neural Networks with Dynamic Confidence. In *Proceedings of the IEEE/CVF International Conference on Computer Vision* (pp. 13350-13360).

[2]: Li, Y., Geller, T., Kim, Y., & Panda, P. (2024). SEENN: Towards Temporal Spiking Early Exit Neural Networks. *Advances in Neural Information Processing Systems*, *36*.
limitations: limitations: I see no significant weaknesses in this paper and I am inclined to accept the paper.

Nevertheless, I have two questions for the authors to answer:
1. if the event frame is divided into two segments, i.e.  $t=1,\cdots, t=T_1$ and $t=T_1+1,\cdots,t=T_1+T_2$. In which segment is the pre-trained ANN conducted? Or is it the whole event frames? I can't figure this out in Figure 3 and Algorithm 1.
2. In the current method, the fine-tuned SNN perform inference in the second segment, i.e., $t=T_1+1,\cdots,t=T_1+T_2$. Can the time steps chosen for the fine-tuning phase be any consecutive sequence? For example, from $t=T_1-1, \cdots, t=T_1+T_2-3$. Would different choices of time steps affect the experimental results?