Title: Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training
Venue: NeurIPS 2025
Accepted: yes
Keywords: Spiking Neural Networks

Self-Distillation.
Abstract: Spiking Neural Networks (SNNs) exhibit exceptional energy efficiency on neuromorphic hardware due to their sparse activation patterns. However, conventional training methods based on surrogate gradients and Backpropagation Through Time (BPTT) not only lag behind Artificial Neural Networks (ANNs) in performance, but also incur significant computational and memory overheads that grow linearly with the temporal dimension. To enable high-performance SNN training under limited computational resources, we propose an enhanced self-distillation framework, jointly optimized with rate-based backpropagation. Specifically, the firing rates of intermediate SNN layers are projected onto lightweight ANN branches, and high-quality knowledge generated by the model itself is used to optimize substructures through the ANN pathways. Unlike traditional self-distillation paradigms, we observe that low-quality self-generated knowledge may hinder convergence. To address this, we decouple the teacher signal into reliable and unreliable components, ensuring that only reliable knowledge is used to guide the optimization of the model. Extensive experiments on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet demonstrate that our method reduces training complexity while achieving high-performance SNN training. Our code is available at https://github.com/Intelli-Chip-Lab/enhanced-self-distillation-framework-for-snn.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an enhanced self-distillation framework for efficient training of Spiking Neural Networks (SNNs). The key ideas include projecting the firing rates of intermediate SNN layers onto lightweight ANN branches and using a reliability-separated self-distillation strategy to ensure that only high-quality knowledge guides the optimization process.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1.The proposed enhanced self-distillation framework and the idea of using rate-based backpropagation combined with a reliability-separated self-distillation strategy are innovative and addresses a critical issue in traditional self-distillation methods where unreliable teacher labels can hinder convergence.\n2.The authors provide extensive experimental results on multiple datasets, demonstrating the effectiveness of their method. The improvements in accuracy, memory consumption, and training time compared to BPTT and other methods are substantial and well-documented.\n3.This paper presents a detailed theoretical analysis of the effects of reliable and unreliable predictions of teacher signals on student models.\nWeaknesses:\n1.The reliability separation strategy proposed in the article may ignore some samples that have potential value in the early stage of training but are currently misclassified in practical operation. For example, in Table 6, when the β value increases from 0.3 to 0.5, although the performance of the classifier in the middle layer improves, the performance of the final classifier begins to decline. This indicates that simply filtering out misclassified samples may cause the model to miss some important learning signals.\n2.It is mentioned in the text that the auxiliary branch uses depth-separable convolution to reduce the computational complexity. However, although this design has advantages in computational efficiency, it may be insufficient in feature extraction ability.\n3.The text mentions the formation of teacher labels by filtering out samples with correct predictions (see Section 3.4), but does not elaborate on how to define the criteria for \"correct predictions.\" For example, is it based on the classification accuracy rate or the confidence threshold?\n\nQUESTIONS:\n1.The paper proposes an enhanced self-distillation framework but does not analyze its convergence in detail. It is recommended that the authors theoretically analyze the convergence rate and convergence conditions of the proposed method, especially the effect on convergence when dealing with unreliable teacher signals.\n2.The influence of different β values on the model performance is shown in Table 6 in the text. It indicates that this method is relatively sensitive to the selection of β values and there exists an optimal range of β values. However, the specific method on how to select the optimal β value is not provided in the text.\n3.The author should consider whether a dynamic reliability assessment mechanism can be introduced, such as how to adjust the reliability assessment criteria of teacher signals as the training progresses.\n4.In Section 3.2, the author mentions the use of eligibility traces to approximate gradients in backpropagation. However, the accuracy and potential errors of this approximate method are not analyzed in detail in the text.\n\nLIMITATIONS:\nPlease reference Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a self-distillation framework based on the RateBP training framework, introducing a lightweight ANN auxiliary branch in the intermediate layers of the SNN to provide more accurate gradients for the intermediate layers, thereby alleviating the gradient distortion problem. In addition, the authors propose the ESD method, which determines the reliability of all branches’ output signals, and combines them into a final high-quality teacher signal for guiding the learning of the auxiliary branch. This paper achieves significant performance improvements and reduced training costs on all conducted datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n1.The paper achieves a simultaneous increase in training efficiency and network performance, representing an excellent attempt to solve the training problem of SNN. \n\n2.This paper identifies potential problem with standard self-distillation in SNN and proposes a simple and effective solution,\n\nWeakness\n\n1.The paper lacks a detailed discussion of the auxiliary branches, such as the structure of the auxiliary branches, the insertion location in the main branch, and the relationship between various choices of auxiliary branches and the performance improvements.\n\n2.Figure 2 is missing the caption. The current caption is a placeholder paragraph.\n\n3.This paper need to provide an in-depth theoretical analysis (or experiment) of how gradient errors are corrected by auxiliary branches.\n\n4.The definition of reliability may be overly simple. Sometimes, misclassification output can contain more useful information than correct classification output. The paper does not further explore more refined reliability measurement methods, which may still have significant room for improvement.\n\n5.The contribution 1 of the paper seems to derive from RateBP method, and a comparative analysis with it should be added.\n\n6.In Table 7, the BPTT memory cost at T=12 is excessively large. This may be caused by a transcription error.\n\nQUESTIONS:\n1.The method in the paper is based on the soft reset LIF model. Can it be applied to the hard reset rules of LIF?\n\n2.In Figure 3 (b), according to the paper, self-distillation alleviates the gradient error problem. Why are methods like MB, ASD, and ESD still effective on ANNs where there is no gradient error?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel training framework for Spiking Neural Networks (SNNs) that enhances performance and efficiency by integrating self-distillation with rate-based backpropagation. The key idea is to project firing rates from intermediate SNN layers into lightweight ANN branches, enabling substructure optimization via self-generated knowledge. To address the issue of unreliable knowledge in standard self-distillation, the authors introduce a decoupling mechanism that filters out low-quality guidance, ensuring only reliable signals influence training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Improved Training Efficiency: The proposed approach shows noticeable reductions in memory usage and training time on standard benchmarks, which may be beneficial for resource-constrained SNN training scenarios.\n2. Refined Self-Distillation Strategy: The idea of decoupling reliable and unreliable self-generated knowledge offers a thoughtful refinement to conventional self-distillation methods, potentially improving training stability.\n3. Good Empirical Coverage: The method is evaluated on multiple datasets, including both static and neuromorphic benchmarks, and shows some promise for generalization, including preliminary applicability to ANN models.\n\nWeaknesses:\n1. Limited Exploitation of Temporal Dynamics: The proposed method does not fully leverage the intrinsic temporal modeling capabilities of SNNs. Instead, it primarily relies on ANN-based distillation mechanisms and rate-based approximations to enhance performance, which may overlook the rich temporal information encoded in precise spike timing that is critical for neuromorphic tasks.\n2. Inconsistency Between SNN and ANN Pathways: Projecting SNN firing rates into ANN branches introduces a modality mismatch, as the ANN surrogate may not faithfully preserve the temporal characteristics of spiking activity.\n\nQUESTIONS:\n1. Could the authors clarify whether the proposed method is able to preserve any spike timing information during the knowledge transfer process, or is all temporal structure reduced to rate-based statistics?\n2. Could the proposed distillation framework remain effective if the auxiliary ANN branches were replaced with SNN-based substructures, thereby maintaining a fully spike-based architecture?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes an enhanced self-distillation framework for efficient Spiking Neural Network training, addressing the high memory and time costs of Backpropagation Through Time. The experiments on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet demonstrate that the method reduces training complexity while achieving high-performance SNN training.

STRENGTHS AND WEAKNESSES:
Strength:
1. The paper proposed a novel method for efficient spiking neural network training.

2. This method has good empirical results. It outperforms previous methods on static/neuromorphic datasets

Weakness:
1. Is this truly self-distillation? Since it involves SNN-to-ANN conversion before distillation.

2. The method's effectiveness remains unverified for modern spiking neural architectures like Spiking Transformers.

3.Insufficient literature review on self-distillation. The paper missed some key references and experiment comparisons with previous self-distillation methods, weakening the contextual grounding of the proposed approach.

QUESTIONS:
In addition to the questions above, there are also the following points：

1. Unclear notation. Some symbols lack proper definitions and explanations. Such as the meanings of $L_{KD}^{good}$ and $L_{KD}^{bad}$ in Eq(7) are missing.

2. Formatting Issues: The highest performance values in the tables should be highlighted in bold for better readability. Table 3 should include percentage signs (%) for clarity.

3. Can you give more details about how to decouple the teacher signal into reliable and unreliable components in your methods?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03