Title: Activity Pruning for Efficient Spiking Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: spike activity pruning

efficient model

Spiking Neural Network
Abstract: While sparse coding plays an important role in promoting the efficiency of biological neural systems, it has not been fully utilized by artificial models as the activation sparsity is not well suited to the current structure of deep networks. Spiking Neural Networks (SNNs), with their event-driven characteristics, offer a more natural platform for leveraging activation sparsity. In this work, we specifically target the reduction of neuronal activity, which directly leads to lower computational cost and facilitates efficient SNN deployment on Neuromorphic hardware. We begin by analyzing the limitations of existing activity regularization methods and identifying critical challenges in training sparse SNNs. To address these issues, we propose a modified neuron model, AT-LIF, coupled with a threshold adaptation technique that stabilizes training and effectively suppresses spike activity. Through extensive experiments on multiple datasets, we demonstrate that our approach achieves significant reductions in average firing rates and synaptic operations without sacrificing much accuracy. Furthermore, we show that our method complements weight-based pruning techniques and successfully trains an SNN with only 0.06 average firing rate and 2.22M parameters on ImageNet, highlighting its potential for building highly efficient and scalable SNN models. Code is available at https://github.com/putshua/Activity-Pruning-SNN.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper designs AT-LIF, a novel activity pruning method for SNN. The paper first discusses the two major challenges in activity regularization pruning method and then addresses these issues by introducing (1) learnable threshold for the neuron, and (2) current-based spike output. Evaluations on three datasets demonstrate the effectiveness of proposed methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. Clear discussion of challenges. The paper presents a well-motivated discussion of the existing challenges in training SNN. It analyzes conflict between minimizing spike activity and preserving task-specific performance and further identifies the gradient vanishing problem. These challenges are theoretically analyzed, providing a strong motivation for the proposed AT-LIF. \n2. The proposed method is simple and easy to implement. It addresses the identified challenges by modifying only the neuron's firing threshold and output range, without introducing complex architectural changes. This lightweight design makes the method promising for practical applications.\n\n\nWeaknesses:\n1.Insufficient comparative experiments. The paper only compares the proposed AT-LIF with a basic activation regularization baseline and two weight pruning methods. This narrow comparison makes it difficult to assess the effectiveness.\n2.Unclear explanation about conflict resolution. In Section 3.4, the authors state that “the conflict between the two terms does not necessarily arise, as the model can still effectively learn from the output representation while we simultaneously maximize the threshold to promote sparser firing events” (Line225-227). However, the paper does not explain why or how the model can simultaneously achieve effective representation learning and sparse spike activity. As a result, the justification for \"resolving the optimization conflict” is not convincing.\n3.Minor clarity issues. Eq.10 and Fig.3 (Right) are not described or referenced in the main text.\n\nQUESTIONS:\nThe following points could be addressed to further improve the clarity and impact of the work.\n\n1.The inclusion of more SOTA pruning methods is encouraged.\n2.In Section 3.4, the authors state that the model can simultaneously learn from the output representation and promote spike sparsity, thereby resolving the optimization conflict. The authors could provide a detailed explanation on why this trade-off is achievable.\n3.The proposed AT-LIF replaces binary output with a current-based output. Could the authors clarify how this design choice how this design contributes to addressing each of the two main challenges discussed in the Sec 3.1?\n4.The choice of hyperparameter \\( \\lambda_s \\) for both the AR baseline and AT-LIF appears somewhat arbitrary and varies within and across datasets without clear explanation. Could the authors clarify how these values were selected?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to improve efficiency of SNN learning while conserving effectiveness. Proposed AT-LIF algorithm reduces spiking activity using sparse regularizer and threshold adaptation. \nThe method is experimentally validated on several datasets widely used in deep learning research.\n\nSTRENGTHS AND WEAKNESSES:\nSuggested algorithm is original and ingenious. The issue of vanishing gradient is duly addressed. Experiments demonstrate significant reduction of spiking activity with minimal loss of accuracy.\nThe design assumes graded spiking, which I think should be stated upfront.\nThe clarity is lacking, some terminology unclear, and overall leaves impression of hurried writing. This makes it difficult to evaluate the technical content properly.\n\nQUESTIONS:\nEq (6) why is the output of the i-th neuron, s_i, a vector? Also, y_hat should have both i and k indices. \n\nThere 1: Why is it derivative in (8) and gradient in (9)? Are (8,9) saying the same thing? Loose index j in (8).\nZero gradient is not exactly the same as a saddle point.\nMaybe give a hint of the proof in the main text.\n\nLine 179: subscript ()_t is never used\n \nLine 196 BPTT requires memory of previous time states? Please clarify.\n\nDoesn’t it make sense to do overall evaluation first, then ablation?\n\nLine 260 what’s synaptic operation numbers?\n\nLine 271: if the threshold \\theta is constant, what’s the difference if the signal is 0-1 or 0-\\theta?\n\nLine 272: what’s synaptic output feature?\n\nLine 281: what is SOTA?\n\nTable 2: some of the best AT-LIF results are with \\lambda_s=0, i.e. no sparsity. Please comment. \nWhat is the column T?\n\nPlease give more details how your method is integrated with other methods in the experiments.\n\nLIMITATIONS:\nIf the method is implemented on a SNN platform, that platform should allow for graded spikes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper investigates the problem of high neuronal activity in SNNs.
The authors identify the limitations of current activity regularization methods and propose a new neuron model, AT-LIF, alongside a threshold adaptation mechanism, to suppress spike activity during training.
Notably, the method achieves promising results on ImageNet with a very low average firing rate and a compact model size.

STRENGTHS AND WEAKNESSES:
## Strengths
1. I think the efficiency gains are clear. The method achieves a substantial reduction in firing rate and synaptic operations.
2. The authors propose a modified neuron model (AT-LIF) and threshold adaptation to promote sparsity in SNNs, addressing a core inefficiency in SNN deployment.
3. I think the compatibility with pruning methods suggests broader applicability and system-level optimization potential.

## Weaknesses
No obvious weaknesses.

QUESTIONS:
1. Has the proposed method been evaluated on real neuromorphic hardware (e.g., Loihi, Speck)?
2. Does the introduction of threshold adaptation increase training complexity or require additional hyperparameter tuning?
3. Can the authors discuss whether the method can be applied to other domains (e.g., NLP, speech) or large architecture like spiking Transformers?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

comments: SUMMARY:
Motivated by the biological principle of sparse neural coding and the need for efficient computation, the authors propose AT-LIF, an adaptive threshold leaky integrate-and-fire neuron model. AT-LIF dynamically adjusts neuron firing thresholds to suppress spike activity while maintaining output fidelity, effectively overcoming optimization conflicts and gradient vanishing issues in training sparse SNNs. The method is evaluated on CIFAR-10, DVS-CIFAR10 ImageNet across different architectures, demonstrating significant reductions in average firing rates and synaptic operations, while preserving classification accuracy in some cases. These results highlight the method's potential for building scalable and energy-efficient SNNs.

STRENGTHS AND WEAKNESSES:
Strengths:
The paper introduces a novel activity-pruning method inspired by sparse coding in the brain. This direction departs from conventional weight-based pruning and aligns well with the natural efficiency of neural systems.
The proposed AT-LIF neuron model is both innovative and effective. It addresses two major challenges in training sparse SNNs, the gradient vanishing and optimization conflict, through adaptive thresholding and current-based output mechanisms. The approach demonstrates compatibility with existing weight-pruning techniques. The combined use of AT-LIF with methods like STDS enables both activity and weights sparsity, significantly improving computational efficiency.

Weaknesses:
The method still exhibits a clear trade-off between accuracy and sparsity. As shown in the experiments, increasing the threshold adaptation coefficient to induce lower spike activity leads to performance degradation.

QUESTIONS:
None.

LIMITATIONS:
None.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01