QP-SNN: Quantized and Pruned Spiking Neural Networks

Authors: Wenjie Wei, Malu Zhang, Zijian Zhou, Ammar Belatreche, Yimeng Shan, Yu Liang, Honglin Cao, Jieyuan Zhang, Yang Yang
Primary area: applications to neuroscience & cognitive science
Keywords: Spiking Neural Networks, Neuromorphic Computing, Spiking Pruning, Spiking Quantization

Abstract:
Brain-inspired Spiking Neural Networks (SNNs) leverage sparse spikes to encode information and operate in an asynchronous event-driven manner, offering a highly energy-efficient paradigm for machine intelligence. However, the current SNN community focuses primarily on performance improvement by developing large-scale models, which limits the applicability of SNNs in resource-limited edge devices. In this paper, we propose a hardware-friendly and lightweight SNN, aimed at effectively deploying high-performance SNN in resource-limited scenarios. Specifically, we first develop a baseline model that integrates uniform quantization and structured pruning, called QP-SNN baseline. While this baseline significantly reduces storage demands and computational costs, it suffers from performance decline. To address this, we conduct an in-depth analysis of the challenges in quantization and pruning that lead to performance degradation and propose solutions to enhance the baseline's performance. For weight quantization, we propose a weight rescaling strategy that utilizes bit width more effectively to enhance the model's representation capability. For structured pruning, we propose a novel pruning criterion using the singular value of spatiotemporal spike activities to enable more accurate removal of redundant kernels. Extensive experiments demonstrate that integrating two proposed methods into the baseline allows QP-SNN to achieve state-of-the-art performance and efficiency, underscoring its potential for enhancing SNN deployment in edge intelligence computing.

Decision: accept (poster)
Reviewer scores: 8, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1730356795218

thread_type:
official_review

<thread_object>
creator: reviewer_aeht

<content>
summary:
This paper presents QP-SNN, a novel approach to creating efficient and hardware-friendly Spiking Neural Networks (SNNs) by combining uniform quantization and structured pruning. The authors first develop a baseline model, then address its performance limitations through two key innovations: (1) a weight rescaling (ReScaW) strategy for more effective bit-width utilization in quantization, and (2) a novel pruning criterion based on the singular value of spatiotemporal spike activities (SVS). The experimental results demonstrate that QP-SNN achieves state-of-the-art performance while significantly reducing model size.
Strengths

strengths:
1. Theoretical Foundation:
- The analysis of bit-width utilization provides clear theoretical justification for the ReScaW strategy.
- The proposed SVS criterion is well-grounded in mathematical principles.

2. Method Innovation:
- The paper presents two novel techniques (ReScaW and SVS) that effectively address specific limitations in existing quantization and pruning approaches.
- The solutions are well-motivated by careful analysis of the underlying problems.


3. Experimental Validation:
- Comprehensive experiments across multiple datasets and architectures.
- Detailed ablation studies that validate each component's contribution.

weaknesses:
1.The selection process for pruning rates appears arbitrary and lacks systematic justification. 
2.This paper is only verified on the simple computer vision classification task.

questions:
1. How does the computational cost of calculating singular values in the SVS criterion compare to existing pruning criteria? Is there a significant overhead during training?

2.What is the impact of the proposed methods on the network's spike sparsity?

3.How sensitive is the method to the choice of ε in the SVS-based pruning criterion?

How does the approach perform when applied to other types of neural networks beyond classification tasks, such as object detection?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730436227562

thread_type:
official_review

<thread_object>
creator: reviewer_likb

<content>
summary:
The paper proposes two technologies including a weight-rescaling strategy and an SVS-based pruning criterion for weight quantization and structured pruning of SNN respectively. These technologies reduce the model size significantly while maintains the SNN accuracy.

strengths:
1. The paper develops a hardware-efficient and lightweight QP-SNN baseline by integrating uniform quantization and structured pruning.
2. The weight-rescaling strategy and the SVS-based pruning criterion works well on SNN benchmarks.

weaknesses:
1. The proposed technologies including the weight-rescaling strategy and the SVS-based pruning criterion do not fully utilize the unique characteristics of SNN, such as the sparsity to reduce synaptic operation and temporal information to maintain the SNN accuracy. The technologies can also be used in ANN models.
2. The comparison with SNNs that are not compressed is missing, such as SEW-ResNet[1] and MS-ResNet[2]. Compared to these works, the accuracy degradation is still large.
3. The results that the saving of synaptic operation is missing.

[1] Fang W, Yu Z, Chen Y, et al. Deep residual learning in spiking neural networks[J]. Advances in Neural Information Processing Systems, 2021, 34: 21056-21069.
[2] Hu Y, Deng L, Wu Y, et al. Advancing spiking neural networks toward deep residual learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024.

questions:
1. What is the effectiveness of this work in saving synaptic operation in SNN inference? The author can use the code in https://github.com/zhouchenlin2096/Spikingformer/tree/master/energy_consumption_calculation to provide the detailed number of synaptic operations of the compressed SNN in this work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730560491097

thread_type:
official_review

<thread_object>
creator: reviewer_lmlx

<content>
summary:
This paper presents QP-SNN, which integrates quantization and structure pruning methods. The quantization approach is based on hardware-friendly uniform quantization, while the pruning method employs SVD techniques. QP-SNN has been validated on classification datasets, achieving state-of-the-art performance.

strengths:
1. The writing of this paper is good and easy to read.

2. The experiments in the paper demonstrate the effectiveness of the proposed ReScaW algorithm and SVD-based pruning.

weaknesses:
1. The ReScaW-based uniform quantization lacks innovation, as it is merely a combination of uniform quantization and weight scaling [1].

2. The SVD-based pruning appears to be an application of SVD pruning in SNNs. There are many methods utilizing SVD decomposition or selecting important singular values for pruning [2][3]. Please clarify the specific advantages of applying this in SNNs.

3. The SVD decomposition method introduces additional time complexity. How much does it specifically affect the training time?

4. The experiments are insufficient from several perspectives. Firstly, why use the 2021 SEW-ResNet and Spiking ResNet instead of the latest SNN models? Secondly, this method has only been tested on simple classification tasks; without testing on more complex downstream tasks. Thus, it is difficult to prove the potential applicability of this work.

5. As an efficiency-focused study, why does the paper not present sufficiently robust efficiency metrics, such as OPs/SOPs, training time, inference speed, and energy consumption?

6. Why there is no presentation of pruning ratio statistics in the main text, which obscures the specific effectiveness of this aspect?

**Reference**

[1] AWQ: ACTIVATION-AWARE WEIGHT QUANTIZATION FOR ON-DEVICE LLM COMPRESSION AND ACCELERATION.

[2] Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification.

[3] HRank: Filter Pruning using High-Rank Feature Map.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601410549

thread_type:
official_review

<thread_object>
creator: reviewer_rh3c

<content>
summary:
This work focuses on the applicability of SNNs on edge devices, integrating weights quantization and structured pruning techniques, and providing an in-depth analysis of the main challenges in each respective area. Specifically, for weights quantization, the work proposes weight scaling; for structured pruning, it introduces a method to remove redundant convolutional kernels by analyzing the temporal-spatial spike activity singular values in the spike feature matrix. This is an effective joint exploration of weight quantization and pruning in SNNs. I appreciate the idea of leveraging quantization, pruning, and event-driven techniques—three approaches that accelerate, conserve energy, and enhance the efficiency of neural networks.

strengths:
1. The illustrations are thoughtfully crafted.

2. The manuscript presents a clear structure, and maintains logical consistency.

3. A detailed analysis of the weight distribution in SNNs provides the motivation for the re-scaling method.

4. The experiments demonstrate the effectiveness of this method on both static and dynamic datasets.

weaknesses:
1. Regarding the selection of the quantization scaling factor $\gamma$, this work employs three different sampling methods. Although experiments indicate that scaling with the 1-norm mean yields the best results, I believe there is still room for improvement, such as using a piecewise $\gamma$. I encourage the authors to provide further explanation.

2. To my knowledge, when two or more model lightweigh techniques are employed, compatibility issues often arise, such as the order of applying these techniques and the training strategies involved. Additional clarification is needed on these aspects.

3. At the end of Algorithm. 1, fine-tuning is required after pruning. The specific details of this fine-tuning process should be provided.

4. In Table. 2 of the experimental section, the baselines for A-F are not consistent. I suggest the authors provide a unified ablation study. Additionally, regarding the compatibility issues of interest, I encourage the authors to conduct an ablation study on the compatibility of quantization and pruning.

questions:
1.To my knowledge, the highest-performing architecture in the SNN field is the Spiking Transformer [1-5]. Please discuss whether the proposed method can be effectively applied to the Spiking Transformer. If feasible, please provide some preliminary experimental results.

[1] Zhou, Z., Zhu, Y., He, C., Wang, Y., Shuicheng, Y. A. N., Tian, Y., & Yuan, L. Spikformer: When Spiking Neural Network Meets Transformer. In The Eleventh International Conference on Learning Representations.

[2] Yao, M., Hu, J., Zhou, Z., Yuan, L., Tian, Y., Xu, B., & Li, G. (2024). Spike-driven transformer. Advances in neural information processing systems, 36.

[3] Zhou, C., Yu, L., Zhou, Z., Ma, Z., Zhang, H., Zhou, H., & Tian, Y. (2023). Spikingformer: Spike-driven residual learning for transformer-based spiking neural network. arXiv preprint arXiv:2304.11954.

[4] Zhou, Z., Che, K., Fang, W., Tian, K., Zhu, Y., Yan, S., ... & Yuan, L. (2024). Spikformer v2: Join the high accuracy club on imagenet with an snn ticket. arXiv preprint arXiv:2401.02020.

[5] Yao, M., Hu, J., Hu, T., Xu, Y., Zhou, Z., Tian, Y., ... & Li, G. Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips. In The Twelfth International Conference on Learning Representations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1745076053361

thread_type:
public_comment

<thread_object>
creator: ~changze_lv1

<content>
comment:
Dear Authors,

Greetings! I greatly appreciate your work and believe it holds significant value in the field of quantization for Spiking Neural Networks. I also noticed your previous paper, Q-SNNs: Quantized Spiking Neural Networks, accepted at ACM MM 2024, which I find very forward-looking and insightful.

Unfortunately, I couldn't find the released code for either of these two papers. May I ask if you have any plans to make the code publicly available?

Thank you very much!
</content>
</thread_object>

</thread>
<thread>
creation date:
1756875140054

thread_type:
public_comment

<thread_object>
creator: ~do_thanh_dat1

<content>
comment:
Dear Authors,

Thank you very much for your excellent work on quantization and pruning for Spiking Neural Networks. I find your research very inspiring and would like to study it in more detail.
May I ask if there is a publicly available version of the source code, or if you plan to release it in the future?

Thank you for your time and support!
</content>
</thread_object>

</thread>
<thread>
creation date:
1763974307029

thread_type:
public_comment

<thread_object>
creator: ~linliang_chen2

<content>
comment:
Dear Authors,

I have read your paper on quantization and pruning for Spiking Neural Networks with great interest. The approach you presented is highly valuable to my current research.

I would be most grateful if you could share the implementation code with me at your convenience. If the code is not yet ready for distribution, I would appreciate any information about potential future release plans.

Thank you very much for your time.

Warm regards,
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a hardware-friendly and lightweight SNN, aimed at effectively deploying high-performance SNN in resource-limited scenarios. After the response, it receives one borderline reject, one borderline accept, and two accept. For reviewer Likb, though the rating is 5, the concerns are well addressed. The strengths of the paper, including the clear motivation, interesting ideas, and good results are well recognized. I agree with them and think the current manuscript meets the requirements of this top conference. Please also incorporate the response in the revised paper.