Improving the Sparse Structure Learning of Spiking Neural Networks from the View of Compression Efficiency

Authors: Jiangrong Shen, Qi Xu, Gang Pan, Badong Chen
Primary area: applications to neuroscience & cognitive science
Keywords: spiking neural networks

Abstract:
The human brain utilizes spikes for information transmission and dynamically reorganizes its network structure to boost energy efficiency and cognitive capabilities throughout its lifespan. Drawing inspiration from this spike-based computation, Spiking Neural Networks (SNNs) have been developed to construct event-driven models that emulate this efficiency. Despite these advances, deep SNNs continue to suffer from over-parameterization during training and inference, a stark contrast to the brain’s ability to self-organize. Furthermore, existing sparse SNNs are challenged by maintaining optimal pruning levels due to a static pruning ratio, resulting in either under or over-pruning.
In this paper, we propose a novel two-stage dynamic structure learning approach for deep SNNs, aimed at maintaining effective sparse training from scratch while optimizing compression efficiency. 
The first stage evaluates the compressibility of existing sparse subnetworks within SNNs using the PQ index, which facilitates an adaptive determination of the rewiring ratio for synaptic connections based on data compression insights. In the second stage, this rewiring ratio critically informs the dynamic synaptic connection rewiring process, including both pruning and regrowth. This approach significantly improves the exploration of sparse structures training in deep SNNs, adapting sparsity dynamically from the point view of compression efficiency.
Our experiments demonstrate that this sparse training approach not only aligns with the performance of current deep SNNs models but also significantly improves the efficiency of compressing sparse SNNs. Crucially, it preserves the advantages of initiating training with sparse models and offers a promising solution for implementing Edge AI on neuromorphic hardware.

Decision: accept (spotlight)
Reviewer scores: 3, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1729499480548

thread_type:
official_review

<thread_object>
creator: reviewer_u3ak

<content>
summary:
This paper proposed an two-stage sparse structure learning framework for SNNs, which is combined with the PQ index technique and can be implemented through a continuous and iterative learning process.

strengths:
1. The authors provided a detailed algorithm pseudo-code for their sparse training strategy.
2. It is interesting to combine the sparse structure learning framework of SNNs with PQ index and rewiring techniques.
3. The authors considered various rewiring scopes, including layer-wise and neuron-wise schemes.

weaknesses:
1. The authors did not comprehensively report the performance of previous works in Table 1. Furthermore, it seems that their performance lags behind previous works. For example, the authors claim that UPR [1] achieves 78.3% (Acc.), 0.77% (Conn.) and 1.81M (Param.) on DVS-CIFAR10, VGGSNN. However, as mentioned in [1], the original text indicates that UPR can also achieve **81.0%** (Acc.), **4.46%** (Conn.) and **2.50M** (Param.) on DVS-CIFAR10, VGGSNN. In addition, STDS [2] can achieve **79.8%** (Acc.), **4.67%** (Conn.) and **0.24M** (Param.). 
In comparison, this work achieves **78.4%** (Acc.), **30%**(Conn.) and **2.76M** (Param.) under the same experimental condition, which is inferior to the previous results.
2. Compared to previous work [1], the authors lack experimental results for SOPs (Synaptic Operations) and power saving ratio.
3. Compared to previous works [1, 2], the authors lack persuasive experimental results on large-scale datasets (e.g. ImageNet-1k).

[1] Shi X, et al. Towards energy efficient spiking neural networks: An unstructured pruning framework. ICLR 2024.

[2] Chen Y, et al. State transition of dendritic spines improves learning of sparse spiking neural networks. ICML 2022.

questions:
See Weakness Section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730101529076

thread_type:
official_review

<thread_object>
creator: reviewer_zymd

<content>
summary:
This paper proposes a dynamic pruning framework for spiking neural networks (SNN). It divides each training iteration into two stages to update the weights and structure separately. It utilizes the PQ index to gauge and dynamically adjust structure of sparse subnetworks according to compressibility. Experimental results show that the proposed method achieves competitive performance while reducing redundancy.

strengths:
1. The proposed method achieves competitive or better performance than the dense model with fewer connections and reduced redundancy.
2. It is a novel idea to dynamically calculate the rewiring ratio.

weaknesses:
1. The details of the training process is unclear. In Figure 2, the density decreases as the iteration increases. However, as described in Algorithm 1, the remove fraction and the regrow fraction are both $c_i$, therefore the density should keep constant. Please explain why the density decreases and describe the training process in detail.
2. The proposed method requires multiple training iterations. Does it require longer training time as compared to existing methods? Please compare the training time with existing methods.

Minor:

1. Duplicate definition of $c_i$ in Algorithm 1: both labels of input data and rewiring ratio.
2. In Equation 1: undefined symbol $n^k$, you mean $n_k$? Symbol $\epsilon$ is also undefined.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730344668750

thread_type:
official_review

<thread_object>
creator: reviewer_mlns

<content>
summary:
This paper addresses over-parameterization in deep spiking neural networks (SNNs) during both training and inference, aiming to achieve an optimal pruning level. The authors propose a novel two-stage dynamic structure learning approach for deep SNNs.

strengths:
Pruning is essential for making SNNs more practical for neuromorphic hardware, distinguishing it from artificial neural network (ANN) pruning, which is often GPU-oriented.

weaknesses:
The paper could benefit from consistent citation formatting. For instance, "Hu et al. (2024)" works well at the beginning of a sentence, whereas "(Hu et al., 2024)" fits better at the end. Maintaining good formatting would enhance readability.

The rationale for using a PQ index in the second stage is unclear. Although PQ is a general technique, it would help to see a clearer connection to SNNs specifically and how it improves this work beyond its original intent. Not clear how PQ index is related to SNN.

The paper lacks an analysis of energy consumption, which is essential for evaluating SNNs’ efficiency, particularly in neuromorphic applications.

There’s no substantial improvement in accuracy or network connectivity, which may limit the paper’s impact.

questions:
What is unique about the proposed work for SNNs specifically? Additionally, given the strong performance of attention-based and Transformer-based SNNs in the field, could this method also be applied effectively to those models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646377956

thread_type:
official_review

<thread_object>
creator: reviewer_a7kg

<content>
summary:
The authors present a two-stage dynamic pruning method tailored for deep Spiking Neural Networks (SNNs). The first stage evaluates the compressibility of existing sparse subnetworks within SNNs using the PQ index. In the second stage, synaptic connections are dynamically adjusted. The authors assert that this method can significantly reduce the network's sparsity. While the paper's motivation is commendable, there are shortcomings in both the presentation and the novelty of the work.

strengths:
1. The authors apply adaptive pruning and sparse training methods to deep SNNs, which can further enhance the energy efficiency of these networks. This initiative is worthy of recognition.
2. The two-stage sparse training and learning method proposed by the authors appears to be feasible based on the experimental results presented.

weaknesses:
1. To my knowledge, adaptive pruning and sparse training are well-established in ANNs. Existing works such as RigL [1]  seem to share similar ideas with the authors' approach. Additionally, DSR [2] also adjusts the rewiring rate. The authors have not provided targeted adaptations of these strategies to accommodate the binary spiking characteristics unique to SNNs.
2. The paper mentions the PQ index but does not provide an explanation, instead merely offering a citation. This could leave readers confused about what the PQ index represents and how it is calculated.
3. The methodology section lacks theoretical discussion. There is no explanation of why these methods are particularly beneficial or suitable for SNNs.
4. The size of Figure 3 is noticeably inconsistent with the other figures, which affects the overall presentation quality.

References:
[1] Evci, U., Gale, T., Menick, J., Castro, P. S., & Elsen, E. (2020). Rigging the Lottery: Making All Tickets Winners. International Conference on Machine Learning (ICML).
[2] Mostafa, H., & Wang, X. (2019). Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization. International Conference on Machine Learning (ICML)

questions:
1. Can the authors explain why the proposed method is specifically suited to SNNs rather than Binary Neural Networks (BNNs)? Have special adaptations been made to leverage the unique characteristics of SNNs, rather than applying ANN techniques to SNNs?
2. I recommend that the authors include additional figures to further illustrate their method. This would greatly aid reader comprehension.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a two-stage dynamic structure pruning framework for deep Spiking Neural Networks and offers an innovative approach to adjust the rewiring ratio. The experimental results demonstrate that the proposed method achieves competitive performance while effectively reducing redundancy. While some concerns were raised about the similarity to existing methods and the lack of theoretical analysis to support its effectiveness, the authors have adequately addressed these issues in their response. Overall, the paper presents a valuable framework for training sparse models and provides a promising solution for deploying efficient SNNs on edge devices.