Title: S$^2$NN: Sub-bit Spiking Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Spiking Neural Networks

Spiking Quantization

Neuromorphic Datasets
Abstract: Spiking Neural Networks (SNNs) offer an energy-efficient paradigm for machine intelligence, but their continued scaling poses challenges for resource-limited deployment. Despite recent advances in binary SNNs, the storage and computational demands remain substantial for large-scale networks. To further explore the compression and acceleration potential of SNNs, we propose Sub-bit Spiking Neural Networks (S$^2$NNs) that represent weights with less than one bit. Specifically, we first establish an S$^2$NN baseline by leveraging the clustering patterns of kernels in well-trained binary SNNs. This baseline is highly efficient but suffers from \textit{outlier-induced codeword selection bias} during training. To mitigate this issue, we propose an \textit{outlier-aware sub-bit weight quantization} (OS-Quant) method, which optimizes codeword selection by identifying and adaptively scaling outliers. Furthermore, we propose a \textit{membrane potential-based feature distillation} (MPFD) method, improving the performance of highly compressed S$^2$NN via more precise guidance from a teacher model. Extensive results on vision reveal that S$^2$NN outperforms existing quantized SNNs in both performance and efficiency, making it promising for edge computing applications.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Sub-bit Spiking Neural Networks (S²NNs), a framework for compressing and accelerating SNNs by representing weights with less than one bit. Extensive experiments on vision and non-vision tasks demonstrate that S²NN achieves state-of-the-art performance and efficiency, making it suitable for edge computing applications.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.The paper introduces the novel concept of sub-bit weight representation in SNNs, pushing the boundaries of model compression and efficiency. The proposed OS-Quant and MPFD methods are well-motivated and address critical challenges in SNN quantization and performance preservation.\n\n2.The paper validates S²NN across diverse tasks (classification, object detection, segmentation, and NLP) and architectures, demonstrating its scalability and robustness. This paper also performs additional hardware verification.\n\nWeaknesses:\n\n1.The reliance on clustering patterns in binary kernels may not hold universally across all architectures or tasks.\n\n2.The MPFD method, while effective, introduces additional computational overhead during training, which could be a limitation for resource-constrained scenarios.\n\n3.The comparison with other sub-bit quantization methods (e.g., in non-spiking networks) is limited.\n\nQUESTIONS:\n1.The OS-Quant method detects and scales outliers based on spatial neighbors. How does this approach perform in cases where outliers are spatially isolated or when kernel sizes vary significantly (e.g., 1x1 vs. 3x3 or others)?\n\n\n2.The paper compares S²NN to BSNNs and FP-SNNs. How does it fare against state-of-the-art quantized ANNs in terms of performance and efficiency, especially for similar bit-widths?\n\n3.The results on NLP tasks (e.g., GLUE benchmark) show a performance drop compared to vision tasks. What are the key challenges in adapting S²NN to non-vision domains?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a sub-bit Spiking Neural Network framework that compresses weights to below 1-bit. The authors build upon clustering patterns in binary kernels and introduce an OS-Quant method to reduce the codeword selection bias caused by outliers. Additionally, a MPFD approach is proposed to guide the highly compressed model more effectively. The paper provides extensive experimental evidence across classification, detection, and segmentation tasks and includes FPGA hardware validations, demonstrating significant improvements in both performance and efficiency compared to state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. For the SNN domain, this paper proposes a novel sub-bit SNN compression approach that significantly reduces model size and computational cost.\n\n2. Includes hardware-oriented results, confirming the efficiency and deployment benefits of S2NN on FPGA platforms.\n\nWeaknesses:\n\n1. The paper does not provide sufficient analysis of model behavior at extremely low bit-width settings (η < 4), leaving questions about the practical lower limits of the proposed compression approach.\n\n2. The paper lacks formal theoretical analysis explaining why sub-bit quantization is particularly effective, missing an opportunity to provide deeper insights into the underlying mechanisms.\n\n3. The paper lacks detailed description of the important component compact codebookd, such as how it is initialized.\n\nQUESTIONS:\n1. Could you provide more details about how the compact codebook is initialized? Is it random or based on some clustering of pretrained weights?\n\n2. Is the codebook generation method sensitive to initialization or data distribution across datasets?\n\n3. For practical deployment, what would be a good η value that balances accuracy and efficiency? Does this vary much across different tasks? Could you provide more details about how it is determined for different layers?\n\n4. Could you elaborate on how the outlier detection threshold (γ=1.5) is chosen? Are there any sensitivity analyses performed?\n\n5. Are the outliers detected by OS-Quant common across different tasks and structures? Can they be determined in advance?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper proposes a sub-bit spiking neural network (S²NN) aimed at further exploring the compression and acceleration potential of spiking neural networks (SNNs). S²NN utilizes a compact codebook to encode weights into representations smaller than 1 bit, significantly reducing model storage and computational overhead. The authors introduce outlier-aware sub-bit weight quantization (OS-Quant), which optimizes codeword selection by identifying and adaptively scaling outliers.

STRENGTHS AND WEAKNESSES:
Strengths: The paper features a comprehensive experimental design and a clear structure. By introducing sub-bit compression into SNNs, with weight representations below 1 bit (0.44 to 0.67 bit), it significantly enhances the potential for edge deployment. The paper features thorough experimentation, covering both vision and non-vision tasks, which validates the universal applicability of the proposed method.
Weaknesses: The paper lacks an in-depth theoretical analysis of OS-Quant and MPFD (e.g., convergence of gradient propagation, and why membrane potential-based distillation provides more precise optimization directions). In the classification task, more methods could be added for comparison.

QUESTIONS:
Does knowledge distillation in the method increase additional energy consumption?

LIMITATIONS:
None.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims at addressing the storage and computational demands for large-scale spiking neural networks (SNNs). Its key idea is to employ sub-bit SNNs that represent weights with less than one bit (S2NNs). To do this, the work employs three main steps. First, it establishes an S2NN baseline that encodes weights using less than 1 bit. Second, it proposes an outlier-aware sub-bit weight quantization (OS-Quant) for improving binary kernel selection. Then third, it employs membrane potential-based feature distillation (MPFD) that aims to preserve the model’s performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n•\tIts idea in applying sub-bit concept in SNNs is interesting and holds promising efficiency benefits.  \n\n•\tThe challenges of performance degradation in realizing sub-bit SNNs are addressed systematically. \n\n•\tIt performs evaluation considering different datasets: image classification (i.e., CIFAR-10, CIFAR-100, ImageNet-1K, and DVSCIFAR-10), object detection (COCO), and semantic segmentation (ADE20K); and the results show promising improvements on model sizes, number of operations, and accuracy.\n\n\nWeaknesses\n\n•\tThe sampling process to establish the compact codebook from the full codebook is not sufficiently discussed.   \n\n•\tThe codeword selection process is somewhat still vague.\n\n•\tThe study does not completely provide power and energy consumption results for all evaluated tasks (i.e., image classification, object detection, and semantic segmentation).\n\n•\tEvaluation for bigger datasets and real-world deployment/scenarios is missing\n\n•\tThere are several issues with the technical descriptions and writing quality as highlighted in the question section below.\n\nQUESTIONS:\n1)\tProvide some evaluations for real-world deployment/scenarios, and if possible for bigger datasets as well.\n\n2)\tIt mentions that “In the case of a 3×3 kernel, an analysis of the distribution of all possible 3×3 kernel values (23×3, representing the full codebook) reveals that only a small subset of binary kernels (i.e., codewords) is frequently activated”. This should be supported with some empirical data and/or illustrations.\n\n3)\tIt also mentions that “… as networks scale up to meet practical application demands, the computational burden remains a challenge even with binary versions”. This should be supported with some empirical data to show that the existing binary versions are not enough.\n\n4)\tHow to perform the sampling process to establish the compact codebook from the full codebook? It is important to clearly describe one of the main steps of the proposed techniques.\n\n5)\tThe codeword selection process is somewhat unclear. Specifically, how the kernel of S2NN is obtained from its FP-SNN domain as shown in Figures 1-3. \n\n6)\tWhat variable “d” represents in Figures 1-3 is also not clear. It seems to be a critical part in understanding the proposed technique, especially for codeword selection. \n\n7)\tHow to decide the suitable number of bits-per-kernel (η)? \n\n8)\tThe sub-bit implementation is somewhat not clear. It is suggested to refine Figure 1(b) to illustrate the sub-bit implementation step-by-step while relating to examples in the description text. The discussion on the role of “d” may be critical here to help explaining the key steps. For this, information from appendix A.11 can be used in the main paper.  \n\n9)\tIt is also suggested to refine Figure 2 to show the steps in showing how actual outlier values from Figure 2(b) lead to outlier-induced codeword selection bias in Figure 2(a). Currently, Figure 2(a) and Figure 2(b) do not look connected to each other. \n\n10)\tWhat is the rationale of using the squared L2 criteria for identifying the outliers?\n\n11)\tThis paper does not completely provide power and energy consumption results for all evaluated tasks (i.e., image classification, object detection, and semantic segmentation). Hence, it is recommended to complete these studies as well.\n\n12)\tThere are several typos which need correction, such as “Nnormalized” in Figure 3(b) and “conclusin” in Ablation Study section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}