Q-SNNs: Quantized Spiking Neural Networks

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 05:53:05

Abstract: Brain-inspired Spiking Neural Networks (SNNs) leverage sparse spikes to represent information and process them in an asynchronous event-driven manner, offering an energy-efficient paradigm for the next generation of machine intelligence. However, the current focus within the SNN community prioritizes accuracy optimization through the development of large-scale models, limiting their viability in resource-constrained and low-power edge devices. To address this challenge, we introduce a lightweight and hardware-friendly Quantized SNN (Q-SNN) that applies quantization to both synaptic weights and membrane potentials. By significantly compressing these two key elements, the proposed Q-SNNs substantially reduce both memory usage and computational complexity. Moreover, to prevent the performance degradation caused by this compression, we present a new Weight-Spike Dual Regulation (WS-DR) method inspired by information entropy theory. Experimental evaluations on various datasets, including static and neuromorphic, demonstrate that our Q-SNNs outperform existing methods in terms of both model size and accuracy. These state-of-the-art results in efficiency and efficacy suggest that the proposed method can significantly improve edge intelligent computing.

Main paper: 
 introduction: 
 Brain-inspired spiking neural networks (SNNs) have emerged as a promising approach for the next generation of machine intelligence. In contrast to traditional deep neural networks (DNNs), the spiking neuron in SNNs transmits information through sparse and binary spikes, which converts the computationally intensive multiply-accumulate (MAC) operations into computationally efficient accumulate (AC) operations. This energy-efficient characteristic of SNNs has triggered a growing interest in the design of neuromorphic hardware, including SpiNNaker [30] , TrueNorth [1] , Loihi [9] , and Tianjic [31] , etc. These neuromorphic hardware draw upon the storage and computing paradigms of the human brain, enabling the energy efficiency advantages of SNNs to be further demonstrated. However, despite the energy efficiency of SNNs, their performance in practical applications requires improvement.In recent years, numerous studies have dedicated efforts to crafting expansive and complex network architectures for SNNs, leading to notable enhancements in performance across various challenging tasks such as image classification [16, 48] , language generation [3, 53] , and object recognition [20, 41] . Despite their commendable performance, these studies come at the cost of massive model parameters and high computational complexity. As a consequence, these large-scale SNNs sacrifice the inherent energy efficiency advantages that are closely associated with SNNs, thereby presenting challenges for their efficient deployment on real-world resourceconstrained edge devices.There is a growing body of research investigating compression techniques for large-scale SNNs. These techniques include pruning [7, 11, 49] , knowledge distillation [23, 42, 47] , neural architecture search [27, 33] , and quantization [11, 18, 19, 50] . Quantization techniques have attracted considerable interest due to their ability to convert synaptic weights from a high-precision floating-point representation to a low bit-width integer representation. This conversion facilitates efficient deployment on resource-constrained devices by enabling the use of lower-precision arithmetic units, which are both cheaper and more energy-efficient. Binarization within this domain stands out for its hardware-friendly feature, which restricts data to two possible values (-1 and +1), leading to significant efficiency in terms of storage and computation. However, existing binarization approaches in SNNs focus solely on quantizing synaptic weights, overlooking the memory-intensive aspect associated with membrane potentials [18, 32, 34, 38, 43] . Therefore, there remains room for further efficiency optimizations. While prior lightweight research has explored the quantization of membrane potentials [50] , the synaptic weights in this work remain non-binary, which results in a performance gap compared to full-precision SNN models.This paper introduces a lightweight spiking neural network (SNN) architecture called Quantized SNN (Q-SNN) which prioritizes energy efficiency while maintaining high performance. We achieve this by quantizing two key elements within the network, namely synaptic weights and membrane potentials. This targeted quantization aims to maximize the energy efficiency of the proposed architecture. In addition, to further enhance performance within Q-SNNs, we leverage the principles of information entropy by introducing a novel Weight-Spike Dual Regulation (WS-DR) method. This method aims to maximize the information content within Q-SNNs, ultimately leading to improved accuracy. The main contributions of this work are summarized as follows:• We introduce a novel SNN architecture, called Q-SNN, designed for efficient hardware implementation and low energy consumption. Q-SNN achieves this goal by employing the quantization technique on two key elements of the network: (1) synaptic weights using binary representation and (2) membrane potentials using low bit-width representation. This targeted quantization significantly improves the efficiency of the network.• We analyze how to enhance the performance of Q-SNNs from the theory of information entropy and propose a novel Weight-Spike Dual Regulation (WS-DR) method. Inspired by information entropy theory, WS-DR adjusts the distribution of both weights and spikes within the network. This method maximizes the information content within Q-SNNs and results in improved accuracy. • We conduct comprehensive experiments on various benchmark datasets, including static and neuromorphic, to validate the efficiency and effectiveness of our method. Experimental results demonstrate that our method achieves state-of-theart results in terms of both efficiency and performance, underscoring its capability to boost the development of edge intelligent computing. [8, 11, 49] . The second category draws inspiration from biological processes in the human brain. These bio-inspired pruning algorithms model the synaptic regrowth process to achieve network compression [5, 6] . Knowledge distillation is a technique that enables the transfer of knowledge from a larger pre-trained model to a smaller model. Existing knowledge distillation methods for SNNs can also be grouped into two main categories. The first category uses the knowledge from pre-trained ANNs or SNNs to guide the training process of smaller student SNN models [23, 42, 47] . However, these methods require the additional computational overhead of training a large-scale teacher model. The second category, referred to as self-distillation [13, 14] , eliminates the need for a separate teacher model. In self-distillation, the model acts as its own teacher and generates the necessary supervisory signals for training. This approach reduces computational costs by eliminating the requirement for training an additional teacher model. Neural Architecture Search (NAS) is an automated approach for devising high-performing neural network structures under resource constraints. Early NAS applications in SNNs were limited by their focus on exploring pre-defined network blocks, thereby hindering the identification of potentially optimal designs beyond these predefined constraints [29] . To address this limitation, researchers have developed algorithms to search for optimal network structures across the entire network. These advancements aim to enable the design of more flexible and potentially more efficient SNN models [4, 21, 27] . 
related work: 
 Quantization is a powerful technique for compressing SNN models. Unlike traditional ANNs which use real-valued activations, SNNs communicate through binary spikes. Consequently, the main energy consumption in SNNs is attributed to the use of floatingpoint synaptic weights. To address this issue, existing quantized SNNs mainly focus on reducing the bit-width of weights, typically down to 2, 4, or 8 bits [7, 11, 40] . However, further reduction to a single bit, i.e. binary weights, often leads to significant performance degradation or even failure of training convergence. This challenge has motivated research on specifically designed binary SNNs. For instance, Wang et al. [43] proposed a weights-thresholds balance conversion method to achieve SNNs with binary weights. Qiao et al. [34] used the surrogate gradient (SG) method to train binary SNNs for efficient processing of event-based data. Building upon this work, Jiang et al. [18] introduced a Bayesian-based learning algorithm that demonstrates superior accuracy and calibration compared to the SG method for binary SNNs. In addition, Kheradpisheh et al. [19] proposed a temporal-based binary SNN where each neuron fires at most once with learning taking place only upon spike emission. While these binary SNN approaches offer significant efficiency advantages, they still have limitations. Firstly, these methods focus solely on quantizing synaptic weights, neglecting the memory footprint associated with membrane potentials. This presents an opportunity for further efficiency improvements. Secondly, binary SNNs often suffer from severe information loss during inference, resulting in a substantial performance gap when compared to their full-precision SNN counterparts.In order to overcome these limitations, we introduce the Quantized SNN (Q-SNN) that quantizes both synaptic weights and membrane potentials to maximize efficiency. Furthermore, to mitigate the performance gap between Q-SNN and full-precision SNNs, we leverage the principles of information entropy and propose a novel Weight-Spike Dual Regulation (WS-DR) method. By integrating WS-DR, we can train efficient and high-performance Q-SNNs from scratch, paving the way for deploying SNNs on resourceconstrained devices. 
preliminary 3.1 leaky integrate-and-fire model: 
 SNNs rely on spiking neurons as their fundamental processing units. These models aim to replicate the information processing capabilities of biological neurons. Several prominent examples include the Hodgkin-Huxley [15] , Izhikevich [17] , and Leaky Integrate-and-Fire (LIF) models [45] . Due to its computational efficiency, we adopt the LIF model. Its membrane potential, a key element of a neuron's firing behavior, is mathematically described as,EQUATIONwhere 𝜏 is the constant leaky factor, 𝑤 𝑙 𝑖 𝑗 is the synaptic weight between neuron 𝑗 in layer 𝑙 -1 and neuron 𝑖 in layer 𝑙, and 𝑠 𝑙 -1 𝑗 [𝑡] is the input spike from presynaptic neuron 𝑗 at time 𝑡. Neuron 𝑖 integrates inputs and emits a spike when its membrane potential exceeds the firing threshold. Mathematically, the spike generation function is stated as,EQUATIONwhere 𝜃 denotes the firing threshold parameter. Following each spike emission, the spiking neuron 𝑖 undergoes a reset mechanismClassification Loss: that updates its membrane potential. This reset process is mathematically defined as,=1 ℒ o[t], y Spike Regulation Loss: ℒ Loss Function Synaptic Weight -1 .6 1 .8 -1 1 -1 -1 2.1 0.3 -1 1 -1 1 -1 0.2 - 0.6 - 1.8 1 - 0.3 1 -1 -1 1 1 -1 -1 1 -1 1 LIF Spiking Neuron ⋮ Weight Regulation 0.2 0.6 1.2 1.8 0.4 0.3 -0.1 1.3 2.1 0.3 -0.7 0.2 0.3 0.7 0.4 0.2 -0.6 -1.8 1.1 -0.3 1.4 -1 0.6 1 1.8 -1 1 -1 -1 2.1 0.3 -1 1 -1 0.4 0.1 0.5 0.4 0.2 -0.6 -1.8 -0.3 Sign Function Q-SNN WS-DR ⋮ ⋮ ⋮ ⋯ ⋯ ⋯ ⋮ ⋮ ⋮ ⋮ ⋯ ⋯ ⋯ ⋮ ⋮ ⋮ ⋮ ⋯ ⋯ ⋯ ⋮ ⋮ ⋮ ⋮ ⋯ ⋯ ⋯ ⋮ ⋯ ⋯ ⋯ ⊕ Before Learning ⋮ ⋮ ⋮ ⋯ ⋯ ⋯ ⋮ ⋮ ⋮ ⋮ ⋯ ⋯ ⋯ ⋮ ⋮ ⋮ ⋮ ⋯ ⋯ ⋯ ⋮ ⋮ ⋮ ⋮ ⋯ ⋯ ⋯ ⋮ After Learning ⋮ ⋮ ⋮ ⋯ ⋯ ⋮ ⋮ ⋮ ⋮ ⋯ ⋯ ⋯ ⋮ ⋮ ⋮ ⋮ ⋯ ⋯ ⋯ ⋮ ⋮ ⋮ ⋮ ⋯ ⋯ ⋯ ⋮ 0.6 -0.5 -0.1 -0.2 0.5 0.6 -0.2 0.1 0.4 1.3 -1.8 -0.3 -0.6 0.9 1.4 -0.2 0.5 0.7EQUATIONThis work uses the hard reset mechanism where the membrane potential of neuron 𝑖 is reset to zero upon emitting a spike and remains unchanged in the absence of a spike.Training of SNNs requires calculating the gradient of the loss function with respect to the synaptic weight. The chain rule provides a powerful tool to achieve this. By applying the chain rule, the derivative of the loss function 𝐿 with respect to the synaptic weight 𝑤 𝑙 𝑖 𝑗 can be decomposed into the following equation,𝜕𝐿 𝜕𝑤 𝑙 𝑖 𝑗 = 𝑇 ∑︁ 𝑡 =1 𝜕𝐿 𝜕𝑠 𝑙 𝑖 [𝑡] 𝜕𝑠 𝑙 𝑖 [𝑡] 𝜕𝑢 𝑙 𝑖 [𝑡] 𝜕𝑢 𝑙 𝑖 [𝑡] 𝜕𝑤 𝑙 𝑖 𝑗 + 𝜕𝐿 𝜕𝑢 𝑙 𝑖 [𝑡 +1] 𝜕𝑢 𝑙 𝑖 [𝑡 +1] 𝜕𝑢 𝑙 𝑖 [𝑡] 𝜕𝑢 𝑙 𝑖 [𝑡] 𝜕𝑤 𝑙 𝑖 𝑗 . (4)However, training SNNs presents a distinct challenge compared to traditional ANNs and Deep Neural Networks (DNNs) due to the nondifferentiable nature of the spiking (i.e. firing) mechanism. Specifically, the term 𝜕𝑠 𝑙 𝑖 [𝑡]/𝜕𝑢 𝑙 𝑖 [𝑡] represents the gradient of the spike generation function (described in Eq. 2). This function evaluates to infinity at the moment of spike emission and to zero elsewhere, making it incompatible with the traditional error backpropagation used in ANN/DNN training. To tackle this issue, existing studies employ surrogate gradients to approximate the true gradient [45] . These surrogate gradients take various shapes, including rectangular [46] , triangular [12] , and linear [44] . In this paper, we use the triangular-shaped surrogate gradient which is mathematically defined as,EQUATIONwhere 𝛽 is the factor that defines the range of gradient computation. 
method: 
 This section introduces the core aspects of our work. First, we present the design of the proposed lightweight Q-SNN architecture, focusing on the key element of quantizing both synaptic weights and membrane potentials for improving efficiency. Then, we explore methods for enhancing its performance through an information theory-based analysis. Leveraging the principles of information entropy, we present a novel Weight-Spike Dual Regulation (WS-DR) method to maximize the information content within Q-SNNs, ultimately leading to improved accuracy.In order to exploit the energy efficiency benefit inherent to SNNs, we introduce a Quantized SNN (Q-SNN). As shown in Figure 1 , the first characteristic of the proposed Q-SNNs is to quantize the synaptic weight. The synaptic weights in SNNs are commonly represented as 32-bit values, which often encounter challenges such as large storage demands, increased computational complexity, and high power consumption. Therefore, in Q-SNNs, we quantize the synaptic weight into a 1-bit representation, which is formulated as,EQUATIONEQUATIONwhere 𝑤 indicates the 32-bit weight, sign(𝑤) represents the binarization operation of obtaining the 1-bit weight, and 𝛼 𝑤 denotes the scaling factor for synaptic weights. The binarization operation sign(𝑤) typically suffers from severe information loss, resulting in performance degradation of Q-SNNs. To overcome this issue, a channel-wise scaling factor 𝛼 𝑤 is introduced to mitigate the impact of information loss, and it is calculated as the average of the absolute value of weights in each output channel [35] .The second characteristic of the proposed Q-SNNs is to quantize the memory-intensive membrane potentials in SNNs. As synaptic weights are reduced to a specific bit width, the membrane potential begins to emerge as the predominant factor in memory storage and computational overhead [50] . Therefore, the membrane potential plays a critical role in improving efficiency, but it is neglected in existing binary SNNs. Building upon this, we quantize the membrane potential within Q-SNNs to a low bit-width integer, with the quantization function described as,EQUATIONEQUATIONwhere 𝑢 is the 32-bit membrane potential, 𝑘 denotes the number of bits assigned to the quantized integer, 𝛼 𝑢 is the scaling factor for membrane potentials, round (•) is a rounding operator, and clip(•) is a clipping operator saturating 𝑥 within the range [-1, 1]. In this paper, 𝑘 is set to 2/4/8, and a layer-wise scaling factor 𝛼 𝑢 is determined as the maximum value in each layer. By integrating Eq. 6 and Eq. 8 into Eq. 1, the membrane potential of the LIF neuron model can be represented as,EQUATIONIn this equation, it can be observed that the membrane potential 𝑢 𝑙 𝑖 [𝑡 -1] is quantized to a low bit-width, thereby requiring diminished storage demands. Furthermore, through the integration of binary spikes and binary weights, the proposed Q-SNNs can leverage cost-effective bitwise operations, i.e., ⊕, to execute the convolutional operation, thereby attaining enhanced computational efficiency in the inference process. In summary, the proposed Q-SNN architecture maximizes the efficiency benefit inherent to SNNs.While the proposed Q-SNNs exhibit significant energy efficiency, it must be acknowledged that their task performance lags significantly behind that of full-precision SNNs. This performance gap can be attributed to the limited information representation capability of Q-SNNs due to the low-precision weights and spiking activities. To address this challenge, we explore into the concept of information entropy and analyze how it can be leveraged to enhance the performance of Q-SNNs.In Q-SNNs, the synaptic weight and spike activity are binary values, i.e., 𝑤 ∈ {-1, 1} and 𝑠 ∈ {0, 1}, both of them follow the Bernoulli distribution. Taking the binary spike 𝑠 as an example for analysis, its probability mass function can be expressed as,EQUATIONwhere 𝑝 𝑠 ∈ (0, 1) is the probability of 𝑠 being value 1. We measure the information content carried by 𝑠 using the theory of information entropy, expressed as,EQUATIONFrom this equation, it can be observed that when 𝑝 𝑠 approaches either 0 or 1, the entropy function H (𝑠) may tend towards its minimum value 0. To evaluate the information content carried by 𝑠 in Q-SNNs, we conduct experiments with ResNet-19 on the CIFAR-10 dataset to obtain 𝑝 𝑠 in each layer, as illustrated in Figure 2 .Unfortunately, 𝑝 𝑠 in each layer tends towards 0, resulting in severely limited information content carried by 𝑠. Even worse, the weight in Q-SNNs also faces the same challenge.To enhance the performance of Q-SNN, increasing the information content of its binary weights and spike activities is necessary. Therefore, we analyze what condition 𝑝 𝑠 satisfies so that the information entropy of binary values is maximized. Maximizing entropy in this context helps ensure that the quantized weight and membrane potential values are spread out as evenly as possible within their range. This uniform distribution helps to ensure that the quantized network's representational capacity is not overly constrained by the quantization process. It allows the quantized network to preserve as much information as possible despite the reduced precision of synaptic weights and membrane potentials in the network. To achieve this, we need to solve the following equation,EQUATIONBy solving Eq. 13, we can determine 𝑝 * = 0.5. Therefore, we design a method to regulate the distributions of binary weights and spikes respectively, enabling their probability to be close to 0.5.To mitigate the performance degradation caused by information loss in the quantization process, we propose a Weight-Spike Dual Regulation method (WS-DR) to increase the information content of weights and spike activities in Q-SNNs. As analyzed in Sec. 4.2, the variable obeying the Bernoulli distribution carries the maximum information content when 𝑝 = 0.5. Therefore, in the following, we illustrate how to adjust the binary weight and spike activity to satisfy this condition as much as possible.We first analyze the regulation for synaptic weights in Q-SNNs. The objective for the binary weight, denoted as 𝐵 𝑤 , to satisfy 𝑝 𝑤 = 0.5 implies that it has an equal probability of taking either 1 or -1. We experimentally observed that the float-point weight in layer 𝑙 before adopting binarization operation generally obeys a Gaussian distribution N (𝜇 𝑙 , 𝜎 2 𝑙 ). However, the mean value 𝜇 𝑙 and standard deviation 𝜎 𝑙 vary across layers. Inspired by this observation, we implement a normalization technique on float-point weights in EQUATIONwhere 𝑊 𝑙 denotes the tensor of float-point weights in layer 𝑙. This equation enables Ŵ 𝑙 to obey a standard normal distribution N (0, 1), with zero serving as the symmetry axis. By subsequently employing the sign function on Ŵ 𝑙 , we can ensure 𝑝 𝑤 = 0.5.We then analyze the regulation for spike activities in Q-SNNs. Similar to the binary weight 𝐵 𝑤 in Q-SNNs, achieving the maximal information entropy of spike activities implies that each neuron in Q-SNNs emits a spike with a probability of 1/2 at all possible spike activity locations, i.e., the firing rate is 50%. This will seriously impair the sparsity characteristic of SNNs and sacrifice their inherent energy efficiency advantage. Therefore, in Q-SNNs, we design a loss function to impose a soft regulation on 𝑠. The goal of this loss function is to increase the information entropy of 𝑠 per layer as much as possible, rather than rigidly enforcing its probability to be 0.5. The proposed loss term L 𝑠 is defined as,EQUATION)where 𝐿 is the total number of layers in the network, 𝑓 𝑙 denotes the firing rate of spiking neurons in layer 𝑙, 𝑁 𝑙 is the number of neurons in layer 𝑙, and 𝑇 indicates the simulation time step. The proposed loss term L 𝑠 only regulates the spike distributions in hidden layers, disregarding the input and output layers as they are correlated with input data and classification outcomes. Consequently, the overall loss function is defined as,EQUATIONwhere o[𝑡] is the output of Q-SNNs at time 𝑡 and 𝑦 is the target label. L 𝑐𝑒 is the cross-entropy loss for classification, L 𝑠 is the soft regulation term designed to increase the information content of 𝑠, and 𝜆 is a hyperparameter that controls the contribution of L 𝑠 . By integrating the normalization technique on 𝑤 and the soft regulation loss term on 𝑠, the weight and spike in Q-SNNs can carry more information content, thus mitigating the performance degradation caused by information loss during the quantization process. Finally, we summarize the training procedure of our method in Algorithm 1. 
experiments: 
 In this section, we first outline the experimental setup and implementation details of the proposed Q-SNN with the WS-DR method.We then conduct a comprehensive evaluation, comparing the performance and model size of our approach against existing quantized SNN techniques. Finally, we employ ablation studies to validate the efficiency and effectiveness of our method.Datasets. We evaluate our method on image classification tasks, encompassing both static image datasets like CIFAR-10 [22] , CIFAR-100 [22] , and TinyImageNet [10] , alongside neuromorphic datasets Algorithm 1 Train a Q-SNN with the Weight-Spike Dual Regulation (WS-DR) method for one epoch. for 𝑙 = 1 → 𝐿 do 5:Input: A Q-SNN model: W = {𝑊 1 , • • • ,𝑊 𝐿 };Regulate the distribution of 32-bit weights: Ŵ 𝑙 = 𝑊 𝑙 -𝜇 𝑙 𝜎 𝑙 ;6:Compute 1-bit weights and channel-wise scaling factors:𝐵 𝑤 = sign( Ŵ 𝑙 ) , 𝛼 𝑤 = | | Ŵ 𝑙 | | 1 𝑛 ; 7:for 𝑡 = 1 → 𝑇 do 8:Compute the membrane potential (𝑈 𝑙 [0] = 0, 𝑆 0 [𝑡] = I):𝑈 𝑙 [𝑡] = 𝜏𝑈 𝑙 [𝑡 -1] + 𝛼 𝑤 𝐵 𝑤 ⊕ 𝑆 𝑙 -1 [𝑡] ; 9:Compute a layer-wise scaling factor: 𝛼 𝑢 = max 𝑈 𝑙 [𝑡] ;10:Quantize the membrane potential to 𝑘 bits:𝑈 𝑙 [𝑡] ← 𝛼 𝑢 2 𝑘 -1 round 2 𝑘 -1 clip 𝑈 𝑙 [𝑡 ]𝛼 𝑢 , -1, 1 ;11:Generate binary spikes via Heaviside step function:𝑆 𝑙 [𝑡] = Heaviside 𝑈 𝑙 [𝑡] -𝜃 ; 12:end for 13:𝑆 𝑙 = concat 𝑆 𝑙 [1], • • • , 𝑆 𝑙 [𝑇 ] ; 14:Compute the firing rate:𝑓 𝑙 = 1 𝑁 𝑙 ×𝑇 𝑁 𝑙 𝑖=1 𝑇 𝑡 =1 𝑆 𝑙 𝑖 [𝑡] ; 15:𝐹 ← 𝐹 .𝑎𝑝𝑝𝑒𝑛𝑑 (𝑓 𝑙 );16:end for 17:Regulate the distribution of spikes: L 𝑠 = 𝐿-1 𝑙=2 (𝐹 [𝑙] -0.5) 2 ;18:Compute the loss:L = 1 𝑇 𝑇 𝑡 =1 L 𝑐𝑒 + 𝜆L 𝑠 ; 19:Backpropagation and update model parameters; 20: end for such as DVS-Gesture [2] and DVS-CIFAR10 [24] . These datasets hold substantial importance within the realms of machine learning and neuromorphic computing, serving as standard benchmarks for evaluating diverse methodologies. Before introducing the experiments, we briefly outline each dataset. The CIFAR-10 and CIFAR-100 are color image datasets, with each dataset containing 50,000 training images and 10,000 testing images. Each image features 3 color channels and a spatial resolution of 32×32 pixels. CIFAR-10 is composed of 10 categories, whereas CIFAR-100 comprises 100 categories. The TinyImageNet dataset is a subset of the ImageNet dataset, consisting of 200 categories, with each category containing 500 training images and 50 test images. Each image has 3 color channels and a spatial resolution of 64×64 pixels. The DVS-Gesture and DVS-CIFAR10 are neuromorphic datasets captured using Dynamic Vision Sensor (DVS) event cameras, both featuring a spatial resolution of 128×128. The DVS-Gesture dataset consists of 1464 samples, with 1176 allocated for training and 288 for testing. The DVS-CIFAR10 is the most challenging neuromorphic dataset, featuring 9,000 training samples and 1,000 testing samples. During the preprocessing process of the DVS-CIFAR10 dataset, we apply the data augmentation technique proposed in [26] . ‡ : Self-implementation results with the same experimental setting. * VGGSNN: 8Conv1FC.Implementation details. We first present the architecture employed for each dataset. For static CIFAR-10 and CIFAR-100 datasets, we employ the well-established structure of ResNet19. For the TinyImageNet dataset, we employ the VGG16 structure to facilitate comparison with [50] . For DVS-Gesture and DVS-CIFAR10 datasets, we implement the structure of VGGSNN commonly used in SNNs [12, 39] . Within these architectures, the weights in the first layer and the last layer are quantized to 8 bit, while the weights in hidden layers are quantized to 1 bit [25] . Subsequently, we elucidate the implementation details within our experiments. We employ the Adam optimizer for the TinyImageNet dataset while the SGD optimizer for other datasets, and the batch size is set to 256 for static image datasets and 64 for neuromorphic datasets. The initial learning rate is set to 0.1 across all datasets and we adopt a cosine learning rate decay schedule during training. Moreover, the leaky factor 𝜏 and the firing threshold 𝜃 of the LIF neuron within Q-SNNs should be specified, which is set to 0.5 and 1, respectively. The time step 𝑇 is set to 2 for CIFAR-10 and CIFAR-100, 4 for TinyImageNet, 16 for DVS-Gesture, and 10 for DVS-CIFAR10. The hyperparameter value for the loss term L 𝑠 is set to 1e-3 for all utilized datasets. All experiments conducted in this paper leverage the PyTorch library, which is a versatile framework widely adopted in the field of deep learning research.We compare the performance and model size of Q-SNNs with existing quantized SNN methods to prove the effectiveness and efficiency of our method, respectively. Our experiments across all datasets involve three bit-width configurations: 1w-8u, 1w-4u, and 1w-2u, where '1w-8u' signifies that the Q-SNN utilizes 1-bit synaptic weights and 8-bit membrane potentials.We first analyze the performance comparison results. As shown in Table 1 , when compared with ANN2SNN learning algorithms that necessitate a large time step to guarantee lossless conversion (i.e., Wang et al. [43] set it to 100 on CIFAR-10 and 300 on CIFAR-100), the proposed Q-SNNs trained from scratch require fewer time steps to attain top-1 performance, such as 2 on CIFAR datasets and 4 on TinyImageNet dataset. When compared with direct learning algorithms, our work also achieves the top-1 accuracy on all datasets, i.e., 95.54% on CIFAR-10 under 1w-8u, 78.82% on CIFAR-100 under 1w-4u, 55.70% on TinyImageNet under 1w-8u, 97.92% on DVS-Gesture under 1w-8u, and 81.60% on DVS-CIFAR10 under 1w-8u. Moreover, it can be observed from Table 1 that the proposed Q-SNNs demonstrate comparable performance to that of full-precision SNNs with extremely compressed bit width across all datasets. Noteworthy, on the DVS-Gesture dataset, Q-SNN configured with 1w-8u achieves an accuracy of 97.92%, surpassing the corresponding full-precision SNN model. This superiority can be In addition to the outstanding performance, our method also demonstrates a compact model size. We evaluate the model size of the proposed Q-SNN alongside existing quantized SNN methods on the CIFAR-10 dataset, with the results illustrated in Figure 3 . Clearly, Q-SNN stands out with the smallest model size of 1.62MB and the highest accuracy of 95.20%. When compared with the work [50] that employs the same architecture of ResNet19, our method demonstrates a remarkable 56.22% reduction in model size and a 4.41% increase in accuracy with fewer time steps. In conclusion, our work further exploits the efficiency advantage inherent to SNNs while upholding superior performance, offering substantial advantages and potential for flexible deployment in real-world resource-limited scenarios.We conduct ablation experiments to validate the efficiency of Q-SNN and the effectiveness of the WS-DR method, respectively. All ablation experiments are performed on the CIFAR-10 dataset with the architecture of ResNet19, and experimental setups follow the description in Sec. 5.1.We first validate the efficiency of the proposed Q-SNN. As illustrated in Table 2 , we compute the memory footprint of our method under different architectures and bit-width configurations, obtaining two conclusions. Firstly, the proposed Q-SNN typically realizes a significant reduction in memory footprint compared to full-precision SNNs. For instance, employing the ResNet19 architecture with the batch size of 1, the Q-SNN with 1w-8u, 1w-4u, and 1w-2u achieves memory footprint reduction of 96.56%, 96.69%, and 96.75% respectively when compared with the full-precision SNN. Secondly, the benefit of the membrane potential quantization is positively correlated with the batch size. Taking VGGSNN as an example, when the batch size is set to 1, the compression of the membrane potential from 8 bit to 2 bit only yields a memory footprint reduction from 96.50% to 96.77%. In contrast, when the batch size is set to 256, the corresponding memory footprint reduction increases from 79.70% to 94.41%, underscoring the importance of membrane potential quantization. In summary, the proposed Q-SNNs have maximized the energy efficiency of the network by considering the quantization of both synaptic weights and memory-intensive membrane potentials.We now validate the effectiveness of the proposed WS-DR method. As depicted in Figure 4 , we plot the distribution of weights and spike activities in the first eight layers of ResNet-19 after applying the WS-DR method, respectively. It can be observed from in Figure 4 (a) that the synaptic weights display a uniform distribution, with the probability 𝑝 𝑤 approaching 0.5, so the 𝑤 in Q-SNNs have carried a greater amount of information. Moreover, it can be seen from Figure 4 (b) that the probability 𝑝 𝑠 becomes larger than Figure 2 , thus also demonstrating the increased information content for 𝑠 in Q-SNNs. In addition to the enhanced information content, we also compare the performance of three models: pure Q-SNN, Q-SNN integrated with the WS-DR method, and full-precision SNN. As illustrated in Figure 5 , under the configuration of 1w-2u, the Q-SNN integrated with the WS-DR method achieves an accuracy of 95.20%, surpasses that of Q-SNN without using the WS-DR method by 1.78%.Notably, the performance of our model is comparable to the fullprecision SNN of 96.36% by only using 1-bit synaptic weights and 2-bit membrane potentials. Therefore, the WS-DR method enhances the information content of both synaptic weights 𝑤 and spike activities 𝑠 in Q-SNNs, thereby resulting in performance comparable to that of full-precision SNN. 
conclusion: 
 Spiking neural networks have emerged as a promising approach for next-generation artificial intelligence due to their sparse, eventdriven nature and inherent energy efficiency. However, the prevailing focus within the SNN research community on achieving high performance by designing large-scale SNNs often overshadowed the energy-efficiency benefits inherent to SNNs. While efforts have been made to compress such large-scale SNNs by quantizing synaptic weights to lower bit widths, existing methods faced two main limitations. Firstly, they tend to overlook the memory footprint associated with membrane potentials in SNNs, leaving scope for further efficiency improvements. Secondly, the employed quantization on weights often leads to performance degradation of the quantized SNNs compared to their full-precision counterparts. This work introduced an efficient and effective Quantized SNN (Q-SNN) architecture to address these limitations. The proposed Q-SNN used efficient 1-bit weights and low-precision (2/4/8-bit) membrane potentials to further maximize SNN efficiency. In addition, inspired by information entropy theory, we proposed a novel Weight-Spike Dual Regulation (WS-DR) method to mitigate the performance degradation caused by quantization. This enabled the training of high-performance Q-SNNs from scratch. Experimental evaluations on static and neuromorphic datasets demonstrated that the proposed Q-SNN with WS-DR achieved state-of-the-art results in efficiency and performance compared to existing quantized SNN approaches. The obtained results demonstrate the potential of the proposed Q-SNNs in facilitating the widespread deployment of neuromorphic intelligent systems and advancing the development of edge intelligent computing.  
 
 Appendix:  


Decision: The paper proposes a novel Quantized Spiking Neural Network (Q-SNN) that quantizes synaptic weights and membrane potentials. According to the reviewers' feedback, acceptance of the paper is being considered. This method significantly reduces memory usage and computational complexity, making it suitable for resource-constrained and low-power edge devices. Furthermore, the paper introduces a novel Weight-Spike Dual Regulation (WSDR) method inspired by information entropy theory to ensure high performance of the model. Experimental results demonstrate its advantages in both model size and accuracy, highlighting its significant potential for edge intelligent computing.

Reviews: review: paper_summary: paper_summary: This paper introduces a method to enhancing the efficiency of spiking neural networks for deployment in resource-constrained environments. The authors propose a quantized snn (Q-SNN) that utilizes quantization techniques on both synaptic weights and membrane potentials, significantly reducing memory usage and computational complexity. Additionally, the paper introduces the weight-spike dual regulation method, inspired by information entropy theory, to mitigate performance degradation due to quantization. The Q-SNN is evaluated on several datasets, demonstrating the effectiveness of the method.
strength_weakness: strength_weakness: 1.The paper is well-organized, guiding the reader through the introduction, methodology.

2.The authors validate their method across several datasets, demonstrating the effectiveness.
limitations: limitations: 1.While the paper demonstrates significant improvements, the practical implementation of the proposed Q-SNN and WS-DR is complex, potentially limiting its adoption in real-world applications.

2.Although the paper provides comparisons with existing methods, a more detailed analysis of how Q-SNNs fare against other compression techniques like pruning and knowledge distillation in various scenarios would strengthen the claims.

3.The scalability of Q-SNNs to very large-scale SNN models or more complex tasks beyond those tested is not thoroughly explored.

4.While WS-DR is introduced and shown to be effective, a more in-depth theoretical analysis and understanding of its workings and potential limitations would provide a clearer picture of its benefits and drawbacks.

5.The WS-DR introduces additional computational overhead during training. The paper does not quantify this overhead or compare it with the benefits gained, leaving a gap in understanding the trade-offs involved.

6.The paper does not explore the sensitivity of the results to different levels of quantization. Understanding how varying the bit-widths for weights and membrane potentials affects performance and efficiency would be important for practical implementation.
paper_summary: paper_summary: This paper proposes an efficient and effective Quantized Spiking Neural Network (Q-SNN) that compresses both weights and membrane potentials. Moreover, the authors introduce a Weight-Spike Dual Regulation method to mitigate performance degradation of Q-SNN. Additionally, experiments on various datasets demonstrate that the proposed approach achieves state-of-the-art results compared to existing quantized SNN approaches.
strength_weakness: strength_weakness: Clear Structure and Presentation: The paper is well-organized and presented in a manner that is easy to follow, enhancing the reader's understanding of the proposed method.

Information Entropy Theory: This paper utilizes information entropy theory to analyze the reasons for performance degradation in model quantization and proposes a novel Weight-Spike Dual Regulation method to effectively enhance information retain.

Performance and Model Size: The proposed Quantized SNN not only surpasses previous works in performance but also demonstrates a significantly more compact model size. This dual achievement in both efficiency and efficacy marks a notable advancement in the field.
limitations: limitations: Symbol Usage Issue: The same symbol \(u_i^l[t]\) is used on both sides of Eq.(3), which may lead to an inaccurate expression. It is recommended that the authors revise the notation to avoid confusion and improve the clarity of the expression.
Equation Expression Issue: The current expression of Eq.(8) may cause the quantized membrane potential to be \( k+1 \) bits instead of the intended \( k \) bits. This issue also appears in Algorithm 1. The authors should review and correct these formulas to ensure the correctness and consistency of the computations.
Impact of Soft Regulation Loss Function: For Q-SNN, this paper introduces a soft regulation loss function on \( s \) to increase the information entropy of the binary spike activities. However, does this regulation lead to a higher spike firing rate, thereby consuming more computational resources? It is suggested that the authors further investigate this issue and provide corresponding experimental results to prove it.
paper_summary: paper_summary: The paper proposes the Q-SNN framework, focusing on synaptic weights and membrane potentials and introducing the Weighted Sum of Diverse Ranks (WSDR) based on information entropy, thereby advancing the field of quantized Spiking Neural Networks (SNNs).
strength_weakness: strength_weakness: The proposed models achieve decent accuracy with fewer time steps and smaller bit widths.
limitations: limitations: Limitations:
1.The selected architectures are relatively simple, while recent years have seen numerous transformer variants in the SNN field. The paper only mentions ResNet and VGG architectures.  
2. Incomplete comparative experiments: There is a lack of comparison between Q-SNN and other quantization methods in the SNN field in terms of fixed bit width, datasets, architectures, etc.  
3.The paper does not discuss dequantization methods, which are essential for quantization techniques.  
4. There is no experimental or explanatory information about the parameter lambda and its role.  
5. In terms of perspective, the author points out that other SNN models are primarily comparing accuracy, but it's worth noting that the author is also essentially comparing accuracy. It is suggested that the author include analyses of energy consumption, memory usage, parameter count, etc., rather than  focusing more on accuracy comparisons. The advantage of SNN lies in its energy efficiency, which the author has not adequately addressed in the paper.  
6. The paper is a modification of the SNN mechanism and does not seem to include any content related to multimodal aspects.  
Questions:  
1.Regarding the bit width setting, if weights can be up to 1w, why can't membrane potential reach 1u?   
2.How was the parameter count of 1.62M calculated? Please introduce in detail.
paper_summary: paper_summary: This paper presents a novel approach to enhancing the efficiency of SNNs for machine intelligence applications. The authors introduce a lightweight SNN architecture that applies quantization to synaptic weights and membrane potentials, significantly reducing memory usage and computational complexity. To enhance accuracy attributed to the compression, the WS-DR regulation method is proposed, drawing on information entropy theory. The paper reports experimental evaluations on various datasets, demonstrating their superiority in model size and accuracy compared to existing methods.
strength_weakness: strength_weakness: 1.This paper presents a quantization strategy applied to both weights and membrane potentials, significantly reducing the computational and memory footprints of SNNs. This advancement is particularly impactful given the increasing demand for efficient machine learning models.
2.Through extensive experimental evaluation, the application of information entropy theory to develop the WS-DR method has been proven effective. This method establishes a theoretical foundation for optimizing information-limited SNNs.
3.The authors conducted solid experiments, including evaluations of accuracy, model size, memory footprint, and various datasets, to demonstrate the effectiveness and energy efficiency of Q-SNNs.
limitations: limitations: 1.Q-SNN performs quantization on weights and membrane potentials. So why does the WS-DR method aim to maximize the information entropy of weights and spikes rather than that of weights and membrane potentials?
2.When designing the soft regulation loss $L_s$, why did the authors choose to calculate it over the entire time window rather than based on each time step? Calculating the loss functon $L_s$ based on each time step to maximize the information entropy of each time, which seems to be more in line with the authors' motivation for designing WS-DR.