Stepwise Weighted Spike Coding for Deep Spiking Neural Networks

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 14:05:38

Abstract: Spiking Neural Networks (SNNs) seek to mimic the spiking behavior of biological neurons and are expected to play a key role in the advancement of neural computing and artificial intelligence. The efficiency of SNNs is often determined by the neural coding schemes. Existing coding schemes either cause huge delays and energy consumption or necessitate intricate neuron models and training techniques. To address these issues, we propose a novel Stepwise Weighted Spike (SWS) coding scheme to enhance the encoding of information in spikes. This approach compresses the spikes by weighting the significance of the spike in each step of neural computation, achieving high performance and low energy consumption. A Ternary Self-Amplifying (TSA) neuron model with a silent period is proposed for supporting SWS-based computing, aimed at minimizing the residual error resulting from stepwise weighting in neural computation. Our experimental results show that the SWS coding scheme outperforms the existing neural coding schemes in very deep SNNs, and significantly reduces operations and latency.

Main paper: 
 introduction: 
 Spiking Neural Networks (SNNs) are known as the third generation of neural network models inspired by the biological structures and functions in the brain [32] . Unlike traditional Artificial Neural Networks (ANNs) that use continuous activation functions, SNNs incorporate discrete spiking events, enabling them to capture temporal dynamics and process information in a manner that closely mimics the brain's functioning [31] . This event-driven paradigm aligns with the brain's energyefficient computation and has the potential for more efficient and lower-power computing systems.[33].Various coding schemes have been proposed to describe neural activities, including rate coding and temporal coding [9] . Rate coding counts the number of spikes fired within a broad time window [23, 3, 18, 6] , which effectively mitigates the impact of short-term interference on the signal. It was widely accepted in the early days and typically outperformed temporal coding [11, 34, 4, 29, 20] .However, the rate coding scheme disregards the information in the temporal domain of the input spike sequence and requires many pulses to represent the input signal value, making it an inefficient coding method that negates the low-power benefits of SNN. Due to the functional similarity to the biological neural network, spiking neural networks can embrace the sparsity found in biology and are highly compatible with temporal coding [31, 33, 27, 28, 21, 15] . Temporal coding relies on the specific timing or patterns of input spikes, allowing for greater information capacity in a single pulse. However, it requires a large number of time steps to provide fine-grained timing, which increases inference latency. Its sensitivity to variations in spike timing also makes it more vulnerable to temporal jitter or delays [25, 24] . Additionally, decoding temporal-coded information usually requires more complex neuron models [30, 36] and training methodologies [17, 26] .Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.In the study of the temporal information dynamics of spikes, Kim et al. [16] discovered a phenomenon of temporal information concentration in SNNs. It is found that after training, information becomes highly concentrated in the first few timesteps. Based on this observation, we hypothesize that, from the perspective of the postsynaptic neuron, the first arriving spikes contain more information and require stronger responses. Consequently, we propose a mechanism whereby the neuron augments its own membrane potential with a specific coefficient prior to processing the subsequent input. This enhancement serves to increase the importance of preceding pulses on neurons, which is why the spikes are designated as Stepwise Weighted Spikes (SWS). Nevertheless, the amplification of the membrane potential makes it difficult for neurons to reduce its value through traditional "soft reset" (i.e. subtracted by an amount equal to the firing threshold), which can result in residual errors after neuron firing. To address this issue, we make the membrane potential reduced by a magnitude exceeding the threshold after firing. As a result, the membrane potential has both positive and negative residual values, which will generate both positive and negative spikes. This neuron is designated as a Ternary Self-Amplifying (TSA) neuron. To further reduce the error caused by the weighting process, a silent period is incorporated into the TSA neuron, allowing it to receive more input information before firing. We perform the classification tasks with SWS-based SNN on MNIST, CIFAR10, and ImageNet. The results show that the SWS coding scheme can achieve better performance with much fewer coding and computing steps. Even in very deep SNN, SWS coding scheme still performs well and achieves similar accuracy to the ANN with the same structure. Our major contributions to this paper can be summarized as follows:• We propose the SWS coding scheme, which enables easy implementation of SNNs with low energy consumption and high accuracy. The stepwise weighting process enhances the information-carrying capacity of the preceding pulses, greatly reducing the number of coding spikes. Negative pulses are introduced in SWS coding to ensure an accurate information transmission.• A novel TSA neuron model is proposed. TSA neuron progressively weights the input by augmenting its residual membrane potential before receiving the subsequent spike. The introduction of negative residual membrane potential and negative thresholds enhances the accuracy of the model's output.• A silent period is added to TSA neuron to markedly improve accuracy at minimal latency cost. By adjusting the silent period step and coding step, SWS-based SNNs can exhibit performance advantages in different aspects, improving the flexibility of applications.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: Stepwise weighting enhances the encoding of information in spikes, as is proved in eq. ( 5) in section 3.1. Our proposed SWS coding scheme achieves high performance and low energy consumption, which is supported by our experimental results in section 4. The TSA neuron model effectively minimizes the residual error, which can be proved from the ablation study in section 4.4.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 SNNs use spike sequences to convey information, making the encoding of real data into pulses a crucial step. Currently, the mainstream schemes of neural coding are rate coding and temporal coding [9, 33, 32] . Rate coding represents different activities with the number of spikes emitted within a specific time window. Due to its simplicity, rate coding is commonly used in deep learning of SNNs. However, it distributes information uniformly across a large number of spikes, resulting in an inefficient transmission process that increases network latency and energy consumption. Numerous researchers have proposed solutions to optimize inference latency in rate coding. Han et al. [11] proposed a "soft reset" spiking neuron model that retains a residual membrane potential after firing to better mimic the ReLU functionality. They demonstrated near lossless ANN-SNN conversion by using 2-8 times fewer inference time steps. Still, a delay of thousands of steps is required in large datasets or deep networks. In [14] , Hu et al. reduced the encode time steps by converting a quantized low-precision ANN to a rate-coded SNN. They also proposed a layer-wise fine-tuning mechanism to minimize the inference latency. However, their neuron model and the subsequent fine-tuning algorithm are relatively complex. Furthermore, in deeper neural networks such as ResNet56, a 1.5% drop in accuracy can be observed. The above rate encoding solutions are limited because they do not consider the significance of each spike.In [15] , Kim et al. proposed phase coding, which assigns different weights to spikes based on their time phase. However, the transmission amount of information is bounded by the global phase, which causes inefficiency in hidden layers, resulting in a latency of up to three thousand steps for a 32-layer network. Burst coding [21] attempts to overcome this issue by introducing burst spikes, which utilize Inter-Spike Interval (ISI). Burst spikes are capable of conveying more information quickly and accurately by inducing Post-Synaptic Potential (PSP) dramatically. Nevertheless, it is still deficient Table 1 : Common symbols and their meanings in this paper.MeaningS l i (t)The spike train fired by the i th neuron in the l th layer u l i (t) The membrane potential of the i th neuron in the l th layer z l i (t) The integrated inputs to the i th neuron in the l th layer V l th The firing threshold of the neurons in the l th layer θ lThe amplitude of the spikes fired by the neurons in the l th layer in terms of latency and efficiency. Rueckauer and Liu [27] proposed an efficient temporal encoding scheme where the analog activation values of the ANN neurons are represented by the inverse Time-To-First-Spike (TTFS) in the SNN neurons. Their new spiking network model generates 7-10 times fewer pulses by utilizing temporal information carried by a single spike. However, as pointed out in [10] , TTFS coding scheme incurs expensive memory access and computational overhead, which diminishes the benefit of reduced pulse count. Furthermore, TTFS necessitates a large number of time steps to differentiate between various time points, which also increases network latency. Han and Roy [10] proposed the Temporal-Switch-Coding (TSC) scheme, in which each input image pixel is represented by two spikes, and its intensity is proportional to the timing between the two pulses. Their results showed a reduction in energy expenditure. However, TSC coding requires a large number of time steps to provide distinguishable time intervals, rendering it an ineffective approach to addressing the issue of the long latency.Overall, rate coding employs a large number of pulses to encode information, which results in a considerable energy overhead and inference delays. On the other hand, temporal coding allows for greater information capacity in a single spike, but this does not reduce the computing latency as a precise time point or period can be identified only with a sufficient number of time steps. Therefore, new neural coding schemes should be developed.3 Stepwise weighted spike coding schemeQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The inclusion of silent periods can lead to increased latency, as noted in section 3.3, which is a limitation we've found so far. However, our experimental results demonstrate that our delay performance still surpasses that of other SNNs.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
stepwise weighting: 
 The spike train S l i (t) of the i th neuron in the l th layer can be expressed as follows:EQUATIONwhere δ(t) is the Dirac delta function, θ l is the spike amplitude of the l th layer, which is usually set to the same value as the firing threshold. f is the index of the spike in the sequence, and F l i denotes a set of spike times which satisfies the firing condition:EQUATION)where u l i (t) denotes the membrane potential and V l th denotes the firing threshold of the neurons in the l th layer.Our basic idea is to amplify the membrane potential before the receipt of the subsequent input, which amplifies and prolongs the impact of the preceding input spikes on membrane potential, emulating the phenomenon of information concentration identified in [16] . For clarity, the meanings of important symbols are provided in table 1 . The action of a neuron in SWS-SNN can be described as follows:EQUATIONwhere β is the amplification factor which should be greater than one, z l j (t) denotes the PSP (i.e. integrated inputs):z l j (t) = i ω l ij S l-1 i (t) + b l j (4)where ω ij is the synaptic weight and b l j is the bias. Begin with the initial value u l j (0) = 0 and iteratively apply eq. ( 3) for each subsequent value until u l j (n) and substitute eq. ( 1) and eq. ( 4) into it, Figure 1 : (a) Illustration of the stepwise weighting process. The meanings of the symbol z l j (t), u l j (t) and S l j (t) can be found in table 1 . The blue dotted line represents the membrane potential prior to the spike firing, and the black exponential function-like dotted line is employed to illustrate the trend of membrane potential amplification. (b) A V l th equal to θ l results in residual errors, leaving a lot of information unencoded. (c) V l th is set to 1 2 θ l , which increases the possibility to fire spikes early to better limit the residual. (d) Use negative spikes to correct the excessively emitted information. eq. ( 3) can be written as:EQUATIONNote that S l j (t) is set to zero for simplicity. From eq. ( 5), it can be seen that the stepwise augment of the membrane potential results in the spike input at time tl-1,(f ) i encoding the value θ l-1 β n-t l-1,(f ) i .This process is thus referred to as stepwise weighting, and β n-t l-1,(f ) i serves as the weight. The earlier the input pulse, the greater its ability to carry information. This solves the problem of excessive encoding steps in previous schemes, allowing faster information transmission.Stepwise weighting effectively assigns more weight to earlier arriving pulses, but it also makes spike generation more tricky. To ensure that input information is efficiently encoded and transmitted to the next layer, the residual membrane potential should be minimized after neural computation is completed. The stepwise weighting, however, amplifies the residual potential from the previous time step. If z l j (t) remains high in subsequent steps, reducing the membrane potential becomes challenging, as shown in fig. 1(b ). This vicious cycle ultimately leads to a persistently high membrane potential, indicating that a substantial amount of information remains unencoded.We refer to this phenomenon as residual error. One contributing factor is that the threshold is set too high, resulting in a pulse being emitted only when the membrane potential exceeds the value θ l .While this prevents excessive information transmission, it results in missed opportunities to bring down u l j (t) by firing a spike.To address this issue, we propose setting the firing threshold V l th to 1 2 θ l . This adjustment facilitates pulse generation and reduces the residual membrane potential. After the neuron firing, the membrane potential is subtracted by θ l , which leads to the emergence of a negative residual that will be stepwise The silent period allows more information to be known when firing pulses. T s is set to 1 here. V l th is amplified by β Ts , and the original threshold is represented by a gray solid line. The orange dashed line represents the amount of membrane potential reduction after firing. (c) The silent period also avoids some unnecessary spikes and increases sparsity. Without the silent period, since u l j (1) exceeds the original threshold, a pulse will be generated at t = 1, which will later be corrected by another negative spike. (d) The impact of the silent period on network latency. The output spike sequences corresponding to different inputs are drawn in blocks of different colors. The pulses drawn in the spike sequence are for illustrative purposes only. weighted over time. The coefficient 1 /2 is selected as it is capable of controlling both positive and negative residuals within a narrow and balanced range. A negative threshold -V l th is introduced into the neuron model, which initiates a negative spike when the membrane potential falls below this threshold. This mechanism allows the excessively emitted information to be corrected by the negative spike, as shown in fig. 1(d ). Given the above characteristics, we designate this neuron model as a TSA neuron.Another contributing factor to residual error is the imbalanced distribution of z l j (t). A burst input of z l j (t) at time point τ results in a sharp rise in membrane potential, making it difficult for subsequent spikes to reduce it, as shown in fig. 2(a) .This can be addressed by incorporating a silent period T s into the TSA neuron model. The neurons only integrates input and performs stepwise weighting, but are not allowed to fire in the first T s steps. This enables the acquisition of more known information before spike generation, resulting in increased accuracy, as illustrated in fig. 2(b) . Since the preceding input information has been amplified by β Ts after the silent period, V l th also needs to be adjusted accordingly, which is set to β Ts 2 θ l . Similarly, after firing, the membrane potential should be subtracted by θ l β Ts . Note that the fired spike amplitude remains unchanged, that is, θ l .The impact of the silent period on network latency is shown in fig. 2(d ). The output results for different input sequences are distinguished by blocks of different colors. It can be observed that as network depth increases, the silent period accumulates, leading to a higher output latency. The inference latency of SWS-SNN can be calculated as follows:T inf = T c + T s • L TSA (6)where T inf is the inference delay, T c is the coding time steps, T s is the length of the silent period and L TSA is the number of TSA neuron layers. The neuron model in other coding schemes yields a zero T s , leading to an output delay equal to the coding time step, which is consistent with the definition in the previous scheme. From fig. 2 (d), it can be seen that different input sequences are processed in a pipeline-like manner, and the value of T c + T s determines the throughput rate of SWS-SNN.According to eq. ( 5), the value that can be losslessly encoded under the SWS coding scheme can be expressed as follows:EQUATIONwhere A j denotes the encoded value. a τ j ∈ {-1, 0, 1} indicates the type of the output spike at time τ :1 for a positive pulse, -1 for a negative pulse and 0 for no pulse. T c denoted the time steps used for encoding. The weight β Tc-τ results from the stepwise weighting process described in section 3.1. θ 0 denotes the spike amplitude of the input encoding layer, which can be assigned an appropriate value based on the range to be encoded.According to eq. ( 7), given a fixed T c and θ 0 , the distribution of A j is determined by β. Setting β to 2 is reasonable, as it ensures A j is evenly distributed within the codable range. Compared to rate coding, which necessitates 2 Tc coding steps to encode the same range with same precision, SWS coding significantly enhances coding efficiency. Note that with the introduction of negative pulses, setting β to 3 can also achieve a uniform distribution of A j and offers even more values for accurate encoding compared to β = 2.1 When β is less than 2, the distribution of A j becomes denser at smaller values, which may be suitable for encoding data that follows a similar distribution.For static image classification tasks, the pixel value p j can be encoded by applying a constant input z 0 j (t) to the TSA neuron. Considering the stepwise weighting process, we can write:EQUATIONwhere z 0 j denotes the amplitude of the constant input z 0 j (t). Solve for z 0 j and we have:EQUATIONGiven that z 0 j (t) is a constant at each step, T s can be set to 0 for the encoding layer. However, the neuron must await T s time steps after the completion of an encoding. This allows neurons in the subsequent layer to complete the previous neural computing before receiving the next encoded input. 
experiments: 
 In this section, we convert quantized ANNs to SWS-based SNNs2 and conduct experiments on MNIST, CIFAR10, and ImageNet. Firstly, an overview of SWS-SNN's performance across various datasets is provided. Subsequently, the network's inference latency and energy consumption is compared with other spike coding schemes. Finally, an ablation study is conducted to investigate the impact of lowered thresholds and silent periods on reducing residuals and enhancing accuracy.ANNs used for conversion are all quantized to 8 bits. β is set to 2 in the experiments to ensure that codable values are evenly distributed. Compared to β = 3, a smaller amplification factor reduces the impact of residual errors, resulting in more accurate output. For simple classification tasks such as CIFAR10, our proposed SWS coding scheme has a faster inference speed than other ANN-SNN models while achieving similar classification accuracy, or has higher classification accuracy than direct learning at similar inference speeds. For example, ResNet18with SWS improves throughput seven times over [20] while simultaneously improving accuracy.Although the network in [5] has a slightly higher throughput, its accuracy is 1.17% lower than our scheme. To fully test the potential of our proposed coding scheme, we conducted experiments on ImageNet using networks with various structures. The experimental results demonstrate that SWS coding has distinct advantages on extremely deep SNNs. Our SWS-based ResNet50 and ResNeXt101 achieved over 80% accuracy on ImageNet with only eight coding steps. The model in [12] achieves an almost lossless conversion with eight time steps. However, their method has to adjust the resting potential of neurons layer by layer, and the calibration effect for deeper networks is unclear. In [14] ,the original ANN needs to be quantized to 3 bits, resulting in a larger conversion loss. Directly trained SNNs typically achieve higher throughput, but their accuracy still requires improvement. In addition, the SWS coding scheme is easy to implement. No further fine-tuning is required after the conversion.The comparison of latency results between SWS-SNN and other ANN-converted SNNs [1, 11, 10, 4, 18, 2, 7] is illustrated in fig. 3 . The latency of the network is calculated with eq. ( 6). In the counterpart models, the variation of delay is mainly caused by the changes in T c . In contrast,T s determines latency in deep SWS-SNNs. Therefore, SWS-SNN has an upper limit on latency: in the shortest time on CIFAR10 and reduce the inference latency on ImageNet by more than one order. Even though the silent period accumulates when the network gets deeper, the results in fig. 3(c ) demonstrate that our scheme still achieves the fastest inference speed with the highest accuracy in a 34-layer network. Note that T s is set to the same value for each TSA layer for simplicity, resulting in discontinuous T inf values. This causes a sharp drop in accuracy at smaller delays.T max inf = T c (1 + L TSA ),To compare the energy consumption of SWS-SNN with SNNs under other encoding schemes, we adopt the method as in [29, 27, 28] to count operations:EQUATIONwhere OPF (Operations Per Frame) denotes the number of operations for the classification of one frame, T c and T s denotes the coding steps and the length of the silent period, respectively. L TSA denotes the number of TSA layers, f l out denotes the fan-out of neurons in layer l, n l (t) denotes the number of spikes fired in layer l at time τ and N TSA denotes the number of TSA neurons. The first term on the right-hand side of the equation arises from the TSA's requirement to amplify the membrane potential. Note that due to the accumulation of T s over the network depth, the time period for counting n l (t) varies with l.Experiments were conducted on MNIST using LeNet-5. We varied the silent periods and adjusted the coding steps to study their effects on OPF. The results are presented in fig. 4 (a). As indicated in eq. ( 10), reducing T c lowers energy overhead. This presents a trade-off between energy consumption and inference accuracy, as fewer coding steps also reduce the number of values that can be accurately encoded. A larger T s requires TSA neurons to perform more operations to amplify the membrane potential. On the other hand, it reduces the number of unnecessary pulse emissions. Overall, silent period has a negligible impact on OPF. In fig. 4(b) , the energy consumption of SWS-based SNN is compared with that of other SNNs. The experimental results demonstrate that our coding scheme can achieve a favorable balance between accuracy and energy consumption. The SWS coding scheme is superior to rate coding and temporal pattern coding in that it requires fewer operations and achieves higher accuracy. In TTFS encoding, each neuron fires at most one spike at a time, theoretically demanding the least OPF. With T c = 4, SWS-SNN can achieve significantly higher accuracy with minimal increase in OPF. Note that if the ANN is quantized to a lower number of bits (e.g., 4 bits), the error caused by the reduced T c can actually be compensated by the quantization algorithm, which can potentially result in a higher performance.D V l th = l , Ts = 0 V l th = l , Ts = 1 V l th = l , Ts = 2 V l th = l 2 , Ts = 0 V l th = l 2 , Ts = 1 V l th = l 2 ,E V l th = l , Ts = 0 V l th = l , Ts = 1 V l th = l , Ts = 2 V l th = l 2 , Ts = 0 V l th = l 2 , Ts = 1 V l th = l 2 , Ts = 2In section 3.2 and section 3.3, we proposed reducing the firing threshold and introducing a silent period to mitigate residual error. To assess the impact of these two adjustments, we conducted experiments on CIFAR10 using ResNet18. After the neural computation, the residuals (absolute values) of the TSA neurons were analyzed. We first scaled the residuals by 1 /β Ts to counteract the effect of membrane potential amplification caused by the silent period, and then normalized them in units of θ l . The probability density of the residuals is shown in fig. 5(a) .The results demonstrate that lowering V l th shifts the residual distribution from around 0.5θ l to approximately 0.25θ l , corresponding to the quantization errors (i.e. rounding errors) under their respective thresholds. The addition of silent periods further concentrates the distribution and reduces large deviations. As can be seen from the green curve in fig. 5 (a), setting T s to 2 and V l th to θ l /2 makes the residuals almost all distributed around the quantization error. Compared to the red curve (without a lowered V l th or a silent period), the residuals are greatly reduced, which fully proves the effectiveness of lowering the threshold and adding a silent period. The inference results on CIFAR10 is shown in fig. 5(b ). When setting V l th to θ l and T s to zero, the network's output is almost random.Lowering the threshold and adding a silent period improve the accuracy to 35.41% and 84.21%, respectively. Ultimately, the combination of both adjustments enabled SWS-ResNet18 to achieve an accuracy of 95.68% on CIFAR10. 
conclusion: 
 In this work, we have proposed a novel SWS spike coding scheme. The stepwise weighting process enhances the information-carrying capacity of the preceding pulses, greatly reducing the number of time steps for encoding. Combined with a silent period, our proposed TSA neuron model solves the problem of residual errors and achieves fast and accurate information transmission. Our experimental results have demonstrated that SWS coding is highly effective in extremely deep SNNs and achieves state-of-the-art accuracy. The SWS coding scheme is also highly flexible and can adapt to various needs.[33] Yamazaki, K., Vo-Ho, V.K., Bulsara, D., Le, N.: Spiking neural networks and their applications:A review. Brain Sci 12(7) (Jun 2022)[34] Yan, Z., Zhou, J., Wong, W.: Near lossless transfer learning for spiking neural networks. In: 6(a) . The activations of the quantized ANN can be mapped to an integer Q between [0, 2 n -1] using a scaling factor C and a zero point Z. With the same weight and bias between Q l i and Q l o , the TSA layer can generate S l , which encodes Q l o , provided that S l-1 encodes Q l i and no residual error occurs. In the actual SNN, the pulse amplitude θ l is normalized to 1. Therefore, the bias need to be further scaled to derive the final weight W l and bias b l for the SWS-SNN.AAAITable 3 : The notations and meanings of parameters in the quantized network. The bias of layer lQuestion: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: Code will be released when the paper is accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The details for acquiring different delays and the OPF calculation method are provided in section 4.2 and section 4.3, respectively. The parameters used during QAT training is outlined in appendix A.2.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We believe it is not necessary to include error bars in the results because each experimental result itself is already the average of a large number of tests (E.g., the test accuracy for an epoch is averaged over N um_of _batches × Batch_size input images, and is therefore very close to each other in every test epoch).Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [No] Justification: We found it difficult to quantify the computing resources used in every experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The pretrained ANN model and the QAT workflow is provided by PyTorch and we cited the original paper in appendix A.1 as [22] .Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The derivation is as follows. After QAT, we have:EQUATIONEQUATIONEQUATIONwhere Q l i , Q l o represent the integers to which the quantized input and output are mapped, respectively.Substitute eq. ( 12) and eq. ( 13) into eq. ( 11), and we can write:EQUATIONwhich gives: whereEQUATIONEQUATIONEQUATIONAs seen in eq. ( 15), with the weight and bias set to W l and bl respectively, the layer outputs Q l o when receiving Q l i . The pulse amplitude θ l can be set to any value as long as the codable range calculated by eq. ( 7) covers [0, 2 n -1]. Then we have:EQUATIONConsidering the membrane potential amplification, b l can be calculated as follows:EQUATIONOnce the T c , β and θ l (θ l-1 is given by the previous layer) have been determined, all values on the right side of eq. ( 18) and eq. ( 19) are known. Consequently, W l and b l in the SWS-SNN can be readily calculated from the weight and bias of the quantized ANN.After configuring the weights and biases as described above, the input pixel must be encoded into a pulse sequence with an amplitude of 1 as well. This process is illustrated in fig. 6(b) . First, map the pixel value to [0, 2 n -1] using C 0 i and Z 0 i obtained from QAT. Assuming this range can be encoded by SWSs with an amplitude of θ 0 , scaling the pixel value by 1 /θ 0 allows the use of a sequence with θ 0 = 1 for encoding. Finally, encode the scaled pixels following section 3.4, and the required input spike sequence is obtained.QAT is the quantization method that typically results in the highest accuracy. We basically follows the workflow provided by PyTorch. The default QAT quantization configuration is chosen to specify the kind of fake-quantization inserted after weights and activations. We choose Stochastic Gradient Descent (SGD) optimizer in QAT, with the value of momentum set to 0.9 and the learning rate set to 1 × 10 -4 since the weights only need to be fine-tuned. QAT is done for 12 epochs and 20 batches in each epoch. We freeze the batch norm mean and variance estimates after three epochs and freeze the quantizer parameters (scaling factor and zero point) after another two epochs.NeurIPS Paper ChecklistJustification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Setting β tointroduces some coding redundancy. E.g., a 1 j = 1, a 2 j = -1 and a 1 j = 0, a 2 j = 1 encodes the same amount of information.2 Details of the conversion process can be found in appendix A.1 and appendix A.2 


Decision: This paper presents a new spike coding scheme. The scores are divided. After rebuttal and discussion, there are still some important concerns, such as the quality of the comprehensive set of experiments, that have not been handled, although this work may offer some interesting insights.

Reviews: review: paper_summary: paper_summary: The authors introduce a new spike coding scheme, which allows them to directly convert quantized ANN to their coding scheme. They demonstrate the effectiveness of their conversion on several pre-trained ANN with minimal loss in performance at the cost of an increase in latency.
strength_weakness: strength_weakness: - limited connection to spiking neurons, a more straightforward motivation would be a temporal encoding of quantized ANN
questions: questions: - could you compare your approach more explicitly to ref. 30?
limitations: limitations: - method only applicable to conversion from pre-trained ANN
- no demonstration of training of a model using this coding scheme.
paper_summary: paper_summary: The authors introduce a novel encoding method called Stepwise Weighted Spike (SWS) and a corresponding new neuron model named Ternary Self-Amplifying (TSA) for classification tasks utilizing the ANN2SNN training method. The proposed SWS encoding method assigns weights to the importance of spikes at each time step. The TSA neuron, which employs the SWS encoding method, features a lower threshold and includes a silent period.
strength_weakness: strength_weakness: 1. Effectiveness of SWS: Various encoding methods, such as rate encoding and Time-to-First-Spike (TTFS) encoding, can be applied to different neurons and models. However, as illustrated in Figure 5, the SWS encoding method alone is ineffective without incorporating a lower threshold and a silent period. It only functions effectively when a neuron employs SWS encoding along with these additional components. Therefore, the paper should emphasize the neuron model rather than the encoding method, as it is not a universally applicable approach.
2. Lack of Experiments: The ablation study shows that the introduction of a silent period is the primary contributor to the improved performance. This raises doubts about the effectiveness of the SWS encoding method itself. Can the authors provide performance metrics for rate encoding combined with a lower threshold and silent period (if applicable) to ensure a fair comparison?
questions: questions: See weaknesses.
limitations: limitations: See weaknesses.
paper_summary: paper_summary: The paper proposes a new coding scheme called Stepwise Weighted Spike (SWS) coding scheme for spiking neural networks to enhance the efficiency and reduce the number of operations and thus energy consumption. The SWS coding scheme tackles challenges associated with temporal and rate coding, such as heightened latency and energy usage. It achieves this by compressing spikes and assigning them varying weights at each computational step. Additionally, the paper introduces the Ternary Self-Amplifying (TSA) neuron model, which incorporates a silent phase to mitigate residual errors arising from the weighting procedure.
strength_weakness: strength_weakness: 1. Which model of a spiking neuron is being employed in equation 3 (line 120)? What is the reset mechanism here after the neuron fires? Are the weights allowed to have negative values? The description of the model is unclear. 

2. The notion of residual error intuitively makes sense but it is confusing. Please define the residual error mathematically (line 139) for better understanding. 

3. Why ANN-(sws)SNN conversion is opted instead of directly training the SWS based SNN?

4. There are some recent works [1,2,3] with TTFS encoding which claims better results in regard to energy-efficiency and low-latency. First, these works need to be cited in the related work section. In my opinion, a detailed comparative analysis with other models and encoding schemes (for instance with [1,2,3]) needs to be carried out. 

[1] Göltz, J., Kriener, L., Baumbach, A. et al. Fast and energy-efficient neuromorphic deep learning with first-spike times. Nat Mach Intell 3, 823–835 (2021).

[2] I. M. Comsa, K. Potempa, L. Versari, T. Fischbacher, A. Gesmundo and J. Alakuijala, "Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function," ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 2020, pp. 8529-8533, doi: 10.1109/ICASSP40776.2020.9053856.

[3] Stanojević, Ana et al. “An Exact Mapping From ReLU Networks to Spiking Neural Networks.” Neural networks : the official journal of the International Neural Network Society 168 (2022): 74-88.
questions: questions: 1. Please address all the weaknesses above. 

2. This is an empirical study. Can you provide theoretical proof that the number of time steps or operations required to interpolate a given set of data points (i.e., fitting the training data) is significantly less compared to other methods, such as the Time to First Spike, when using Equation 10 as a metric?
limitations: limitations: There is no potential negative societal impact and and one limitation related to the inclusion of silent period is noted in the main text.
paper_summary: paper_summary: This paper proposes a novel Stepwise Weighted Spike (SWS) coding scheme designed to improve the efficiency of Spiking Neural Networks (SNNs) by compressing spikes and weighting their significance in each step of neural computation. This method addresses the issues of high delays and energy consumption associated with existing SNN coding schemes, as well as the complexity of neuron models and training techniques. The authors also introduce a Ternary Self-Amplifying (TSA) neuron model, incorporating a silent period to support SWS-based computing. This model is designed to minimize the residual error resulting from the stepwise weighting process. The experimental results provided in the manuscript demonstrate that the proposed SWS coding scheme significantly outperforms existing neural coding schemes, particularly in very deep SNNs. Key improvements include reduced operations and latency, enhanced overall performance, and lower energy consumption.
strength_weakness: strength_weakness: 1.	Clarity and Detail: Some sections of this paper could benefit from more detailed explanations, particularly in the description of the SWS coding scheme and TSA neuron model. This would help in understanding the underlying mechanisms and their advantages.
2.	Comparative Analysis: While the experimental results are promising, there is no proof from the experimental results that the encoding method proposed is more advantageous.
3.	There are some grammatical errors in the paper. Such as the second paragraph of Section 3.3, "The neurons only integrates input and performs stepwise weighting". It is recommended that a uniform representation be used for "spike" and "pulse".
4.	Symbol design problem, "t" in Eq. (3) becomes "n" in Eq. (5).
5.	There are many long paragraphs and sentences in the paper, making it difficult for readers to accurately understand the meaning of the paper.
6.	The description of the problem in the third paragraph of Section 1 and the end of Section 2 is not clear, making it difficult for readers to understand the problem that the article really wants to solve.
7.	The description of the encoding method in Eq. (7) is difficult to understand. According to Eq.  (7), the encoded value $A_j$ should have no time step. However, in the experimental part, the method of this paper has 8 time steps.
questions: questions: 1.	The paper proposes a new encoding method. Is this encoding method designed for the TSA neuron model, or is it an encoding method in more general scenarios? If it is a targeted approach, would it be more appropriate as a sub-module of the proposed method?
2.	Three data sets are used in the experimental section. Why don’t show the comparison results on MNSIT in Table 2?
limitations: limitations: The paper mentioned that due to the setting of the neuron's silent period, the delay increases. It can be seen from the experiments that the overall latency of the method is lower, which can be regarded as solving this limitation. At the same time, this article does not have potential negative social impacts.
paper_summary: paper_summary: This work belongs to ANN2SNN and proposes a novel coding scheme and neuron model to enhance the efficiency and accuracy of Spiking Neural Networks (SNNs) while reducing energy consumption. The Stepwise Weighted Spike (SWS) coding scheme improves information encoding by stepwise weighting input signals and introducing negative pulses, reducing the number of coding spikes needed. The Ternary Self-Amplifying (TSA) neuron model further enhances accuracy by progressively weighting the input through residual membrane potential adjustments and incorporating negative residuals and thresholds. Introducing silent periods allows the neuron to receive more input information before firing, significantly improving accuracy with minimal latency. Experimental results on datasets like MNIST, CIFAR10, and ImageNet demonstrate that the SWS coding scheme achieves better performance with fewer coding and computing steps, performing well even in very deep SNNs and achieving accuracy comparable to Artificial Neural Networks (ANNs) with the same structure.
strength_weakness: strength_weakness: In the ImageNet experiments in Table 2, SWS and other comparative ANN-SNN methods used different baselines, which is why the '$SNN\  Acc$' results are much higher than those of the comparative methods. However, the ‘$\Delta ACC$’ does not seem to show a significant difference (except for Hybrid training and Spiking ResNet). Using the same network architecture and pre-trained weights would be more credible.
questions: questions: 1)  

For the experiments on ImageNet in Table 2, it seems that the ANN baseline of SWS has a higher accuracy than that for the comparative methods. Why weren't other methods tested on the same architecture and pre-trained weights?

2)  

Figure 2 describes the 'silent period' proposed in this paper, during which neuron potentials accumulate and are not allowed to fire spikes. After experiencing a silent period of $T_s$, the corresponding $\theta^l$ and $V_{th}^l$ are amplified by a factor of $\beta^{T_s}$. Firstly, should the $V_{th}^l$ in Figures 2(b) and 2(c) be $V_{th}^l\beta^{T_s}$? Otherwise, it does not correspond to $\theta^l\beta^{T_s}$. Secondly, how many time steps do the 'burst' spikes last after the silent period? Why, in the middle graph of Figure 2(b), are there larger spikes at the second and fourth time steps and a smaller spike at the third time step? In the middle graph of Figure 2(c), why is there a smaller spike at the third time step and a larger spike at the fourth time step? Is the amplification of $V_{th}^l$ and $\theta^l$ by $\beta^{T_s}$ maintained for several time steps or continuously after the silent period?
limitations: limitations: The authors have not explicitly addressed the limitations or potential negative societal impacts of their work. To improve the transparency and completeness of their research, the authors could consider the following constructive suggestions:

1)  Limitations:

Create a dedicated "Limitations" section in the paper to discuss any constraints, assumptions, or potential weaknesses of the proposed SWS coding scheme.

Reflect on the robustness of the results to violations of assumptions, such as noiseless settings, model specifications, or dataset dependencies.

Discuss the scope of the claims made in the paper, including the generalizability of the approach across different datasets and scenarios.
Address factors that may influence the performance of the SWS coding scheme, such as computational efficiency and scalability with varying dataset sizes.

Consider possible limitations related to privacy and fairness concerns in the implementation of the SWS coding scheme.

2)  Negative Societal Impact:

Explicitly acknowledge the potential negative societal impacts of the SWS coding scheme, such as privacy risks, fairness considerations, or unintended consequences.

Discuss how the technology could be misused or lead to harmful outcomes, even if not intended by the authors.

Consider mitigation strategies to address any identified negative societal impacts, such as controlled release of models, monitoring mechanisms, or additional safeguards.

Emphasize the importance of ethical considerations and responsible deployment of the SWS coding scheme in real-world applications.

Comments: comment: Thank you for addressing my concerns, I will keep the rating.
I appreciate the authors' responses, which have addressed most of my concerns. The proposal of a new encoding scheme is interesting; however, given that this is primarily an empirical study, further comparison with TTFS-based encoding schemes—both ANN-SNN conversion and direct training—is essential. Since TTFS encoding also emphasizes energy efficiency, low latency, and the number of operations based on optimal parameter training, a more thorough evaluation is necessary to truly assess the significance of the proposed approach. Based on this, I prefer to keep my rating unchanged.