Advancing Spiking Neural Networks for Sequential Modeling with Central Pattern Generators

Authors: changze  lv
dongqi  han
yansen  wang
xiaoqing  zheng
xuanjing  huang
dongsheng  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 08:48:36

Abstract: Spiking neural networks (SNNs) represent a promising approach to developing artificial neural networks that are both energy-efficient and biologically plausible.
However, applying SNNs to sequential tasks, such as text classification and time-series forecasting, has been hindered by the challenge of creating an effective and hardware-friendly spike-form positional encoding (PE) strategy.
Drawing inspiration from the central pattern generators (CPGs) in the human brain,  which produce rhythmic patterned outputs without requiring rhythmic inputs, we propose a novel PE technique for SNNs, termed CPG-PE.
We demonstrate that the commonly used sinusoidal PE is mathematically a specific solution to the membrane potential dynamics of a particular CPG.
Moreover, extensive experiments across various domains, including time-series forecasting, natural language processing, and image classification, show that SNNs with CPG-PE outperform their conventional counterparts.
Additionally, we perform analysis experiments to elucidate the mechanism through which SNNs encode positional information and to explore the function of CPGs in the human brain.
This investigation may offer valuable insights into the fundamental principles of neural computation.

Main paper: 
 introduction: 
 Spiking neural network (SNN) [1] has increasingly attracted research interests in recent years, primarily due to its energy efficiency, event-driven paradigm, biological plausibility, and other distinctive properties. The spiking neurons in SNN are dynamical systems that generate binary signals (spike or non-spike) and communicate these signals like artificial neural networks (ANNs) for computation [2] [3] [4] [5] [6] [7] [8] [9] . Many advanced architectures and methodologies developed for ANNs are also applicable to SNNs, enhancing their capabilities. Notable among these are backpropagation [10] , batch normalization [11, 12] , and Transformer architecture [4, 13, 5, 6] , which collectively broaden the functional scope of SNNs.Despite the promising advances in SNNs, several challenges persist when adapting them to diverse tasks. A fundamental challenge is that SNNs, which are event-triggered, lack robust and effective mechanisms to capture indexing information, rhythmic patterns, and periodic data. This limitation can adversely affect SNNs' ability to process and analyze different data modalities, including natural language, and time series. Meanwhile, while SNNs aim to emulate the neural circuits of the brain, their reliance on spike-based communication imposes limitations. Consequently, not all deep learning techniques applicable to ANNs can be directly transferred to SNNs. For instance, methods like HiPPO [14] or trigonometric positional encoding [15] are not readily compatible with the spike format used in SNNs. Moreover, even the most state-of-the-art ANNs still lag significantly behind human capabilities in many tasks [16, 17] . Therefore, to enhance the functionality of SNNs, one promising approach is to draw further inspiration from biological neural mechanisms. In this regard, we propose the analogy of central pattern generators (CPGs) [18] , a kind of neural circuit well-known in neuroscience, with positional encoding (PE), a technique extensively utilized in deep learning. This analogy is designed to operate within the SNN framework, potentially bridging the gap between biologically inspired models and modern deep learning techniques.In neuroscience, a CPG (See Figure 2 for an illustration) is a group of neurons capable of producing rhythmic patterned outputs without requiring rhythmic inputs [19, 20] . These neural circuits are found in the spinal cord and brainstem and are responsible for generating the rhythmic signals that control vital activities such as locomotion, respiration, and chewing [21] .On the other hand, PE is an important technique for ANNs, particularly within models tailored for sequence processing task [15, 22, 23] . By endowing each element of the input sequence with positional information, typically achieved through diverse mathematical formulations or learnable embeddings, neural networks acquire the capability to discern the order and relative positions of the elements within the sequence.We argue that these two concepts, despite seemingly unrelated, can be connected profoundly. Intuitively, CPG and PE both generate periodic outputs (with respect to time for CPG and with respective to position for PE). Moreover, in this paper, we reveal a deeper relationship between these two concepts by showing that the widely used sinusoidal PE is mathematically a particular solution of the membrane potential dynamics of a specific CPG.However, current SNN architectures exhibit a notable deficiency in implementing an effective and biologically plausible PE mechanism. Existing so-called positional encoding methods for SNNs [4, 5] rely on input data, often resulting in non-spike and repetitive outputs for different positions. Furthermore, incorporating PE techniques designed for ANNs necessitates the calculation of membrane potentials, which is incompatible with the spike format of SNNs. To address these issues, we draw inspiration from the spiking properties of the CPGs and propose a straightforward yet versatile PE technique for SNNs, termed CPG-PE. This method encodes positional information with multiple neurons with various patterns of spike trains. To summarize the highlights of our study:• Novel Positional Encoding for SNNs. We introduce a bio-plausible and effective PE approach tailored specifically for SNNs. This innovative strategy draws inspiration from the central pattern generator found in the human brain. Additionally, we propose a straightforward implementation of CPG-PE in SNNs, which is also compatible with neuromorphic hardware as it can be realized using circuits of leaky integrate-and-fire neurons.• Consistent Performance Gain. Our proposed methods significantly and consistently enhance the performance of SNNs across a wide range of sequential tasks, including timeseries forecasting and text classification.• Insightful Analysis. Our research represents one of the pioneering efforts to comprehensively analyze (1) the mechanism by which SNNs capture positional information and (2) the role of CPGs in the brain. This analysis provides valuable insights into the underlying principles of neural computation. 
spiking neural networks: 
 The basic unit in SNNs is the spiking neuron, such as the leaky integrate-and-fire (LIF) neuron [1] , which operates based on an input current I(t) and influences the membrane potential U (t) and the spike S(t) at time t. The dynamics of the LIF neuron are described by the following equations:EQUATIONEQUATIONEQUATIONHere, I(t) is the spatial input to the LIF neuron at time step t, calculated using the function f with x as input and θ as learnable parameters. ∆t is the discretization constant that determines the granularity of LIF modeling, and H(t) is the temporal output of the neuron at time step t. The spike S(t) is defined as a Heaviside step function based on the membrane potential. When U (t) reaches the threshold U thr , the neuron fires, emitting a spike, and the temporal output H(t) resets to V reset .If the membrane potential U (t) does not reach the threshold, no spike is emitted, and U (t) decays to H(t) at a decay rate of β.In this paper, we choose direct training with surrogate gradients as our method to train SNNs. we follow [24] to choose the arctangent-like surrogate gradients as our error estimation function when backpropagation, which regards the Heaviside step function as:S(t) ≈ 1 π arctan( π 2 αU (t)) + 1 2, where α is a hyper-parameter to control the frequency of the arctangent function. Therefore, the gradients of S are ∂S(t)∂U (t) = α 2 1 (1+( π 2 αU (t)) 2 )and thus the overall model can be trained in an end-to-end manner with back-propagation through time (BPTT) [25] .In the field of sequential tasks, PE is crucial for models like Transformers to understand the sequential order of input tokens. Absolute PE and relative PE are two prominent methods used to incorporate positional information into these models. Absolute PE [15] assigns fixed embeddings to each position in the input sequence using trigonometric functions like sine and cosine. These embeddings are based solely on the position index and are not influenced by the token content, which are predefined and are generated as follows:EQUATIONHere, pos is the position and d is the dimension. In contrast, relative PE [26] [27] [28] captures the relationships between tokens by considering their relative distances. This dynamic approach allows models to learn position-specific patterns and dependencies, which is beneficial for tasks requiring different sequence lengths or hierarchical structures.However, existing SNN architectures reveal a notable deficiency in the integration of an effective and biologically plausible PE mechanism. As shown in Figure 1 , current Transformer-based SNNs [4, 5] are primarily tailored for image classification and predominantly rely on a convolutional layer to capture the relative positional information of image patches. However, this approach resembles more of a spike-element-wise (SEW) residual connection [2] rather than a classic PE module, as it does not ensure that each image patch has a unique spike-form positional representation. Furthermore, the addition between positional spikes and the original input spikes within these models may yield hardware-unfriendly non-binary integers (i.e., neither 0 nor 1), resulting from the addition of "1" and "1". Additionally, our investigation reveals that even SNNs designed for sequential tasks, such as SpikeBERT [29, 30] , SpikeGPT [31] , and SpikeTCN [32] , also exhibit a notable absence of an effective spike-form PE mechanism for capturing positional information.We think that an effective PE strategy should possess the following characteristics: uniqueness of each position and the capacity to capture positional information from the input data. Furthermore, an optimal PE designed for SNNs should be hardware-friendly and in spike-form.Central Pattern Generators (CPGs) are neural networks capable of producing rhythmic patterned outputs without sensory feedback [18, 20] . These networks are fundamental for understanding motor control in vertebrates and invertebrates and are often applied to robotics and neural control systems. Mathematically, CPGs can be modeled using systems of coupled nonlinear oscillators, and the general form can be written as:EQUATIONwhere x and y are the state variables (can be seen as membrane potential) of two coupled oscillators, F and H are intrinsic dynamics of the oscillators, and G and K are the coupling functions. 
methods: 
 In biological systems, CPGs as well as other neurons do not transmit information directly through membrane potential but through spikes. A burst of spikes will be generated only when the membrane potential of a CPG exceeds a certain threshold. Therefore, we introduced the Heaviside step function in SNN, selecting only the part that exceeds the threshold, to design the CPG-PE. In this section, we will first reveal the relationship between CPGs and PE. Then we will introduce our proposed CPG-PE and its implementations.Consider one of the simplest CPGs with the following assumptions:1. The CPG is a coupled nonlinear oscillator with 2 neurons whose states are represented as x(t) and y(t).2. Both neurons are autonomic neurons and will gain membrane voltage with constant speed, i.e., F(x) = b > 0, H(y) = d > 0. 3. Neuron represented by x will inhibits y while y excites x. And the influence is proportional to the other neuron's state. Formally, G(x, y) = ay, K(x, y) = -cx where a > 0, c > 0.Now the coupled oscillators can be represented as:EQUATIONThe general solution of this differential equation system is:EQUATIONEQUATIONwhere k 1 and k 2 are arbitrary constants. To simplify, we can further re-parameterize t with t ′ = t + arctan(k 1 /ak 2 ) as is to choose another start point, then we can rewrite Equations ( 7) and (8) as:EQUATIONEQUATION)Comparing Equations ( 9) and (10) and Equation ( 4), we are astonished to find that the PE in Transformers [15] is a particular solution of the membrane potential variations in a specific type of CPG with properly chosen a, b, c, d. This finding suggests that the use of sinusoidal PE in Transformers is actually a bio-plausible choice that could possibly advance the model's ability to learn indexing and periodic information. Consider a system with N pairs of CPG neurons, resulting in a total of 2N cells. Then for i = 1, 2, ..., N , the equations governing the CPG-PE are as follows:EQUATIONEQUATION)where η is a constant to control the period, τ represents the base period, and v thres denotes the membrane potential threshold. Note that this threshold is different from the U thr of spike neurons described in Equation (3). The Heaviside step function H reflects a spike when the membrane potential exceeds the threshold.It is important to clarify that the t in Equation ( 11) and 12 is neither the time step in SNNs nor the position index. Suppose the input spike matrix X ∈ {0, 1} T ×B×L×D , where T is the time step in SNNs, B is the batch size, L is the sequence length of the input sample, D is the feature dimension.To ensure the uniqueness of each position at every time step, we flatten the dimensions T and L into a new dimension T × L. Therefore, t ranges from 0 to T × L. Notably, the entire CPG-PE operates in spike-form and is parameter-free. To better understand CPG-PE, we draw a simple approximation of the resulting CPG spiking patterns under the assumption of a sequence length of L = 128 and N = 20 pairs of CPG neurons, illustrated in Figure 2 (b).We design a simple implementation to apply CPG-PE to SNNs in a pluggable and hardware-friendly manner, shown in Figure 3 . Before diving into the details, we want to emphasize that the data transmitted in SNNs should always be in spike-form. Therefore, the direct addition operation between two spike matrices, as used in [4, 5] , should be forbidden. Initially, CPG-PE encodes the positional information of the input spike matrix X, resulting in X ′ . Then, to maintain binary values and avoid introducing non-binary elements, we opt to concatenate X and X ′ along the feature dimension. Lastly, a linear layer is employed to map the feature dimension from D + 2N back to D, where D is the feature dimension of X, and N is the number of CPG pairs. This effectively neutralizes the dimensional increase caused by concatenation. The whole process can be formalized as follows:EQUATIONEQUATION)where BN represents batch normalization and SN is a spike neuron layer. Furthermore, CPG-PE necessitates that input samples be sequential data, making it directly applicable to time series data and natural language. For image data, however, an adaptation is required: images must be segmented into patches similar to the approach used in the Vision Transformer [23] . Considering the compatibility with neuromorphic hardware, we also (1) implement CPG-PE with LIF neurons, and (2) integrate CPG-PE into a classic linear layer. Please refer to Appendices C and D for details. 
experiments: 
 In this section, we conduct experiments to investigate the following research questions: RQ1: Is our design of CPG-PE strategy effective and robust in sequential tasks? RQ2: Can CPG-PE work well on image patches that have no inherent order? RQ3: How will CPG's inner properties influence CPG-PE? RQ4: Does our CPG-PE satisfy the requirements of a good PE tailored for SNNs?To assess the PE capabilities of the compared models and answer RQ1, we conduct two sequential tasks: time-series forecasting, and text classification. Following [32] , we choose 4 real-world datasets for time-series forecasting: Metr-la [33] : This dataset contains the average traffic speed data collected from the highways in Los Angeles County. Pems-bay [33] : It consists of average traffic speed data from the Bay Area. Electricity [34] : This dataset captures hourly electricity consumption measured in kilowatt-hours (kWh). Solar [34] : It includes data on solar power production. For text classification, we follow [29] to conduct experiments on 6 benchmarks including: Movie Reviews [35] , SST-2 [36] , SST-5, Subj, ChnSenti, and Waimai. In addition, to answer RQ2, we also conduct image classification experiments on 1 static datasets CIFAR and 1 neuromorphic datasets CIFAR10-DVS [37] . The dataset details and metrics are provided in Appendix A.As discussed in Section 3.3, our proposed CPG-PE can be seamlessly integrated into any SNN capable of sequence processing. Consequently, we applied CPG-PE to the SNN counterparts of Temporal Convolutional Networks (TCN) [38] , Recurrent Neural Networks (RNN) to assess the efficacy of our method in enabling SNNs to capture positional information. The results for TCN, SpikeTCN w/o PE, RNN, and Spike-RNN w/o PE are sourced from the previous study by [32] . In addition, we deliberately conducted experiments on PE in Spikformer to explore whether our specially designed CPG-PE is truly more suitable for SNNs than all previous PEs. Notably, we also investigated the modularization of CPG, i.e., replacing all Linear layers with CPG-Linear layers (See Appendix D), and its impact on the Spikformer model for time-series forecasting, i.e., Spikformer w/ CPG-Full. We report the results on 4 time-series forecasting benchmarks with various prediction lengths in Table 1 .We also list results from ANNs for reference.In summary, the results presented in Table 1 indicate that SNNs equipped with the CPG-PE module significantly outperform their counterparts lacking the PE feature. This finding effectively addresses RQ1 from a time-series analysis perspective. Detailed findings include:(1) CPG-PE enables SNNs to successfully capture positional information. SNNs, including models such as Spike-TCN, Spike-RNN, and Spikformer, when integrated with CPG-PE, show superior performance compared to those without PE. Notably, CPG-PE also reduces the performance disparity between SNNs and traditional ANNs in time-series forecasting tasks, evidenced by an average increase of 0.013 in R 2 and a decrease of 0.022 in RSE.(2) CPG-PE is the most suitable position encoding strategy for Spikformer. In addition to CPG-PE, other encoding strategies such as Float-PE (the original PE in Transformer) and RPE (the original PE in Spikformer) were also evaluated. The Spikformer equipped with CPG-PE emerged as the top-performing variant, confirming CPG-PE as the most suitable PE strategy for SNNs.(3) CPG-Full module can also effectively model the positional information of time series data.The CPG-Full module's performance in modeling positional information of time-series data is comparable to that of CPG-PE, with average R 2 values nearly identical to those of Spikformer with CPG-PE and significantly better than those of other models.In addition to time-series forecasting, natural language processing (NLP) serves as another critical domain to assess the efficacy of the CPG-PE module in encoding positional information. Following the pioneering work of [29] , who first employed Spikformer for text classification tasks, we extended this application to 6 benchmark datasets. We also include results from fine-tuned BERT for reference. The results presented in Table 2 shows that Spikformer enhanced with CPG-PE achieves the state-ofthe-art performance across 6 benchmarks, effectively addressing RQ1. Meanwhile, we conducted a set of ablation experiments to eliminate the effects of increased parameter counts on model performance. Specifically, we replaced the spike-form positional encoding matrix obtained from CPG with a randomly generated spike matrix (See "Spikformer w/ Random PE" Row). By comparing these two configurations, we confirmed the effectiveness of our proposed CPG-PE.In this section, we aim to answer RQ2. To adapt the CPG-PE for image classification, it is essential to conceptualize the array of image patches as sequential data. Consequently, some SNN models that do not incorporate a concept of "sequence length" in their spike matrices, such as SEW-Resnet [2] , are incompatible with the integration of a CPG-PE module. Therefore, we only consider ViT-liked SNN, i.e. Spikformer, in this experiment. We also include results from ViTs for reference. We report the parameter counts and classification accuracy in Table 5 . To elaborate, Spikformer with CPG-PE outperforms other variants, demonstrating the effectiveness of CPG-PE even when the sequence is an array of image patches lacking inherent order. Notably, owing to our streamlined implementation, the parameter count for Spikformer with CPG-PE is significantly reduced compared to the original Spikformer w/ RPE [4] , with a reduction of 1.16 M. What's more, we conducted ablation experiments on model parameters by reducing the parameter count of Spikformer with CPG-PE to be comparable to Spikformer w/o PE, allowing for a more direct performance comparison, as shown in the last line in Table 5 . The results on ImageNet are reported in Appendix E.However, it is essential to acknowledge that the improvements in image classification are relatively modest compared to those observed in time series and text applications. This phenomenon can largely be attributed to the intrinsic non-ordered nature of image patches. Unlike text or time series data, where sequential order is crucial and inherently informative, image patches do not possess a natural or fixed sequence. This lack of order means that traditional methods of positional encoding, which significantly benefit ordered data by providing contextual positioning, are less effective. Thus, the application of our positional encoding techniques, optimized for data with inherent sequential order, does not translate as effectively to the domain of image classification.In this section, we investigate the influence of CPG properties on the ability to model positional information, addressing RQ3. To this end, we evaluated the Spikformer model with CPG-PE by varying the base period τ and the number of CPG pairs N (see Equations ( 11) and ( 12)) in time-series forecasting and image classification tasks.From Figure 4 (a) and (b), we observe that CPG-PE is insensitive to the base period τ (in biological neurons, τ is affected by the physiological properties of the CPG circuit such as RC constant and synaptic delay). The sequence lengths (T × L) of the time series and image patches are no larger than 672 (4 × 168) for all benchmarks, preventing repetitions in CPGs. Therefore, when N = 20, sweeping τ ∈ {100, 1000, 5000, 10000} makes minor influence on performance. Furthermore, Figure 4 (c) and (d) demonstrate that when τ = 10000, increasing the number of CPG pairs N enhances Spikformer's performance. This is reasonable because more CPG neurons reduce repetitions in positional representations of X ′ . In this section, we want to address RQ4. As mentioned in Section 2.2, an ideal PE method for SNNs should include the following characteristics: (1) Uniqueness of each position; (2) Compatibility with neuromorphic hardware; (3) Formulation in spike-form. Our implementations ensure compatibility with neuromorphic hardware (2), and the CPG-PE is inherently formulated in spikeform, satisfying (3). Therefore, in order to assess (1), the uniqueness of each position, we would like to compare the CNN-based RPE in [4, 5] and our proposed CPG-PE, focusing specifically on their capacity to provide distinct positional signals. This analysis was conducted using the CIFAR10-DVS dataset, where we calculated the repetition rate of spike positional representations across all positions. Our findings were notable: the positional spike matrices produced by RPE exhibited a repetition rate as high as 12.19%, which significantly undermines its effectiveness for PE. In contrast, our proposed CPG-PE exhibited no repetition, demonstrating that our CPG-PE is well-suited for serving as the PE module in SNNs. Please refer to Appendix B for details. 
spike encoding methods: 
 Spiking neural networks employ several coding methods to encode input information, each offering unique advantages. Direct coding [5, 40] , the simplest form and widely-used in image tasks, directly associates spikes with specific values or events, providing straightforward and interpretable outputs but often lacking efficiency for complex tasks. Rate coding [8, 41] , where the input is represented by the frequency of spikes within a given timeframe, is more robust and widely used but can be less precise due to its reliance on averaged spike rates. Temporal coding (a.k.a latency coding) [42, 43] encodes information based on the timing of individual spikes, allowing for high temporal precision and efficient representation of dynamic inputs, though it can be computationally demanding. In addition, delta coding [44] represents changes in input signals through spikes, focusing on differences rather than absolute values, which can enhance efficiency and response times but may introduce complexity in decoding. Each of these methods contributes to the versatility and applicability of SNNs in various domains, from neuroscience to artificial intelligence. The SNNs we considered in this paper should fall into the category of rate coding since back-prop is conducted on spike rate. Meanwhile, CPG-PE can be considered converting temporal information into spike rate of a group of neurons (Equations 11 and 12), and this is why CPG-PE can improve performance for sequential data. It is possible to introducing learning algorithms of temporal coding for the CPG neurons to tackle more complex sequence structure, which remains as future work.Currently, few works have demonstrated the importance of PE approaches in SNNs. Spikformer [4] and Spike-driven Transformer [5] utilize a combination of "one convolutional layer + one batch normalization layer + one spiking neuron layer " to generate learnable "relative positional encoding". From our perspective, this strategy is more like a spike-element-wise residual connection [2] , rather than a classic PE module. The unique representation of each position is a fundamental requirement for a robust PE module. However, the spike position matrix generated by their method may result in the same spike representation for different positions. Additionally, the addition of the input spike matrix and the position spike matrix will result in the occurrence of non-binary numbers (i.e., 2) due to the addition of 1 and 1. For spiking graph neural networks, [45] proposed learnable positional graph spikes, aiming to capture neighbor information within graphs rather than sequences. Therefore, drawing inspiration from the periodic automatic spike generation pattern of CPGs, we propose a biologically plausible and effective spike-form absolute position encoding method called CPG-PE. 
rethinking the role of cpgs in neuroscience: 
 Our study also provides novel insights into neuroscience on understanding the role of CPG in nervous systems. While traditionally CPG is believed to play a crucial role in producing the rhythmic motor patterns necessary for locomotion and other repetitive movements [18, 20] , the analogy to PE in this work reveals that CPG can make a significant contribution in processing sequential data by encoding the positional information into unique spiking patterns at different times. This does not only work for time-series sensory input like auditory signals but also for visual sensory data: e.g. when a person looks at an image, saccades (eye movements) allow retinal neurons to receive different parts of the image at different times. This indicates that CPG neurons could potentially be utilized to encode positional information. Another extensive thought is that as PE can be learnable in ANNs, CPG may also benefit from adaptability to the data [46] . The hypothesis, however, remains to be examined through neuroscientific experiments [47] .Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have shown our experimental settings and implementation details in Appendices A and B respectively.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In conclusion, inspired by central pattern generators, we introduce a pioneering position encoding approach termed CPG-PE, specifically tailored to mitigate the constraints associated with current PE techniques within SNNs. We mathematically prove that abstract PE in the Transformer is a particular solution of the membrane potential variations in a specific type of CPG. Furthermore, through comprehensive empirical investigations across diverse domains including time-series forecasting, natural language processing, and image classification, we demonstrate that the CPG-PE satisfies all the requirements of PE tailored for SNNs. The limitations and future work are discussed in Appendix F. resolution of 32 × 32 pixels. The CIFAR10-DVS dataset represents a neuromorphic adaptation of this original set, where static images have been transformed to accommodate the recording capabilities of a Dynamic Vision Sensor (DVS) camera. This conversion results in a dataset consisting of 9, 000 training samples and 1, 000 test samples with 128 × 128 resolution.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Our reported results are all averaged over several random seeds. We have reported the error bar of the results in Table 2 .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Metrices The metrics we used in time-series forecasting are the coefficient of determination (R 2 ) and the Root Relative Squared Error (RSE).EQUATION)EQUATION)In these formulas, M symbolizes the size of the test sets, C denotes the number of channels, and L signifies the length of predictions. Ȳ represents the average of Y m . The term Y m c,l refers to the l-th future value of the c-th variable for the m-th sample, while Ȳc,l indicates the mean of Y m c,l across all samples. The symbols Ŷm and Ŷ m c,l are used to represent the ground truth values. Compared to Mean Squared Error (MSE) or Mean Absolute Error (MAE), these metrics offer greater resilience against the absolute values of the datasets, making them particularly useful in the time-series forecasting setting.Model Architecture All SNNs take 4 time steps. For SpikeTCNs and SpikeRNNs, we follow the same settings as [32] . We construct all Spikformer as 2 blocks, setting the feature dimension as 256, and the hidden feature dimension in FFN as 1024. For CPG-PE settings, we set τ = 10000.0, N = 20, η = 1, and v thres = 0.8.Training Hyper-parameters we set the training batch size as 64 and adopt Adam [48] optimizer with a cosine scheduler of learning rate 1 × 10 -4 . An early stopping strategy with a tolerance of 30 epochs is adopted. We conducted time-series forecasting experiments on 24G-V100 GPUs. On average, a single experiment takes about 1 hour under the settings above.Model Achirecture All models are with 12 encoder blocks and 768 feature embedding dimension. It is important to note that the original implementation of [29] incorporates a layer normalization module that poses challenges to hardware compatibility. To address this, we have substituted layer normalization with batch normalization in our directly-trained Spikformer models for text classification tasks. For CPG-PE settings, we set τ = 10000.0, N = 20, η = 1, and v thres = 0.8.Training Hyper-parameters We directly trained Spikformers with arctangent surrogate gradients on all datasets. We use the BERT-Tokenizer in Huggingface § to tokenize the sentences to token sequences. We pad all samples to the same sequence length of 256. We conducted text classification experiments on 4 RTX-3090 GPUs, and set the batch size as 32, optimizer as AdamW [49] with weight decay of 5 × 10 -3 , and set a cosine scheduler of starting learning rate of 5 × 10 -4 . What's more, in order to speed up the training stage, we adopt the automatic mixed precision training strategy. On average, a single experiment takes about 1.5 hours under the settings above.Model Architecture For all Spikformer models, we standardized the configuration to include 4 time steps. Specifically, for the CIFAR10 and CIFAR100 datasets, the models were uniformized § https://huggingface.co/ with 4 encoder blocks and a feature embedding dimension of 384. For the CIFAR10-DVS dataset, the models were adjusted to have 2 encoder blocks and a feature embedding dimension of 256. For CPG-PE settings, we set τ = 10000.0, N = 20, η = 2π, and v thres = 0.8.We honestly follow the experimental settings in [4] , whose source code and configuration files are available at https://github.com/ZK-Zhou/spikformer. As the training epochs are quite big (300 epochs) in their settings, we choose to use one 80G-A100 GPU, and it takes about 3 hours to conduct a single experiment, on average.We conducted positional encoding analysis experiments on the CIFAR10-DVS dataset. For the original Spikformer with relative positional encoding (RPE) as described by [4] , the input and output channels of Conv2d are both set to 384. In our Spikformer with CPG-PE, the parameters are set to τ = 10000.0 and N = 20. Given the time step T = 4 and the sequence length L = 160 for the image patches in CIFAR10-DVS samples, the total "length" T × L in CPG-PE is 640. We then calculated the repetition rate of positions. The results showed that the repetition rate for RPE is 12.19%, whereas for CPG-PE, it is 0.00%.In this section, we demonstrate that CPG-PE is a hardware-friendly design. While implementing the sinusoidal potential on the neuromorphic chips is not challenging (e.g., by maintaining additional LC circuits), we show how a CPG-PE neuron can be physically implemented with only 2 LIF neurons defined by Equations ( 1) to ( 3) and thus introducing no extra efforts on chip designs.A CPG-PE neuron, after discretization, can be viewed as an autonomic neuron that will emit a burst of K spikes after resting for R time steps. The key idea is to set two LIF neurons, namely the Emitter and the Resetter. The emitter will draw constant current from the source, and as soon as its membrane potential reaches the threshold after R time steps, it will start emitting spikes constantly until receiving the reset signal from the resetter. The resetter, which will remain at the resting potential until it receives signals from the emitter, will count the number of spikes and emit a reset signal (inhibition signal) to the emitter after receiving K spikes.We first prove the following Lemma, which establishes the relationship between the start time of the first spike and a constant input current. Lemma 1. Given an LIF neuron defined by Equations (1) to (3) with decay rate β and threshold U thr , starting with resting potential U (0) = 0, if fed with the constant current I(t) = I c > 0, the first spike will emit at:EQUATIONProof. By definition, before the time to emit the first spike, we have S(t) = 0. Thus Equation ( 1) can be rewrite as:EQUATIONSimplifying the recurrence relation, we can obtain:EQUATIONThe first spike is generated when U (k∆t) ≤ U thr , thus we have:EQUATIONthat is to say,EQUATIONNow we can implement the CPG-PE with LIF neurons: Theorem 1. Given 2 LIF neurons, the emitter and the resetter, with decay rate β, threshold U thr , and reset potential V reset , starting with resting potential U e (0) = U r (0) = 0. IfEQUATIONEQUATIONEQUATIONEQUATIONthen the system will have the period of T = (R + K)∆t, andEQUATIONProof. Assuming the first spike generated by the emitter emits at time step T 1 . For every 0 ≤ t < T 1 , we have:EQUATIONEQUATIONSince the input current of the emitter is a constant, by Lemma 1, we immediately get:EQUATIONStarting from time step R, let's assume the first spike generated by the resetter emits at time step T 2 .Then for every T 1 ≤ t < T 2 , we have:EQUATIONEQUATIONStarting from T 1 , for the emitter, the input current allows a spike event for every time step. And the input current of the resetter is a constant. Again, by applying Lemma 1, we can get:EQUATIONNow Consider the state at time step R + K:EQUATIONEQUATIONEQUATIONThis is the same as the membrane potential at time step 0. Therefore, the system will behave periodically with period T = (R + K)∆T .Theorem 1 gives a possible CPG-PE design with 2 LIF neurons, with the emitter generating K consecutive spikes every R + K time steps. This demonstrates that incorporating CPG-PE into the current SNN architecture is completely bio-plausible and will not introduce any burden of redesigning hardware.We have developed a simple modularization implementation to integrate our proposed CPG-PE with original linear layers, as depicted in Figure 3 (b) . Consider the original linear layer's input and output dimensions as D in and D out , respectively. Our objective is to incorporate CPG-PE within this framework. Following the application of the CPG-PE module, the modified input X ′ is obtained. X is then input into Linear 1 and X ′ into Linear 2 , resulting in outputs X 1 and X 2 , respectively. Both Linear 1 and Linear 2 maintain an output dimension of D out . The final step involves summing X 1 and X 2 to produce X 3 , which is subsequently processed through batch normalization (BN) and spike normalization (SN ). We term this implementation as "CPG-Linear" and formulize as follows:EQUATIONEQUATIONEQUATIONX output = SN (BN (X 3 )) , X out ∈ {0, 1} T ×B×L×Dout (40) where + denotes element-wise addition. This implementation described above is fundamentally identical to Figure 3 , within the context of a single linear layer. However, the CPG-Linear can seamlessly replace any linear layer in SNNs.We have conducted experiments with Spikformer without positional encoding (PE), Spikformer with relative positional encoding (RPE), and Spikformer with our proposed CPG-PE on the ImageNet dataset. The results are as follows: In this section, we will discuss the limitations and future works of our paper.As mentioned in Section 3.3, our CPG-PE can not be directly applied to those SNNs where spike matrices do not have a "sequence length" dimension. Our CPG-PE is optimized for processing sequential data, making it ideal for applications involving time series or natural language. This intrinsic design, however, does not naturally extend to image data, which typically benefits from direct convolutional operations that capture spatial relationships across the entire image dimensions-height and width. In contrast, CPG-PE requires the segmentation of images into patches, a method inspired by the Vision Transformer. This adaptation contrasts with approaches like the Convolutional 2D layer, which applies convolution operations directly across the height and width of an image without requiring segmentation into smaller, discrete patches. The necessity to adapt CPG-PE for image data through patching can introduce complexities and potential performance bottlenecks, as it may not effectively capture the continuous spatial relationships and local features in the image, which are crucial for tasks such as object recognition and scene understanding.To enhance the applicability of the CPG-PE model to a broader range of data types, especially image data, future research could focus on developing a hybrid model that integrates the strengths of CPG-PE with traditional convolutional layers. This integration could potentially allow the model to handle both sequential and spatial data efficiently without the need for extensive pre-processing or adaptation. Specifically, integrating direct convolution operations that work across the entire spatial dimensions of an image within the CPG-PE architecture could help preserve spatial relationships and improve feature extraction capabilities. Additionally, exploring the use of adaptive patch sizes or dynamically adjusting the patching mechanism based on the nature of the input data could also provide a more flexible and performance-optimized approach. These advancements would make the model more versatile and capable of tackling a wider array of tasks across different domains.Additionally, considering that CPG-PE is an absolute positional encoding designed for SNNs, it could be beneficial to explore the potential of implementing learnable relative positional encodings in SNNs. Such encodings would need to be developed to meet specific criteria: they must maintain the spikeform characteristic essential to SNNs and ensure the uniqueness of each position's encoding. This approach could significantly enhance the model's ability to capture and utilize the temporal dynamics of input data more effectively, potentially leading to more nuanced and context-aware processing capabilities. Exploring adaptive patch sizes or dynamically adjusting the patching mechanism based on the nature of the input data could also provide a more flexible and performance-optimized approach. These advancements would make the model more versatile and capable of tackling a wider array of tasks across different domains.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We have submitted our source code in Supplementary Material. We will upload our code and data to Github upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.‡ Note that this is not a real relative PE. This term is adopted from the original papers. 


Decision: All reviewers agree that this is an interesting work that can be accepted for NeurIPS. The paper proposes biologically inspired central pattern generators (CPGs) as a novel method for position encoding in spiking neural networks (SNNs).
The reviewers noted that the work
-    covers new ground for sequential processing with SNNs,
-    has a strong theoretical foundation,
-    provides a comprehensive empirical evaluation on a variety (and also large) data sets,
-    shows significantly improved performance over previous SNN models,
-    is well written and has insightful analysis.
I therefore recommend to accept the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces central pattern generators (CPGs) from neuroscience into the SNN framework as a novel method for position encoding. Through mathematical derivation, it is proven that the existing abstract PE methods in transformers are actually a particular solution for a specific type of CPG. The effectiveness of CPG is validated through experiments across several domain benchmarks.
strength_weakness: strength_weakness: 1.	The authors have not convinced me why the problem addressed in this paper is very important, i.e., why the existing PE methods in SNNs are such a big problem that they need to be improved by CPG.


2.	The method proposed in the paper is simple and does not provide enough inspiring insights; the contribution is relatively limited.

3.	The implementation of CPG-PE on hardware involves coupled nonlinear oscillators that require frequent updates of neuron membrane potentials, entailing floating-point computations and memory read-write operations, which result in additional energy expenditures. The paper should scrutinize and analyze whether the performance enhancements afforded by this encoding method justify the additional energy costs. This trade-off demands a detailed examination to assess its viability in practical applications.

4.	In Sec3.1, why is "F(x)=b<=0, H(y)=d<=0" followed after “...gain membrane voltage with constant speed” instead of “F(x)=b>0, H(y)=d>0”?

5.	In Sec4.2, does the CPG-Full method replace all linear layers in the model with CPG-Linear layers? Why is its performance not as good as that of CPG-PE? Can you provide further analysis and explanation?
questions: questions: Please refer to weakness 3, 4, 5
limitations: limitations: NA
paper_summary: paper_summary: The lack of an effective and hardware-efficient spike-frm position encoding strategy in  SNNs has been a consistent motivation for this study.
Drawing inspiration from the central pattern generators (CPGs) in the human brain, which produce rhythmic patterned outputs without requiring rhythmic inputs, this work proposes a novel PE technique for SNNs, termed CPG-PE. 
Extensive experiments across various domains show the superior performance with CPG-PE.
strength_weakness: strength_weakness: 1.This type of positional encoding, through aggregation, introduces a small number of additional parameters and computational overhead. Please provide ablation experiments demonstrating that the performance improvement is not solely due to these factors.
2.I suggest the authors include results on ImageNet to demonstrate the effectiveness on large-scale datasets.
questions: questions: 1.In ANNs, positional encoding is typically added to features. Please analyze the similarities and differences of this category aggregation-based positional encoding compared to ANNs.
limitations: limitations: I believe the authors' discussion on limitations is comprehensive.
paper_summary: paper_summary: This paper introduces a novel positional encoding method for SNNs called CPG-PE, inspired by central pattern generators (CPGs) in biological neural systems. The authors demonstrate both theoretically and empirically that CPG-PE can effectively capture positional information in sequential data while maintaining the spike-based nature of SNNs. The approach is evaluated on a range of tasks including time-series forecasting, text classification, and image classification.
strength_weakness: strength_weakness: 1. Typographical error on line 158 where X is incorrectly specified as belonging to the real number domain instead of the binary domain {0,1} for spike data.
2. The authors should clarify the similarities and differences between the decay mechanism in CPG-PE and that of LIF neurons.
questions: questions: See my weakness part.
limitations: limitations: The authors have addressed limitations and future work in Appendix E, which is commendable. They discuss the challenges of applying CPG-PE to non-sequential data like images and propose potential solutions.
paper_summary: paper_summary: This paper proposes a new position encoding technique CPG-PE for SNNs, which is inspired by the central pattern generator in the human brain and improves the ability of SNNs to process sequence data. Experimental results show that CPG-PE outperforms traditional SNNs in multiple fields such as time series prediction, text classification and image classification.
strength_weakness: strength_weakness: 1. No experiments were conducted on the large-scale image classification Imagenet dataset.
2. Other encoding methods in SNN such as rate coding [1] and temporal coding [2;3], were not discussed.

[1] Kim Y, Park H, Moitra A, et al. Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks?[C]//ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 71-75.

[2] Han B, Roy K. Deep spiking neural network: Energy efficiency through time based coding[C]//European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 388-404.

[3] Comsa I M, Potempa K, Versari L, et al. Temporal coding in spiking neural networks with alpha synaptic function[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 8529-8533.
questions: questions: 1. I suppose the traditional positional encoding in Spikformer can be seen as direct encoding in SNNs. In contrast, the CPG-PE proposed in this paper is more like a temporal encoding with dynamic expressions related to time series. Could the authors discuss the differences between other encoding schemes [1;2;3] and positional encoding in SNNs within the related work?

2. Can the authors provide experimental results of CPG-PE on the ImageNet image classification dataset?
3. Some tyops: Formulas 4, 5, 6, 7, 8, and 10 are missing commas. There should be a comma after each formula.

I would be pleased to raise the score if the authors address my concerns.

[1] Kim Y, Park H, Moitra A, et al. Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks?[C]//ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 71-75.

[2] Han B, Roy K. Deep spiking neural network: Energy efficiency through time based coding[C]//European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 388-404.

[3] Comsa I M, Potempa K, Versari L, et al. Temporal coding in spiking neural networks with alpha synaptic function[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 8529-8533.
limitations: limitations: The author has discussed the limitations

Comments: title: Thank you
Response  to the rebuttal
Response to the rebuttal
comment: The reviewer thanks the authors for the discussion. It addressed all of my concerns. I decide to keep my score.
Thank you for your reply. I think this is a nice bit of discussion and could be added to the manuscript. In light of the additional discussion, I'd like to raise my score to a 7. This is an interesting piece of work and would be a nice addition to NeurIPS.
Thank you for your detailed response. The experiments have alleviated my concerns about the overhead of CPG-PE. The further validation on the large-scale ImageNet has demonstrated its generalization and effectiveness. Therefore, I would like to increase my score.
Thank you for the response to my questions. The idea of building positional encoding in SNN is interesting and I am willing to increase my score to 6.