Advancing Spiking Neural Networks for Sequential Modeling with Central Pattern Generators
Authors: Changze Lv
Dongqi Han
Yansen Wang
Xiaoqing Zheng
Xuanjing Huang
Dongsheng Li
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Spiking Neural Networks
Central Pattern Generators
Positional Encoding
Primary area: neuroscience_and_cognitive_science
TL;DR: Inspired by central pattern generators, we propose a novel positional encoding technique tailored for spiking neural networks.

Abstract: Spiking neural networks (SNNs) represent a promising approach to developing artificial neural networks that are both energy-efficient and biologically plausible.
However, applying SNNs to sequential tasks, such as text classification and time-series forecasting, has been hindered by the challenge of creating an effective and hardware-friendly spike-form positional encoding (PE) strategy.
Drawing inspiration from the central pattern generators (CPGs) in the human brain,  which produce rhythmic patterned outputs without requiring rhythmic inputs, we propose a novel PE technique for SNNs, termed CPG-PE.
We demonstrate that the commonly used sinusoidal PE is mathematically a specific solution to the membrane potential dynamics of a particular CPG.
Moreover, extensive experiments across various domains, including time-series forecasting, natural language processing, and image classification, show that SNNs with CPG-PE outperform their conventional counterparts.
Additionally, we perform analysis experiments to elucidate the mechanism through which SNNs encode positional information and to explore the function of CPGs in the human brain.
This investigation may offer valuable insights into the fundamental principles of neural computation.

Advancing Spiking Neural Networks for Sequential Modeling with Central Pattern Generators
Changze Lv Dongqi Han Yansen Wangat Xiaoqing Zheng"
Xuanjing Huang' Dongsheng Li²
School of Computer Science, Fudan University Microsoft Research Asia
(czlv22)@m. fudan. edu. cn, {zhengxq, xjhuang)@fudan, edw. cn, (yansenwang, dongqihan, dongsl 1 jemicrosott.com
Abstract
Spiking neural networks (SNNs) represent a promising approach to developing artificial neural networks that are both energy-efficient and biologically plausi- ble. However, applying SNNs to sequential tasks, such as text classification and time-series forecasting, has been hindered by the challenge of creating effective and hardware-friendee spike-form positional encoding (PE) strategy. Drawing inspiration from the central pattern generators (CPGs) in the human brain, which produce rhythmic patterned outputs without requiring rhythmic inputs, we propose novel PE technique for SNNs, termed CPG-PE. We demonstrate that the com- monly used sinusoidal PE is mathematically a specific solution to the membrane potential dynamics of a particular CPG. Moreover, extensive experiments across various domains. including time-series forecasting, natural language processing, and image classification, show that SNNs with CPG-PE outperform their conven- tional counterparts. Additionally, we perform analysis experiments to elucidate the mechanism through which SNNs encode positional information and explore the function of CPGs in the human brain. This investigation may offer valuable insights into the fundamental principles of neural computation. Our code is available at https: thub. wtt.cccccccc.cceccccc comm/mmccrosoot///ssS
1 Introduction
Spiking neural network (SNN) 1 has increasingly attracted research interests in recent years, primarily due to its energy efficiency, event-driven paradigm, biological plausibility, and other distinctive properties. The spiking neurons in SNN are dynamical systems that generate binary signals (spike or non-spike) and communicate these signals like artificial neural networks (ANNs) for computation [2-91. Many advanced architectures and methodologies developed for ANNs also applicable to SNNs, enhancing their capabilities. Notable among these are backpropagation [TO), batch normalization 12). and Transformer architecture 4 13
6), which collectively broaden
the functional scope SNNs.
Despite the promising advances in SNNs. several challenges persist when adapting them to diverse tasks. A fundamental challenge is that SNNs, which are event-triggered, lack robust and effective mechanisms to capture indexing information, rhythmic patterns, and periodic data. This limitation can adversely affect SNNs' ability to process and analyze different data modalities. including natural language, and time series. Meanwhile, while SNNs aim to emulate the neural circuits of the brain,
'The work was conducted during internship Changze Lv Microsoft Research Asia. Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

their reliance on spike-based communication imposes limitations. Consequently, not all deep learning techniques applicable ANNs can be directly transferred SNNs. For instance, methods like HiPPO [14] or trigonometric positional encoding [[5 are not readily compatible with the spike format used in SNNs. Moreover, even the most state-of-the-art ANNs still lag significantly behind human capabilities in many tasks [16 17.. Therefore, to enhance the functionality of SNNs, one promising approach is to draw further inspiration from biological neural mechanisms. In this regard, propose the analogy of central pattern generators (CPGs) [8). kind of neural circuit well-known in neuroscience, with positional encoding (PE), a technique extensively utilized in deep learning. This analogy is designed to operate within the SNN framework, potentially bridging the gap between biologically inspired models and modern deep learning techniques.
In neuroscience, a CPG (See Figure for an illustration) is group of neurons capable of producing rhythmic patterned outputs without requiring rhythmic inputs [9 20]. These neural circuits are found in the spinal cord and brainstem and are responsible for generating the rhythmic signals that control vital activities such as locomotion, respiration, and chewing (1
On the other hand, PE is an important technique for ANNs, particularly within models tailored for sequence processing task [15]
By endowing each element of the input sequence with positional information, typically achieved through diverse mathematical formulations or learnable embeddings. neural networks acquire the capability discern the order and relative positions of the elements within the sequence.
2
We argue that these two concepts, despite seemingly unrelated, be connected profoundly. Intu- itively, CPG and PE both generate periodic outputs (with respect to time for CPG and with respective to position for PE). Moreover, in this paper, we reveal deeper relationship between these two concepts by showing that the widely used sinusoidal PE is mathematically a particular solution of the membrane potential dynamics of a specific CPG.
However, current SNN architectures exhibit a notable deficiency implementing an effective and biologically plausible PE mechanism. Existing so-called positional encoding methods for SNNs [4] 5 rely on input data, often resulting in non-spike and repetitive outputs for different positions. Furthermore, incorporating PE techniques designed for ANNs necessitates the calculation of membrane potentials, which incompatible with the spike format of SNNs. To address these issues, we draw inspiration from the spiking properties of the CPGs and propose straightforward yet versatile PE technique for SNNs, termed CPG-PE. This method encodes positional information with multiple neurons with various patterns of spike trains. To summarize the highlights of our study:
Novel Positional Encoding for SNNs. We introduce bio-plausible and effective PE approach tailored specifically for SNNs. This innovative strategy draws inspiration from the central pattern generator found in the human brain. Additionally, we propose a straightfor- ward implementation of CPG-PE in SNNs, which is also compatible with neuromorphic hardware as it can be realized using eircuits of leaky integrate-and-fire neurons.
Consistent Performance Gain. Our proposed methods significantly and consistently enhance the performance of SNNs across a wide range of sequential tasks, including time- series forecasting and text classification.
Insightful Analysis. Our research represents one of the pioneering efforts to comprehen- sively analyze (1) the mechanism by which SNNs capture positional information and (2) the role of CPGs in the brain. This analysis provides valuable insights into the underlying principles of neural computation.
2 Preliminaries
2.1 Spiking Neural Networks
The basic unit in SNNs the spiking neuron, such as the leaky integrate-and-fire (L.IF) neuron D. which operates based on input current [(t) and influences the membrane potential (t) and the

---

Uniqueness Each Position
Non-uniqueness of Each Position in Spike-form
Uniqueness of Each Position in Spike-form
Time-series Forecasting
x
Non-spike Output +
Spike Output
X
0. 0,719
Addition
Addition
Concatenation
wil PE CPG-PE
Sin Cos
CPGs
Classification
79.41
Batch Norm
Conv2d
wio PE CPG-PE Image Classification
X
#1.25
Depends the
Depends the Sequence Length of
Depends the Whole
Sequence Length of
in Transformer
(b) PE in Spike Transformer
CPG-PE (Ours)
Performance
Figure 1: (a) Positional encoding (PE) in ANN Transformers. (b) Relative PE in Spike Transformers
6 (c) Our Proposed CPG-PE method. (d) CPG-PE consistently improves learning performance various tasks. CPG-PE is an ideal PE method tailored for SNNs, detailed in Section
spike Stt) at time The dynamies of the LIF neuron are described by the following equations:
U()) = H(I - At) + T()), T(t) = f(x;0),
H(1 Vreset
S(()) (U(),,
[1, U() 2 Uthr
S(t) = 0, if U() < Uthr
Here, I(t) is the spatial input to the LIF neuron at time step f, calculated using the function f with x as input and 0 as learnable parameters. At is the discretization constant that determines the granularity of LIF modeling, and (t) is the temporal output of the neuron at time step t. The spike (()) is defined as Heaviside step function based the membrane potential. When (t) reaches the threshold Uthr the neuron fires, emitting a spike, and the temporal output (t)) resets to Vreset- If the membrane potential (t) does not reach the threshold, no spike is emitted, and (t) decays to (t) at a decay rate of B.
In this paper, choose direct training with surrogate gradients our method to train SNNs. follow [24] to choose the aretangent-like surrogate gradients as our error estimation function when backpropagation, which regards the Heaviside step function as: S(t) 1 arctan( (t)( + 1 where a is a hyper-parameter to control the frequency the arctangent function. Therefore, the as(()
gradients of S are au(() 2 (L=(5((())))) and thus the overall model can be trained in an end-to-end manner with back-propagation through time (BPTT) 25
2.2 Positional Encoding
In the field of sequential tasks, PE is crucial for models like Transformers to understand the sequential order of imput tokens. Absolute PE and relative PE are two prominent methods used incorporate positional information into these models. Absolute PE [5] assigns fixed embeddings to each position in the input sequence using trigonometric functions like sine and cosine. These embeddings based solely on the position index and are not influenced by the token content, which are predefined and are generated as follows:
pos
pos
PE PE = sin (10000274))
PE (pos, = cos 10000
(4)
Here, pos is the position and d is the dimension. In contrast, relative PE 26 captures the relationships between tokens by considering their relative distances. This dynamic approach allows models to learn position-specific patterns and dependencies, which is beneficial for tasks requiring different sequence lengths hierarchical structures.
Note that this is real relative PE. This term is adopted from the original papers.

---

However, existing SNN architectures reveal a notable deficiency in the integration of an effective and biologically plausible PE mechanism. As shown in Figure current Transformer-basee are primarily tailored for image classification and predominantly rely on a convolutional layer to capture the relative positional information of image patches. However, this approach resembles more of a spike-element-wiss (SEW) residual connection [2] rather than a classic PE module, as it does not ensure that each image patch has unique spike-form positional representation. Furthermore, the addition between positional spikes and the original input spikes within these models may yield hardware-unfriendll non-binary integers (i.e., neither O nor 1), resulting from the addition of "1" and "1". Additionally, our investigation reveals that even SNNs designed for sequential tasks, such as SpikeBERT [29] 30), SpikeGPT [31], and SpikeTCN [32, also exhibit a notable absence of an effective spike-fore form PE mechanism for capturing positional information.
SNNs 4
We think that an effective PE strategy should possess the following characteristics: uniqueness of each position and the capacity to capture positional information from the input data. Furthermore, an optimal PE designed for SNNs should be hardware-friendly and in spike-form.
2.3 Central Pattern Generators
Central Pattern Generators (CPGs) are neural networks capable of producing rhythmic patterned outputs without sensory feedback [18 20). These networks are fundamental for understanding motor control in vertebrates and invertebrates and are often applied to robotics and neural control systems. Mathematically, CPGs can be modeled using systems of coupled nonlinear oscillators, and the general form can be written as:
F(x) + Gx,,)), ý H + k(x,))
(5)
where x and y are the state variables (can be seen as membrane potential) of two coupled oscillators, F and H are intrinsic dynamics of the oscillators, and G and K are the coupling functions.
3 Methods
In biological systems, CPGs as well as other neurons do not transmit information directly through membrane potential but through spikes. A burst of spikes will be generated only when the membrane potential of CPG exceeds certain threshold. Therefore, we introduced the Heaviside step function in SNN, selecting only the part that exceeds the threshold, to design the CPG-PE. In this section, we will first reveal the relationship between CPGs and PE. Then we will introduce our proposed CPG-PE and its implementations.
3.1 Relationship between Central Pattern Generators and Positional Encoding
Consider one of the simplest CPGs with the following assumptions:
1. The CPG is coupled nonlinear oscillator with 2 neurons whose states are represented as x (t) and (t).
2. Both neurons are autonomic neurons and will gain membrane voltage with constant speed, i.c., F(x) = > 0, H(y) = d > O.
3. Neuron represented by x will inhibits y while y excites x. And the influence is proportional to the other neuron's 's state. Formally, G = ay, K(x, y) where > 0, 0.
Now the coupled oscillators can be represented as:
x(() ay(() bb, ý = -xx=t) d.
(6)
The general solution of this differential equation system is:
x(1) hi cos(Var 1) + / a sin( 1) + d y(t) -ki "sin(Vac 1) his cos( /
b
(8)
a"

---

where 1ı and ke are arbitrary constants. To simplify, we can further re- re-parameterize & with t = t + arctan(k1 /ak2) is to choose another start point, then we can rewrite Equations and as:
" 3 Sacinvactto (') v(() c if + k2 (ooosvacc)e (')
x(t')= V
d c
A sin(wit() + b1.
(9)
A2 cos(wal') + b2.
(10)
Comparing Equations and 10) and Equation 4 we are astonished to find that the PE in Transformers [15] particular solution of the membrane potential variations in specific type of CPG with properly chosen a, b, c, d. This finding suggests that the use of sinusoidal PE in Transformers is actually a bio-plausible choice that could possibly advance the model's ability to learn indexing and periodic information.
10
16
20
90
36
40
46
50
10
babiba
Central Pattern Generators
time dhp
(a)
Figure 2:
(a) Illustration of pair of CPG neurons demonstrating mutual inhibition through spiking activity. The spikes represent neural spikes that inhibit each other, exemplifying the coordination mechanism in CPG networks. (b) Spike trains of the first 4 CPG neurons. The curve represents the membrane potential, while the vertical lines represent spikes.
3.2 CPG-based Positional Encoding
Consider a system with N pairs of CPG neurons, resulting in total of 2N cells. Then for i 1, 1,2, N, the equations governing the CPG-PE are as follows:
CPG-PE2i-1 H
cos
17
y
thres
(11)
T
CPG-PE" H sin
17 T
1 thres
(12)
where is constant to control the period, т represents the base period, and thres denotes the membrane potential threshold. Note that this threshold is different from the Uthr of spike neurons described in Equation (3) The Heaviside step function H reflects spike when the membrane potential exceeds the threshold.
important to clarify that the in Equation I1 and is neither the time step SNNs nor the position index. Suppose the input spike matrix X {0, BxXD where T is the time step in SNNs, B is the batch size, L is the sequence length of the input sample, D is the feature dimension. To ensure the uniqueness of each position every time step. flatten the dimensions T and L into a new dimension T x L. Therefore, f ranges from 0 to T L. Notably, the entire CPG-PE operates in spike-form and is parameter-free. To better understand CPG-PE, draw simple approximation of the resulting CPG spiking patterns under the assumption of a sequence length of 128 and N 20 pairs of CPG neurons, illustrated in F
3.3 Implementations
We design a simple implementation to apply CPG-PE to SNNs a pluggable and hardware-friendly manner, shown in Figure 3 Before diving into the details, we want to emphasize that the data transmitted SNNs should always be in spike-form. Therefore, the direct addition operation between two spike matrices, used in 45,, should be forbidden.
5

---

CPG-PE
Map (D 2N) To D
Spiking Neuron Layer
X"
X Input Spike Matrix
1=
X' Positional Encoding
1=2 2
Hneey
Na
X output
Time Steps SNNs
L Sequence Length of
1= 2N
N Number of CPG Pains
X
TXL
Index of CPG Neurons
Figure 3: Illustration of applying CPG-PE to SNNs. X. X', and Xoutput are all spike matrices.
Initially, CPG-PE encodes the positional information of the input spike matrix X, resulting in X'. Then, to maintain binary values and avoid introducing non-binary elements, we opt to concatenate X and X' along the feature dimension. Lastly, a linear layer is employed map the feature dimension from D + 2N back to D. where D is the feature dimension of X, and N is the number of CPG pairs. This effectively neutralizes the dimensional increase caused by concatenation. The whole process can be formalized as follows:
X' CPX))
X {0, (0,1]TxBxLxD, X' {0, (0, (0,1IXXXXXX2N
(13)
X1 = X + X',
X1 E TxxxXxxXX))
(14)
X Xoutput SN (BN (Linear (X1))),
Xoutput E (0.11
(15)
where BN represents batch normalization and SN is spike neuron layer. Furthermore, CPG-PE necessitates that input samples be sequential data, making it directly applicable to time series data and natural language. For image data, however, an adaptation required: images must be segmented into patches similar to the approach used in the Vision Transformer [23]. Considering the compatibility with neuromorphic hardware, we also (1) implement CPG-PE with LIF neurons, and (2) integrate CPG-PE into a classic linear layer. Please refer to Appendices C and for details.
4 Experiments
In this section, we conduct experiments investigate the following research questions: RQ1: Is our design of CPG-PE strategy effective and robust in sequential tasks?
RQ2: Can CPG-PE work well on image patches that have no inherent order?
RQ3: How will CPG's inner properties influence CPG-PE?
RQ4: Does our CPG-PE satisfy the requirements of a good PE tailored for SNNs?
4.1
Datasets
To assess the PE capabilities of the compared models and answer RQ1, we conduct two sequential tasks: time-series forecasting, and text classification. Following 32 we choose 4 real-world datasets for time-series forecasting: Metr-la [33]: This dataset contains the average traffic speed data
collected from the highways in Los Angeles County. Pems-bay :33: It consists average traffic speed data from the Bay Area. Electricity 34 This dataset captures hourly electricity consumption
measured in kilowatt-hours (kWh). Solar 34 It includes data on solar power production. For text classification, we follow [29] to conduct experiments on 6 benchmarks including: Movie Reviews [35), SST-2 (36). SST-5, Subj. ChnSenti, and Waimai. In addition, to answer RQ2, we also conduct image classification experiments static datasets CIFAR and neuromorphic datasets CIFAR DVS 37.. The dataset details and metrics are provided Appendix A
4.2 Time-Series Forecasting
As discussed in Section 3.3 our proposed CPG-PE can be seamlessly integrated into any SNN capable of sequence processing. Consequently, we applied CPG-PE to the SNN counterparts of Temporal Convolutional Networks (TCN) (38] Recurrent Neural Networks (RNN) to assess the efficacy of
our method in enabling SNNs to capture positional information. The results for TCN, SpikeTCNN w/o PE, RNN, and Spike-RNi w/o PE are sourced from the previous study by [32]. In addition,

---

Table 1: Experimental results of time- time-series forecasting on 4 benchmarks with various prediction lengths 6, 24, 48, 96. "PE" stands for positional encoding. "w/o" denotes "without" while "w/" denotes "with". The best results of SNNs are formatted in bold font format. t (L) indicates the higher (lower) the better. Shaded ones are ours. All results are averaged across 3 random seeds.
Madel
SWN
Sprie P1
Stunc
24
Matr-la
95
24
Peass bay
56
24
Solac
15
06
Electriitty
24
96
App
TANO
RSE.
.446
A20
2201
.065
.725
.456
.554
3350
.573
351
-723
-641
-283
-55
1567
1001
200
2525
.543
57.
63.5
122
0911
1563
375
222
3775
-257
.319
2955
.245-
2622
.77
.463
SpAcTCN
-0133
3775
311
7027
1551
101
837
105
1003
3210
315
2155
SpikTTNN of CRG-E
RSD
.089
.0164
-220
.859
-435
-016
4032
8550
-3735
-521
1035
.276
-340
.959
-375
.570
.505
RNN (ANN)
92-1
.114
.997
2000
577
1403
1935
435
5.89
1700
7750
316
SpacicNNN wibtE L3
RSEJ
.886
.413
.122
.948
-796
1355
.355
355
.587
873
-526
713
-888
1885
1684
185
221
183
-425
1835
-512
.335
585
711
.977
-257
.073
-236
-346
902
-880
.511
.500
7706
SpieRRN PE
2014
2031
-396
.974
-763
-888
.667
-034
-333
811
-734
.977
.865
355
-773
Tranforme (ANN)
.277
21%
.115
.254
-75
7.41
1.88
0723
[51
723
715
375
017
375
7661
7702
SpikSomer Rob PI)
RSEE
.351
.255
-528
-327
3521
.843
152
-019
1445
0922
.374
-375
-356
.450
-350
Apiklomerst of RPE 4
713
315
327
-050
.1%
-07
900
714
51.
10h
130)
100
191
177
477
1715
1211
519
011
545
3310
375
3115
1155
-376
1311
773
Spi8komeer
/330
226
.00
101
029
Spidomer CPG-TE
RSR
.553
-726
7701
.805
.890
-267
7780
508
100
.880
.002
1229
1995
-257
.347
-566
-757
535
707
.290
.3772
3770
-300
3866
311
.355
980
.519
7781
Spiksmmr CMG
RSB,
871
7710
350
-739
.680
.177
-07
.55
255
-907
7700
-377
7714
1880
-45
.820
868
1036
.117
J816
757
585
5548
770
971
334
.011
.511
.968
.062
430
.744
-345
deliberately conducted experiments on PE in Spikformer to explore whether our specially designed CPG-PE is truly more suitable for SNNs than all previous PÊs. Notably, we also investigated the modularization of CPG, i.e., replacing all Linear layers with CPG-Linear layers (See Appendix D). and its impact on the Spikformer model for time-series forecasting, i.c., Spikformer w/ CPG-Full. We time-series forecasting benchmarks with various prediction lengths in Table [
report the results We also list results from ANNs for reference.
In summary, the results presented in Table I indicate that SNNs equipped with the CPG-PE module significantly outperform their counterparts lacking the PE feature. This finding effectively addresses RQ1 from time- sene-series analysis perspective. Detailed findings include:
(1) CPG-PE enables SNNs to successfully capture positional information. SNNs, including models such as Spike-TCN, Spike-RNN, and Spikformer, when integrated with CPG-PE, show superior performance compared to those without PE. Notably, CPG-PE also reduces the performance disparity between SNNs and traditional ANNs in time-series forecasting tasks, evidenced by an average increase of 0.013 in R² and a decrease of 0.022 in RSE
(2) CPG-PE is the most suitable position encoding strategy for Spikformer. In addition to CPG-PE, other encoding strategies such as Float-PE (the original PE in Transformer) and RPE (the original PE in Spikformer) were also evaluated. The Spikformer equipped with CPG-PE emerged as the top-performing variant, confirming CPG-PE as the most suitable PE strategy for SNNs.
(3) CPG-Full module can also effectively model the positional information time series data. The CPG-Full module's performance in modeling positional information of time-series data is comparable to that of CPG-PE, with average R² values nearly identical to those of Spikformer with CPG-PE and significantly better than those of other models.
4.3 Text Classification
In addition to time-series forecasting, natural language processing (NLP) serves as another critical domain to assess the efficacy of the CPG-PE module in encoding positional information. Following the pioneering work of (29], who first employed Spikformer for text classification tasks, we extended this application to 6 benchmark datasets. We also include results from fine-tuned BERT for reference.
Table 2: Accuracy on 6 text classification benchmarks. The best results of SNNs and ANNs are formatted in bold font format. Experimental results are averaged across 5 random seeds.
Model
SNN
Spikke PE
Puram (M)
MIR
S51:2
English Datasee
Subj
S513
Canscaal
Chinese Dataset
Waimal
Avg.
Fine sundd BERT 19
x
109.
87.63.0.15
92.3148.77
95, 95.90.a.0
50.41
89 89.48...15
10.27a0
$4.33
Spikformer who PE 20
100.3
S1.71
91 91.0lan30
21.54
80.87an34
17.25
Spikformer w/ Rundom-PP PC
110.4
75.00
81.64s5.m
91-40aa3
41.88.2441
85 63.60
86 13.00am
17 77.23
Spikformer Ploat PL
109.
706674036
82.18.0.34
92.211033
42.584.41
85.7Lan.5
88.34a631
Spikformer CPG-PF [Durs]
110.4
K2.42an22
82.90.5035
92.50
43.02:634
80.54
88.49
79.41
7

---

The results presented in Table 2 shows that Spikformer enhanced with CPG-PE achieves the state-of- the-ar art performance across 6 benchmarks, effectively addressing RQL. Meanwhile, we conducted set of ablation experiments to eliminate the effects of increased parameter counts on model performance. Specifically, we replaced the spike-form positional encoding matrix obtained from CPG with randomly generated spike matrix (See "Spikformer w/ Random PE" Row). By comparing these two configurations, we confirmed the effectiveness of our proposed CPG-PE.
4.4 Image Classification
In this section, we aim answer RQ2. to conceptualize the array of image patches as sequential data. Consequently, some SNN models that do not incorporate concept of "sequence length" in their spike matrices, such SEW-Resnet (2), are incompatible with the integration of a CPG-PE module. Therefore, we only consider ViT-liked SNN, i.e. Spikformer, in this experiment. We also include results from ViTs for reference.
To adapt the CPG-PE for image classification, it is essential
Table 3: Evaluation on image classification benchmarks. Float-PE denotes the original PE of the Transformer, while RPE denotes the original PE of the Spikformer. Numbers with denote our implementation. The best results of SNNs and ANNs are formatted in bold font format. All results averaged across 4 random seeds.
Model
SNN
Spike PE
Param (M)
Accurccy
Param (M)
CIFARIO-DYS
Accuracy
Param (M)
CIFARI0O
Accuracy
Avg.
Visson- Trunsforner 23
9.32
990.73
9:30
51.02
Spikforner w/o
913.11
Spikformer w/ Random-PP
/
8.17
933.55
2.06
76.44
8.20
73.54
81.27
Spikfomerr Float-PE
x 8.00
94.-12
1.89
77.60
8.01
74.78
882225
Spikformer RPI:
94.64
83.12
Spikfomner CPG-PE [Ours]
8.17
94.82
2.06
78.06
8:20
77.27
83.38
Spikformer w/ CPG- PE [Equal Parum]
7.99
94.60
1.90
78.00
8.02
76.91
83.17
We report the parameter counts and classification accuracy in Table 5 To elaborate, Spikformer with CPG-PE outperforms other variants, demonstrating the effectiveness of CPG-PE even when the sequence is an array of image patches lacking inherent order. Notably, owing to our streamlined implementation, the parameter count for Spikformer with CPG-PE is significantly reduced compared to the original Spikformer w/ RPE with a reduction of 1.16 M. What's more, we conducted ablation experiments on model parameters by reducing the parameter count of Spikformer with CPG-PE to be comparable Spikformer w/o PE, allowing for a more direct performance comparison, as shown in the last line in Table 5]] The results on ImageNet are reported in Appendix
However, is essential acknowledge that the improvements in image classification are relatively modest compared to those observed in time series and text applications. This phenomenon can largely be attributed to the intrinsic non-ordered nature of image patches. Unlike text or time series data, where sequential order is crucial and inherently informative, image patches do not possess a natural or fixed sequence. This lack of order means that traditional methods of positional encoding, which significantly benefit ordered data by providing contextual positioning, are less effective. Thus, the application of our positional encoding techniques, optimized for data with inherent sequential order, does not translate as effectively to the domain of image classification.
4.5 Sweeping CPG properties
In this section, we investigate the influence of CPG properties on the ability to model positional information, addressing RQ3. To this end, we evaluated the Spikformer model with CPG-PE by varying the base period T and the number of CPG pairs N (see Equations [IT and 12 in time-scries forecasting and image classification tasks.
From Figure 4 (a) and (b), we observe that CPG-PE is insensitive the base period (in biological neurons, is affected by the physiological properties of the CPG circuit such as RC constant and synaptic delay). The sequence lengths (T x L) of the time series and image patches are no larger than 672 (4 x 168) for all benchmarks, preventing repetitions in CPGs. Therefore, when N = 20, sweeping T ( (100, 1000, 5000, 10000] makes minor influence on performance. Furthermore, Figure 4 (c) and (d) demonstrate that when 7 10000, increasing the number of CPG pairs N enhances Spikformer's performance. This is reasonable because more CPG neurons reduce repetitions in positional representations of X'.
8

---

03
442
7V9
CIMPOONT
(a)
(d)
Figure 4: (a)c) R versus and N time-series forecasting tasks. (b)(d) Accuracy versus T and N on image classification tasks. T {100, 1000, 5000, 10000], N E (5, 10, 203.
4.6 Positional Encoding Analysis
In this section, we want to address RQ4. As mentioned in Section 2.2. an ideal PE method for SNNs should include the following characteristies: (1) Uniqueness of each position; (2) Compatibility with neuromorphic hardware; (3) Formulation in spike-form. Our implementations ensure compatibility with neuromorphic hardware (2), and the CPG-PE is inherently formulated in spike- form, satisfying (3). Therefore, in order to assess (1), the uniqueness of each position, we would like to compare the CNN-based RPE in [4] (5) and our proposed CPG-PE, focusing specifically on their capacity to provide distinct positional signals. This analysis was conducted using the CIFARIO-DVS dataset, where we calculated the repetition rate of spike positional representations across all positions. Our findings were notable: the positional spike matrices produced by RPE exhibited a repetition rate as high as 12.19%, which significantly undermines its effectiveness for PE. In contrast, our proposed CPG-PE exhibited no repetition, demonstrating that our CPG-PE is well-suited for serving as the PE module in SNNs. Please refer to Appendix [B for details.
Related Work
5.1
Spike Encoding Methods
Spiking neural networks employ several coding methods to encode input information, each offering unique advantages. Direct coding [5]40] the simplest form and widely-used in image tasks, directly associates spikes with specific values or events, providing straightforward and interpretable outputs but often lacking efficiency for complex tasks. Rate coding [8 41), where the input is represented by the frequency of spikes within a given timeframe, is more robust and widely used but can be less precise due to its reliance averaged spike rates. Temporal coding (a.k.a latency coding) 442 43
encodes information based on the timing of individual spikes, allowing for high temporal precision and efficient representation of dynamic inputs, though it can be computationally demanding. In addition, delta coding 44 represents changes in input signals through spikes, focusing on differences rather than absolute values, which can enhance efficiency and response times but may introduce complexity in decoding. Each of these methods contributes to the versatility and applicability of SNNs in various domains, from neuroscience artificial intelligence. The SNNs we considered in this paper should fall into the category of rate coding since back-prop is conducted on spike rate. Meanwhile, CPG-PE can be considered converting temporal information into spike rate of a group of neurons (Equations I1 and 12). and this is why CPG-PE can improve performance for sequential data. is possible to introducing learning algorithms of temporal coding for the CPG neurons tackle more complex sequence structure, which remains as future work.
5.2 Positional Encoding in SNNs
Currently, few works have demonstrated the importance of PE approaches in SNNs. Spikformer 4 and Spike-driven Transformer (5] utilize a combination of "one convolutional layer + one batch normalization layer + one spiking neuron layer generate learnable "relative positional encoding". From our perspective, this strategy is more like saike-eleeent-wiiie residual connection (2), rather than classic module. The unique representation of each position a fundamental requirement for robust PE module. However, the spike position matrix generated by their method may result in the same spike representation for different positions. Additionally, the addition of the input spike matrix and the position spike matrix will result in the occurrence of non- non-binary numbers (i.c., 2) due

---

proposed learnable positional graph spikes, aiming to capture neighbor information within graphs rather than sequences. Therefore, drawing inspiration from the periodic automatic spike generation pattern of CPGs, we propose biologically plausible and effective spike-form absolute position encoding method called CPG-PE.
to the addition of 1 and 1. For spiking graph neural networks, 45
6
Rethinking the Role of CPGs in Neuroscience
Our study also provides novel insights into neuroscience understanding the role of CPG in nervous systems. While traditionally CPG is believed to play crucial role producing the rhythmic motor patterns necessary for locomotion and other repetitive movements [18 20], the analogy to PE in this work reveals that CPG can make significant contribution in processing sequential data by encoding the positional information into unique spiking patterns at different times. This does not only work for time-series sensory input like auditory signals but also for visual sensory data: e.g. when a person looks an image, saccades (eye movements) allow retinal neurons receive different parts of the image at different times. This indicates that CPG neurons could potentially be utilized to encode positional information. Another extensive thought is that as PE can be learnable in ANNs, CPG may also benefit from adaptability to the data [46]. The hypothesis, however, remains to be examined through neuroscientific experiments 47
7 Conclusion
In conclusion, inspired by central pattern generators, we introduce pioneering position encoding approach termed CPG-PE, specifically tailored to mitigate the constraints associated with current PE techniques within SNNs. We mathematically prove that abstract PE in the Transformer is a particular solution of the membrane potential variations in specific type of CPG. Furthermore, through comprehensive empirical investigations across diverse domains including time-series forecasting, natural language processing, and image classification, we demonstrate that the CPG-PE satisfies all the requirements of PE tailored for SNNs. The limitations and future work are discussed in Appendix
References
[1] Wofgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural Networks, 14:1659-1671, 1997.
[2] Wei Fang, Zhaofei Yu, Yanging Chen, Tiejun Huang. Timothée Masquelier, and Yonghong Tian. Deep residual learning spiking neural networks. In Neural Information Processing Systems, 2021.
[3] Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ANN-SNN conversion for fast and accurate inference deep spiking neural networks. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI 2021, pages 2936, 2021.
[4] Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
[5] Man Yao, JiaKui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, XU Bo, and Guoqi Li. Spike- driven transformer. In seirth-seventh Conference on Neural Information Processing Systems, 2023.
[6] Man Yao, JiaKui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, XU, and Guoqi Li. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of gexe-generttion neuromorphic chips. In The Twelfth International Conference on Learning Representations, 2024.
10

---

[7] Wei Fang, Zhaofei Yu, Yanqing Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant enhance learning of spiking neural networks. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2641-2633 2020.
[8] Changze Lv, Jianhan Xu, and Xiaoqing Zheng. Spiking convolutional neural networks for text classification. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
[9] Yuhang Li, Tamar Geller, Youngeun Kim, and Priyadarshini Panda. Seenn: Towards temporal spiking early exit neural networks. Advances in Neural Information Processing Systems, 36, 2024.
[10] Yujie Wu, Lei Deng. Guoqi Li, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:323875, 2018.
[11] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with firectly-trainet larger spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11062-1 11700 2021.
[12] Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. Advances in Neural Information Processing Systems, 35:34377-34390 2022.
[13] Chenlin Zhou, Liutao Yu. Zhaokun Zhou, Han Zhang, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual learning for transformer-bass spiking neural network. arXiv preprint arXhv:2304.11944 2023.
[14] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems,
33:1474-1487, 2020.
[15] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/1706.03762, 2017.
[16] Anthony M Zador. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature communications, 10(1):3770, 2019.
[17] Melanie Mitchell. Why ai is harder than we think. In Proceedings of the Genetic and Evolution- ary Computation Conference, pages 3-3, 2021.
[18] Eve Marder and Dirk Bucher. Central pattern generators and the control of rhythmic movements. Current biology, (1(23) 2001.
[19] Eve Marder and Ronald L Calabrese. Principles of rhythmic motor pattern generation. Physio- logical reviews, 76(3):687-717, 1996.
[20] Sten Grillner. Biological pattern generation: the cellular and computational logic of networks in motion. Neuron, 52(5):751-766 2006.
[21] Ole Kiehn. Decoding the organization of spinal circuits that control locomotion. Nature Reviews Neuroscience, 17(4):224-238 2016.
[22] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transfor- mation for multi-view 3d object detection. In European Conference on Computer Vision, pages 531-548. Springer, 2022.
[23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[24] Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Yonghong Tian, and other contributors. Spikingjelly, 2020.

---

[25] Paul J. Werbos. Backpropagation through time: What it does and how to do it. Proc. IEEE, 78:1550-1560, 1990.
[26] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position rep- resentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468 Association for Computational I Linguistics, 2018.
[27] Vighnesh Shiv and Chris Quirk. Novel positional encodings to enable tree-based transformers. Advances in neural information processing systems, 32, 2019.
[28] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
[29] Changze Lv, Tianlong Li, Jianhan Xu, Chenxi Gu, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Spikebert: A language spikformer learned from bert with knowledge distillation. 2023.
[30] Malyaban Bal and Abhronil Sengupta. Spikingbert:
Distilling bert to train spiking language models using implicit differentiation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10998-1 11066, 2024.
[31] Rui-Jie Zhu, Qihang Zhao, and Jason K Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint arXiv: 13939, 2023.
[32] Changze Lv, Yansen Wang. Dongqi Han, Xiaoqing Zheng, Xuanjing Huang, and Dongsheng Li. Efficient and effective time-series forecasting with spiking neural networks. In Forty-first International Conference Machine Learning (ICML), 2024.
[33] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. arXiv preprint arXiv: 2017.
[34] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 95-104, 2018.
[35] Bo Pang and Lillian Lee. Sceing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, 2005.
[36] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
[37] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs: An event- stream dataset for object classification. Frontiers in Neuroscience, 11, 2017.
[38] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: vrrXiv/1003.000 1803.0121,, 2018.
[39] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
[40] Zhaokum Zhou, Kaiwei Che, Wei Fang, Keyu Tian, Yuesheng Zhu, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer v2: Join the high accuracy club imagenet with snn ticket. arXiv preprint arXiv:2401.02022 2024.
[41] Youngeun Kim, Hyoungseob Park, Abhishek Moitra, Abhiroop Bhattacharjee, Yeshwanth Venkatesha, and Priyadarshini Panda. Rate coding or direct coding: Which one is better for accurate, robust, and energy -fffgy-efficient spiking neural networks? In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 71 71-75. IEEE, 2022.
12

---

[42] Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. European conference on computer vision, pages 388-404 Springer, 2020.
[43] Comsa, K Potempa, Versari, T Fischbacher, Gesmundo, and Alakuijala Temporal coding in spiking neural networks with alpha synaptic function: Learning with backpropagation. IEEE Transactions on Neural Networks and Learning Systems, 33(10):5939- 5952, 2022.
[44] Young C Yoon. Lif and simplified srm neurons encode signals into spikes via a form of asynchronous pulse sigma-delta delta modulation. IEEE transactions neural networks and learning systems, 28(5):1122-12) 2016.
[45] Han Zhao, Xu Yang, Cheng Deng, and Junchi Yan. Dynamic reactive spiking graph neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16970-16978, 2024.
[46] Rafael Yuste, Jason N Macleen, Jeffrey Smith, and Anders Lansner. The cortex as a central pattern generator. Nature Reviews Neuroscience, 6(6):477- 483, 2005.
[47] Adam Marblestone, Greg Wayne, and Konrad Kording. Toward an integration of deep learning and neuroscience. Frontiers in computational neuroscience, 10:94, 2016.
[48] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[49] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
13

---

Acknowledgments
The authors would like thank the anonymous reviewers for their valuable comments. This work was supported partially by National Natural Science Foundation of China (No. 62076068).
Broader Impact
This work aims to advance the field of spiking neural networks (SNNs). Unlike artificial neural networks (ANNs) which have been applied widely in people's lives, SNNs are still undergoing fundamental research. We do not see any negative societal impacts of this work.
Reproducibility Statement
The authors have diligently worked to ensure the reproducibility of the empirical results presented in this paper. The datasets, experimental setups, evaluation metrics, and hyperparameters are thoroughly described in Appendices [A] and B Furthermore, the source code for the proposed PE method has been available at hhhhtss:: wtwp....oommccrooooo////eo//o
A
Datasets
A.1 Time-series Forecasting
Detailed statistical characteristics and distribution ratios for each dataset are provided in the following:
Table 4:
The statistics of time-series datasets.
Dataset
Samples
Variables
Observation Length
Train- Valid Test Ratio
Metr-la
34, 272
207
12, (short-term)
(0.7,0.2, 0.1) (0.7,0.2,0.1
Pems-bay
52, 126
325
12, (short-term)
enerry
52,560
137
168, (long-term)
(0.6, 0.2,
Electricity
26,304
321
168, (long-term)
(0.6,0.2,00 0.2)
A.2 Text Classification
Here are the datasets we used in text classification experiments:
MR: MR, which stands for Movie Review, is a dataset containing movie-review documents
labeled based on their overall sentiment polarity (positive or negative) or subjective rating [35].
SST-5:
SST-5 includes 11, 855 sentences from movie reviews for sentiment classification
across 5 categories: very negative, negative, neutral, positive, and very positive (36).
SST-2: SST-2 the binary version of SST-5, containing only classes: positive and
negative.
Subj: The Subj dataset is designed to classify sentences as either subjective or objective"
ChnSenti: ChnSenti consists of approximately 7, 000 Chinese hotel reviews, each annotated with a positive or negative labe
Waimai: This dataset contains around 12, 000 Chinese reviews from food delivery platform, intended for binary sentiment classification (positive and negative [
A.3 Image Classification
Here are the datasets we used in image classification experiments: CIFAR dataset comprises collection of 60, 000 images, partitioned into 50, 000 training and 10, 000 testing images, each with
https: htwttps:/www....
https://ram, http:///e.........cceeeeeeeeeear
ChnSentiCarp CarptttiCrrphhhaa__e_ hal_app_ht__all.eee_e csv
14

---

resolution of 32 x 32 pixels. The CIFARIO-DVS dataset represents neuromorphic adaptation of this original set, where static images have been transformed accommodate the recording capabilities of Dynamic Vision Sensor (DVS) camera. This conversion results in dataset consisting of 9, 000 training samples and 1, 000 test samples with 128 128 resolution.
B Experiment Settings
B.1 Time-series Forecasting
Metrices The metrics we used in time-series forecasting are the coefficient of determination (R²) and the Root Relative Squared Error (RSE).
C
(Y
pm)z Y,,1
n²= MCL
ΣΣΣ
o.1
(16)
(Y
1=1
Y
Yml!
RSE
(17)
In these formulas, M symbolizes the size of the test sets, C denotes the number of channels, and signifies the length of predictions. Y represents the average of Y The term Y refers to the I-th future value of the e-th variable for the m-th sample, while Yo, indicates the mean of Y across all samples. The symbols Y and Ym are used to represent the ground truth values. Compared to Mean Squared Error (MSE) or Mean Absolute Error (MAE), these metrics offer greater resilience against the absolute values of the datasets, making them particularly useful in the time-series forecasting setting.
Model Architecture All SNNs take 4 time steps. For SpikeTCNS and SpikeRNNs, follow the same settings as [32]. We construct all Spikformer as 2 blocks, setting the feature dimension as 256, and the hidden feature dimension in FFN as 1024. For CPG-PE settings, set 10000.0, N 20, 1, and y thres = 0.8.
Training
set the training batch size 64 and adopt Adam 48 optimizer
with a cosine scheduler of learning rate 1 x 10 An early stopping strategy with a tolerance of 30 epochs is adopted. We conducted time-series forecasting experiments 24G- V 100 GPUs. On average, a single experiment takes about 1 hour under the settings above.
B.2 Text Classification
Model Achirecture All models are with 12 encoder blocks and 768 feature embedding dimension. important to note that the original implementation of [29] incorporates a layer normalization module that poses challenges to hardware compatibility. To address this, we have substituted layer normalization with batch normalization in our directly-trained Spikformer models for text classification tasks. For CPG-PE settings, we set 10000.0. N = 20, r 1, and thhres = 0.8.
We directly trained Spikformers with arctangent surrogate gradients on all datasets. We use the BERT-Tokenizer in Huggingfac to tokenize the sentences to token sequences. We pad all samples to the same sequence length of 256. We conducted text classification experiments on 4 RTX-3090 GPUs, and set the batch size as 32, optimizer as AdamW [49 with weight decay of 5 10 10-3 and set cosine scheduler of starting learning rate of 5 x 10-4 What's more, in order to speed up the training stage, we adopt the automatic mixed precision training strategy. On average, single experiment takes about 1.5 hours under the settings above.
Training
B.3 Image Classification
Model Architecture For all Spikformer models, we standardized the configuration to include 4 time steps. Specifically, for the CIFAR10 and CIFAR100 datasets, the models were uniformized
https://huggiingfacc.ooo
15

---

with 4 encoder blocks and a feature embedding dimension of 384. For the CIFARIO-DVS dataset, the models were adjusted to have 2 encoder blocks and a feature embedding dimension of 256. For CPG-PE settings, we set 10000.0, N 20, 11 = 2Tr, and ehere = 0.8.
We honestly follow the experimental settings in [4] whose source code and configuration files are available at https:: thub. com/ ZK Zhou/ thhou/spiformet As the training epochs are quite big (300 epochs) in their settings, we choose to use one 80G-A100 GPU, and it takes about 3 hours to conduct single experiment, on average.
Training Hyper-parameed
B.4 Details about Positional Encoding Analysis
We conducted positional encoding analysis experiments on the CIFARIO-DVS dataset. For the original Spikformer with relative positional encoding (RPE) as described by [4, the input and output channels of Conv2d are both set to 384. In our Spikformer with CPG-PE, the parameters are set 20. Given the time step T = 4 and the sequence length L = 160 for
to r = 10000.0 and the image patches in CIFARIO 10-DVS samples, the total "length" T x L in CPG-PE is 640. We then calculated the repetition rate of positions. The results showed that the repetition rate for RPE is 12. 12.19%, whereas for CPG-PE, it is 0.00%.
C Implement CPG-PE with LIF Neurons
In this section, we demonstrate that CPG-PE a hardware-friendly design. While implementing the sinusoidal potential on the neuromorphic chips is not challenging (e.g.. by maintaining additional LC circuits), we show how a CPG-PE neuron can be physically implemented with only 2 LIF neurons defined by Equations I to (3 and thus introducing
extra efforts on chip designs.
CPG-PE neuron, after discretization, can be viewed as an autonomic neuron that will emit burst of K spikes after resting for R time steps. The key idea is to set two LIF neurons, namely the Emitter and the Resetter. The emitter will draw constant current from the source, and as soon as its membrane potential reaches the threshold after R time steps, it will start emitting spikes constantly until receiving the reset signal from the resetter. The resetter, which will remain the resting potential until it receives signals from the emitter, will count the number of spikes and emit reset signal (inhibition signal) to the emitter after receiving K spikes.
We first prove the following Lemma, which establishes the relationship between the start time of the first spike and a constant input current.
Lemma 1. Given an LIF neuron defined by Equations ① to (3) with decay rate 3 and threshold
Uter starting with resting potential (O) if fed with the constant current I(t) Ie > 0, the first spike will emit at:
3the - 3) )
Tmin- = Lo (B=
(18)
Le
Proof. By definition, before the time to emit the first spike, we have S(t) = 0. Thus Equation can be rewrite as:
U(kit) BU - 1)A() + Te-
(19)
Simplifying the recurrence relation, we can obtain:
To / (1 3h
U(kAt) 8
(20)
3
The first spike is generated when Ul:at) S Ulrr, thus we have:
3thr 3)
k 2 loge(B
(21)
Ta
that is to say,
3ehri31 - (3)
Tmán lo
(22)
1e
16

---

R. Now we can implement the CPG-PE with LIF neurons:
Theorem 1. Given LIF neurons, the emitter and the resetter, with decay rate 3, threshold Usa and reset potential V resets starting with resting potential (,(0) (0) 0.
(t) = Le1 + S. (U Le1 Vrrsett S, 1t)
(23)
(t) S,(t Lt S,(t At)(Ece + Vreset),
(24)
Ulh 3)
L1 3 - 3
(25)
Te = U
(26)
B -33K-1
then the system will have the period of = (R+ K)A,, and Vi E R+ K 11, N:
[0, 0sich R, 1, R
Sa(il + KT)
(27)
Proof. Assuming the first spike generated by the emitter emits at time step Ti. For every 0 2 <T we have:
Se(t) (t) = 0, T2(()= = Tai, TT(()= = 0.
(28)
(29)
Since the input current of the emitter constant, by Lemmal we immediately get:
logs( 31h B
Ti
(30)
Ie1
Starting from time step R, let's assume the first spike generated by the resetter emits at time step Tz. Then for every S < T2, we have:
(t) 1, (t) O,
(31)
(t) = Upr Vreset Ix(t)= = Lc2.
(32)
Starting from Ti, for the emitter, the input current allows spike event for every time step. And the input current of the resetter is a constant. Again, by applying Lemma
I we can get:
To = Ti + log,(8 Uth - 3)
R 1-
(33)
102
Now Consider the state time step + K:
Sc((R K 1)At) S K 1)At) = 1,
(34)
Ia((+ + K)A() (((R+K)AI) = -Veesets U U.((R+ K)At) = 0.
(35)
(36)
This is the same as the membrane potential at time step 0. Therefore, the system will behave periodically with period T (R K)AT.
Theorem [ gives possible CPG-PE design with 2 LIF neurons, with the emitter generating K consecutive spikes every R+ K time steps. This demonstrates that incorporating CPG-PE into the current SNN architecture is completely bio-plausible and will not introduce any burden of redesigning hardware.
D Implement CPG-Linear
We have developed simple modularization implementation to integrate our proposed CPG-PE with original linear layers, as depicted in Figure 3 (b). Consider the original linear layer's input and output dimensions as D.n. and Dout, respectively. Our objective is to incorporate CPG-PE within this framework. Following the application of the CPG-PE module, the modified input X' is obtained. X is then input into Linear; and X' into Linearg, resulting in outputs X1 and X2, respectively. Both
17

---

Integrate CPG-PE into Linear Layers
Batch Norm
Batch Norm
Spiking Neuron Layer
Linear
CPG-Linear
Concatenation
Element-wise Addition
Extra Linear
Original Linear
Original Linear
X Input Spike Matrix
X"
X' Positional Embedding
CPG-PE
T Time Steps in SNNs
Sequence Length
X
X
(a) Original Linear
(b) Modularization as CPG-Linear
Figure 5: An illustration of the implementation of integrating CPG-PE into linear layer.
Linear, and Linear2 maintain output dimension of Dout- The final step involves summing X1 and X₂ to produce X3, which is subsequently processed through batch normalization (BN) and spike normalization (SN). We term this implementation as "CPG-Linear" and formulize follows:
CPG-PE(X),
X {0, (0,1)XXBXXXD, X'' E TxBxXx2N
(37)
X1 = Linear, (X), X2 Linear2 (X'),
X,, X2 E TTXXXLXD
(38)
X3 = X1 + X2,
X3 E RxXXLLDD..
(39)
X outpuut SN (BN (Xs)),
Xout E (0, 10 TxBxLxXDD
(40)
where denotes element-wise addition. This implementation described above fundamentally identical to Figure 3 within the context of single linear layer. However, the CPG-Linear can seamlessly replace any linear layer in SNNs.
E Results on ImageNet
We have conducted experiments with Spikformer without positional encoding (PE), Spikformer with relative positional encoding (RPE), and Spikformer with our proposed CPG-PE on the ImageNet dataset. The results are as follows:
Table 5: Evaluation on ImageNet benchmarks. We employed encoder blocks and 384 feature
embedding dimensions across all models.
ImageNet
Model
SNN
Spike PE
Param (M)
Accuracy
Spikformer w/o
15.50
Spiktormer RPF
16.81
70.24
Spikformer CPG-PE [Ours]
15.66
71.17
Specifically, we set the depth to 8 and the dimension of representation to 384, From the table, we that CPG-PE performs well large-scale image datasets. We believe that the above results demonstrate the effectiveness our proposed CPG-PE positional encoding.
F Limitations and Future Works
In this section, we will discuss the limitations and future works of our paper.
18

---

E1 Limitations
As mentioned in Section our CPG-PE can not be directly applied to those SNNs where spike matrices do not have a "equeenee length" dimension. Our CPG-PE is optimized for processing sequential data, making it ideal for upplications involving time series or natural language. This intrinsic design, however, does not naturally extend to image data, which typically benefits from direct convolutional operations that capture spatial relationships across the entire image dimensions- and width. In contrast, CPG-PE requires the segmentation of images into patches, method inspired by the Vision Transformer. This adaptation contrasts with approaches like the Convolutional 2D) layer, which applies convolution operations directly across the height and width of an image without requiring segmentation into smaller, discrete patches. The necessity to adapt CPG-PE for image data through patching can introduce complexities and potential performance bottlenecks, as it may not effectively capture the continuous spatial relationships and local features the image, which are crucial for tasks such as object recognition and scene understanding.
F.2
Future Works
To enhance the applicability of the CPG-PE model broader range of data types, especially image data, future research could focus on developing hybrid model that integrates the strengths of CPG-PE with traditional convolutional layers. This integration could potentially allow the model to handle both sequential and spatial data efficiently without the need for extensive pre-processing or adaptation. Specifically, integrating direct convolution operations that work across the entire spatial dimensions of image within the CPG-PE architecture could help preserve spatial relationships and improve feature extraction capabilities. Additionally, exploring the use of adaptive patch sizes dynamically adjusting the patching mechanism based the nature of the input data could also provide a more flexible and cectoorannc--ppimnizee approach. These advancements would make the model more versatile and capable of tackling wider array of tasks across different domains.
Additionally, considering that CPG-PE is an absolute positional encoding designed for SNNs, it could be beneficial to explore the potential of implementing learnable relative positional encodings in SNNs. Such encodings would need to be developed to meet specific criteria: they must maintain the spike- form characteristic essential to SNNs and ensure the uniqueness of each position's 's encoding. This approach could significantly enhance the model's 's ability to capture and utilize the temporal dynamics of input data more effectively, potentially leading to more nuanced and context-aware processing capabilities. Exploring adaptive patch sizes or dynamically adjusting the patching mechanism based on the nature of the input data could also provide more flexible and performance-optinize approach. These advancements would make the model more versatile and capable of tackling a wider array of tasks across different domains.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have clarified our claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations and future work in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
We have provided the full set of assumptions and a complete (and correct)
proof in the Method Section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have shown our experiment results in the Experiment Section. We have submitted our source code in Supplementary Material. We will upload our code and data to Github upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have submitted source code in Supplementary Material. We will upload our code and data to Github upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cc/
wublcc//uddes/ddddsiiiiiisoooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht
/ //nips. cttps.cccpbllll wtlllccgguiiio de///odeSummmm/mi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have shown our experimental settings and implementation details in Appendices A and BndBBun respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our reported results are all averaged over several random seeds. We have reported the error bar of the results in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the compute resource in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:://wwuurrr..cccc//////eeeee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive societal impacts and negative societal impacts of the work in Broader Impact Section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The datasets used in the paper are all public datasets. Please refer to Appendix for details of datasets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com commdaaasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "CzTKrQIIk7", "reviewer": "Reviewer_P6YU", "summary": "This paper introduces central pattern generators (CPGs) from neuroscience into the SNN framework as a novel method for position encoding. Through mathematical derivation, it is proven that the existing abstract PE methods in transformers are actually a particular solution for a specific type of CPG. The effectiveness of CPG is validated through experiments across several domain benchmarks.", "questions": "Please refer to weakness 3, 4, 5", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThis article connects existing abstract Positional Encoding (PE) methods in transformers with Central Pattern Generators (CPGs) in the human brain through mathematical derivation, showing that the former can be viewed as a specific mathematical solution to the membrane potential dynamics of the latter. This presents an interesting viewpoint.\n\n2.\tThe paper is well-organized and clearly written, offering high readability. Readers can effortlessly grasp the authors' intentions, supported by both the textual explanations and accompanying illustrations.", "weaknesses": "1.\tThe authors have not convinced me why the problem addressed in this paper is very important, i.e., why the existing PE methods in SNNs are such a big problem that they need to be improved by CPG.\n\n\n2.\tThe method proposed in the paper is simple and does not provide enough inspiring insights; the contribution is relatively limited.\n\n3.\tThe implementation of CPG-PE on hardware involves coupled nonlinear oscillators that require frequent updates of neuron membrane potentials, entailing floating-point computations and memory read-write operations, which result in additional energy expenditures. The paper should scrutinize and analyze whether the performance enhancements afforded by this encoding method justify the additional energy costs. This trade-off demands a detailed examination to assess its viability in practical applications.\n\n4.\tIn Sec3.1, why is \"F(x)=b<=0, H(y)=d<=0\" followed after “...gain membrane voltage with constant speed” instead of “F(x)=b>0, H(y)=d>0”?\n\n5.\tIn Sec4.2, does the CPG-Full method replace all linear layers in the model with CPG-Linear layers? Why is its performance not as good as that of CPG-PE? Can you provide further analysis and explanation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and suggestions, which are valuable for enhancing our paper. The follows are responses to your individual concerns:\n\n### **1.Why is positional encoding important to SNNs (W1)?** \nCurrently, SNNs have been applied to a variety of tasks beyond image processing, including time-series forecasting [1] and text classification [2]. However, due to lack of suitable positional encoding (PE) method, it remains challenging for SNNs to capture indexing information, rhythmic patterns, and periodic data. Existing PE methods for SNNs do not meet the essential criteria: uniqueness of each position and formulation in spike-form. Inspired by CPG neurons, we propose CPG-PE, which performs well on both sequential and image tasks. Additionally, we ensure its hardware-friendliness and provide insights into the role of CPGs in neuroscience (Section 6).\n\n### **2.The method is simple and does not provide enough inspiring insights; contribution is relatively limited. (W2).** \n> \"Everything should be made as simple as possible, but no simpler.\" – Albert Einstein\n\n(1) We believe simplicity is a strength of our method (as it is effective), not a weakness. As it is easy to implement, there is a higher probability that more researchers can benefit from leveraging our method on their model to improve performance. Furthermore, since CPG-PE is novel to the SNN community, complex models and algorithms with even better capacity may emerge following our work, but it is necessary that we first comprehensively address that a simple CPG-PE can consistently improve the performance of SNNs. \n\n(2) Our contributions can be summarized as: **(1) Pioneer in the exploration of spike-version PE for SNNs.** To the best of our knowledge, our proposed CPG-PE is the first work on spike-version PE in SNNs, which are both brain-inspired and hardware-friendly. We corrected the inappropriate PE designs previously used in SNNs. **(2) Consistent Performance Gain.** CPG-PE enhances the performance of SNNs across a wide range of tasks, including time-series forecasting, text classification, and image classification. **(3) Insights on neuroscience and neuromorphic hardware.** We mathematically proved that the traditional PE in Transformers is a particular solution of the membrane potential variations in a specific type of CPG, and we also proved that CPG-PE can be implemented with 2 LIF neurons so that it will not introduce any burden of redesigning hardware.\n\n### **3.Frequent updates of neuron membrane potentials bring additional floating-piont operations and energy on hardware (W3).**\nWe fully agree, and indeed, we have discussed this aspect in our paper.\n\n(1) **Physical Implementation of CPG-PE** We have proposed an approach to physically implement CPG-PE with only 2 LIF neurons, introducing no extra efforts on chip designs. \nA CPG-PE neuron can be viewed as an autonomic neuron that will emit a burst of $K$ spikes after resting for $R$ time steps. Through rigorous derivation (see **Appendix C**), we prove that the system can behave periodically with a period $T = (R + K)∆T$, i.e., CPG-PE can be implemented with 2 LIF neurons. This approach eliminates the need for additional floating-point computations and memory read-write operations on neuromorphic hardware to frequently update neuron membrane potentials.\n\n(2) **Support from Existing Circuit Implementations** Existing circuit implementations on hardware [3][4] strongly support our proposed CPG-PE, as these works are based on the similar idea of **coupled LIF neurons**. Circuits based on complementary metal-oxide-semiconductor technology [3] simplify membrane potential calculations, reduce power consumption significantly, and eliminate the need for additional memory to store membrane potentials. Furthermore, memristor-based CPG circuits [4] have been shown to minimize circuit complexity and increase energy efficiency.\nTherefore, we believe that our proposed approach is hardware-friendly, without requiring additional operations on hardware, which has also been acknowledged by other reviewers (3Lmi, s3xa, Pjez).\n\n### **4.Misleading in Section 3.1 (W4).** \nWe apologize for this typo and the resulting confusion. It should be: \"$\\mathbf{F}(\\mathbf{x})=b>0, \\mathbf{H}(\\mathbf{y})=d>0$.\"\nAfter our thorough inspection and verification, we confirm that this does not affect the derivation results, and Equations 6-10 remain valid. We will correct this typo in our revised manuscript.\n\n###  **5.Why is the performance of CPG-Full not as good as that of CPG-PE (W5).**\n\n(1) CPG-Full stands for replacing all linear layers with CPG-Linear layers. CPG-Linear was introduced in Appendix D. The original CPG-PE models the inputs, while CPG-Full models the hidden states.\n\n(2) The difference between the two is not significant. In terms of metrics, the average $R^2$ is identical, and the $RSE$ difference is not statistically significant (p-value=$0.9351$, student *t*-test).\n\n(3) We conducted this experiment for two main reasons: Firstly, to explore how to implement modular CPG, hypothesizing that simply replacing Linear with CPG-Linear would allow the PE functionality to work effectively. Secondly, this experiment is supplementary, aiming to confirm that the performance improvement of SNNs is due to the functionality of PE rather than an enhancement in representation capability, as CPG-Full does not affect representation capability.\n\nPlease kindly let us know if there is any remaining concern. We are committed to addressing your concerns.\n\n[1] Lv C, et al. Efficient and Effective Time-Series Forecasting with Spiking Neural Networks. ICML, 2024.\n\n[2] Bal M, et al. Spikingbert: Distilling bert to train spiking language models using implicit differentiation. AAAI, 2024.\n\n[3] Vogelstein J, et al. Dynamic control of the central pattern generator for locomotion. Biological Cybernetics, 2006.\n\n[4] Dutta S, et al. Programmable coupled oscillators for synchronized locomotion. Nature Communications, 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YVps8wERUI", "reviewer": "Reviewer_3Lmi", "summary": "The lack of an effective and hardware-efficient spike-frm position encoding strategy in  SNNs has been a consistent motivation for this study.\nDrawing inspiration from the central pattern generators (CPGs) in the human brain, which produce rhythmic patterned outputs without requiring rhythmic inputs, this work proposes a novel PE technique for SNNs, termed CPG-PE. \nExtensive experiments across various domains show the superior performance with CPG-PE.", "questions": "1.In ANNs, positional encoding is typically added to features. Please analyze the similarities and differences of this category aggregation-based positional encoding compared to ANNs.", "limitations": "I believe the authors' discussion on limitations is comprehensive.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.To the best of my knowledge, this is the first work on position encoding in SNNs, laying the foundation for efficient sequence modeling in SNNs.\n\n2.The approach utilizes the coupling of multi-neuron pulse signals as position encoding, which is innovative.\n\n3.The authors demonstrated the effectiveness of this method across various tasks.\n\n4.The spike-position encoding generating method through mutual inhibition between two groups of spiking neurons is brain-inspired and hardware-friendly.", "weaknesses": "1.This type of positional encoding, through aggregation, introduces a small number of additional parameters and computational overhead. Please provide ablation experiments demonstrating that the performance improvement is not solely due to these factors.\n2.I suggest the authors include results on ImageNet to demonstrate the effectiveness on large-scale datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions, which are valuable for enhancing our paper. We are pleased that our contributions are well recognized.\n\nResponses to your concerns and questions are hereby presented:\n\n###  **1.Ablation experiments on parameters (W1).** \n\nThanks for your suggestion.\n\n(1) Firstly, while our method increases the parameter count compared to the version without positional encoding (w/o PE), this increase is relatively small. For example, on CIFAR10, the paramter numbers of Spikformer with CPG-PE are only $0.17$M(~$2.17$% of the total parameters) larger than that of Spikformer w/o PE.\n\n(2) Secondly, we conducted ablation experiments by reducing the parameter count of Spikformer with CPG-PE to be comparable to Spikformer w/o PE, allowing for a more direct performance comparison. The results are as follows:：\n\n| Model                               | CIFAR10                        | CIFAR10-DVS                    | CIFAR100                       |\n| ----------------------------------- | ------------------------------ | ------------------------------ | ------------------------------ |\n| Spikformer w/o PE                   | Param: $8.00$M, Accuracy: $93.77$% | Param: $1.99$M, Accuracy: $76.40$% | Param: $8.04$M, Accuracy: $73.59$% |\n| Spikformer w/ CPG-PE \\[Equal Param\\] | Param: $7.99$M, Accuracy: $94.60$%     | Param: $1.99$M, Accuracy: $78.00$% | Param: $8.02$M, Accuracy: $76.91$%     |\n\nSpecifically, we adjusted the representation dimension of Spikformer with CPG-PE to make the parameter counts similar to those of Spikformer without PE. For instance, on CIFAR10, we changed the dimension from $384$ to $380$, resulting in a parameter count of $7.99$M, which is almost equal to $8.00$M. As shown in the table above, we can conclude that when the parameter counts are equal, Spikformer with CPG-PE significantly outperforms Spikformer without PE across all image classification datasets, proving our proposed CPG-PE is effective.\n\n### **2.ImageNet Dataset (W2).** \n\nThanks for your suggestion. We have conducted experiments with Spikformer without positional encoding (PE), Spikformer with relative positional encoding (RPE), and Spikformer with our proposed CPG-PE on the ImageNet dataset. The results are as follows:\n\n| Model                       | Param(M) | ImageNet |\n| --------------------------- | -------- | -------- |\n| Spikformer w/o PE           |    $15.50$   |   $69.46$%    |\n| Spikformer w/ RPE \\[1\\]      |    $16.81$     |   $70.24$%    |\n| Spikformer w/ CPG-PE \\[ours\\] |    $15.66$   |    $71.17$%   |\n\nSpecifically, we set the depth to 8 and the dimension of representation to 384. From the table, we can see that CPG-PE performs well on large-scale image datasets. However, due to time constraints, we were unable to conduct experiments on Random PE or Float PE. We believe that the above results demonstrate the effectiveness of our proposed CPG-PE in positional encoding. We will add these experiment results to Table 3 in our revised manuscript.\n\n### **3.Similarities and differences between CPG-PE and addition-based PE in ANNs (Q1).** \n\nIn ANNs, especially in language tasks, the difference between using addition and concatenation for positional encoding is not significant \\[2\\]\\[3\\]. However, in spiking neural networks (SNNs), addition is not suitable for spike matrices because it can result in non-binary values, which is one of the motivations for designing SNN-PE. This paper addresses this issue by first concatenating the spike matrix and then using a projection layer to project it back to the initial dimension.\n\n### Reference\n\n[1] Zhou Z, Zhu Y, He C, et al. Spikformer: When Spiking Neural Network Meets Transformer[C]//The Eleventh International Conference on Learning Representations (ICLR). 2023.\n\n[2] Rosendahl J, Tran V A K, Wang W, et al. Analysis of positional encodings for neural machine translation[C]//Proceedings of the 16th International Conference on Spoken Language Translation. Association for Computational Linguistics. 2019.\n\n[3] Ke G, He D, Liu T Y. Rethinking Positional Encoding in Language Pre-training[C]//International Conference on Learning Representations (ICLR). 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZOzES7ZKjA", "reviewer": "Reviewer_s3xa", "summary": "This paper introduces a novel positional encoding method for SNNs called CPG-PE, inspired by central pattern generators (CPGs) in biological neural systems. The authors demonstrate both theoretically and empirically that CPG-PE can effectively capture positional information in sequential data while maintaining the spike-based nature of SNNs. The approach is evaluated on a range of tasks including time-series forecasting, text classification, and image classification.", "questions": "See my weakness part.", "limitations": "The authors have addressed limitations and future work in Appendix E, which is commendable. They discuss the challenges of applying CPG-PE to non-sequential data like images and propose potential solutions.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Strong theoretical foundation: The authors mathematically demonstrate how CPG-PE relates to conventional sinusoidal positional encoding used in transformers.\n2. Comprehensive empirical evaluation across time-series forecasting, text classification, and image classification tasks shows consistent performance improvements when incorporating CPG-PE.\n3. The method is biologically plausible and potentially compatible with neuromorphic hardware, as it can be implemented using leaky integrate-and-fire neurons.\n4. Well-structured paper with clear motivation, methodology, and insightful analysis of CPG-PE properties and their relationship to biological CPGs.", "weaknesses": "1. Typographical error on line 158 where X is incorrectly specified as belonging to the real number domain instead of the binary domain {0,1} for spike data.\n2. The authors should clarify the similarities and differences between the decay mechanism in CPG-PE and that of LIF neurons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and suggestions that enhanced the quality of our paper. Responses to your concerns and questions are hereby presented:\n\n###  **1.Typographical error (W1)** \n\nApologies for the confusion caused by this typo error. In line 158, the $X$ is the input spike matrix and it belongs to  the binary domain $\\{0,1\\}$ rather than real number domain $\\mathbb{R}$, i.e., $X \\in \\{0,1\\}^{T \\times B \\times L \\times D}$. This error will be corrected in our revised manuscript.\n\n### **2.Similarities and differences between the decay mechanism of CPG-PE and that of LIF neurons (W2)**\n\n(1) If we understand your question correctly (please kindly let us know if we misunderstood), the question is about the differences between the decay mechanisms of *CPG neurons* and LIF neurons instead of *CPG-PE* . To be clear, CPG indicates that the neuron (together with other neurons in a CPG circuit) plays the role of rhythmic output generator, while LIF is a specific dynamics of spiking neuron model. CPG-PE is our propose method which leverages the spikes of CPG neurons for encoding positional information in sequential data. Therefore, CPG-PE and LIF neurons are orthogonal concepts, and we have proved that CPG-PE could be composed of LIF neurons (as shown in Appendix C), or be manually set (for simplicity, as used in our experiments).\n\n(2) Regarding the decay mechanism, as shown in Equation (2), the $\\beta$ control the decay rate of LIF neurons. However, our proposed CPG-PE does not have a decay mechanism. We would greatly appreciate it if you could provide further clarification on this issue, and we are happy to address it.\n\nPlease let us know if you have any remaining questions or concerns, we are commited to addressing the issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BX8cm04fxz", "reviewer": "Reviewer_Pjez", "summary": "This paper proposes a new position encoding technique CPG-PE for SNNs, which is inspired by the central pattern generator in the human brain and improves the ability of SNNs to process sequence data. Experimental results show that CPG-PE outperforms traditional SNNs in multiple fields such as time series prediction, text classification and image classification.", "questions": "1. I suppose the traditional positional encoding in Spikformer can be seen as direct encoding in SNNs. In contrast, the CPG-PE proposed in this paper is more like a temporal encoding with dynamic expressions related to time series. Could the authors discuss the differences between other encoding schemes [1;2;3] and positional encoding in SNNs within the related work?\n\n2. Can the authors provide experimental results of CPG-PE on the ImageNet image classification dataset?\n3. Some tyops: Formulas 4, 5, 6, 7, 8, and 10 are missing commas. There should be a comma after each formula.\n\nI would be pleased to raise the score if the authors address my concerns.\n\n[1] Kim Y, Park H, Moitra A, et al. Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks?[C]//ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 71-75.\n\n[2] Han B, Roy K. Deep spiking neural network: Energy efficiency through time based coding[C]//European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 388-404.\n\n[3] Comsa I M, Potempa K, Versari L, et al. Temporal coding in spiking neural networks with alpha synaptic function[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 8529-8533.", "limitations": "The author has discussed the limitations", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The bio-inspired CPG-PE technique is proposed to enhance the sequence processing capability of SNNs.\n2. The effectiveness of CPG-PE is verified in multiple fields, including time series, text classification, and image classification.\n3. The performance of SNNs in various tasks is significantly improved, outperforming traditional models.\n4. CPG-PE is designed with compatibility with neuromorphic hardware in mind, facilitating deployment in practical applications.", "weaknesses": "1. No experiments were conducted on the large-scale image classification Imagenet dataset.\n2. Other encoding methods in SNN such as rate coding [1] and temporal coding [2;3], were not discussed.\n\n[1] Kim Y, Park H, Moitra A, et al. Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks?[C]//ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 71-75.\n\n[2] Han B, Roy K. Deep spiking neural network: Energy efficiency through time based coding[C]//European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 388-404.\n\n[3] Comsa I M, Potempa K, Versari L, et al. Temporal coding in spiking neural networks with alpha synaptic function[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 8529-8533.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are cheerful that our contribution is well recognized. And thanks for your valuable suggestions that truly enhance the quality of our paper and make it more understandable for the wider community. \n\nResponses to your concerns are presented as follows:\n\n### **1.ImageNet Dataset (W1, Q2)**\n\nThanks for your suggestion. We have conducted experiments with Spikformer without positional encoding (PE), Spikformer with relative positional encoding (RPE), and Spikformer with our proposed CPG-PE on the ImageNet dataset. The results are as follows:\n\n| Model                       | Param(M) | ImageNet |\n| --------------------------- | -------- | -------- |\n| Spikformer w/o PE           |    $15.50$   |   $69.46$%    |\n| Spikformer w/ RPE \\[1\\]      |    $16.81$     |   $70.24$%    |\n| Spikformer w/ CPG-PE \\[ours\\] |    $15.66$   |    $71.17$%   |\n\nSpecifically, we set the depth to 8 and the dimension of representation to 384. For other experimental settings, we have faithfully followed the guidelines in \\[1\\]. From the table, we can see that CPG-PE performs well on large-scale image datasets. However, due to time constraints, we were unable to conduct experiments on Random PE or Float PE. We believe that the above results demonstrate the effectiveness of our proposed CPG-PE in positional encoding. We will add these experiment results to Table 3 in our revised manuscript.\n\n\n### **2.Discussion on temporal coding and rate coding (W2, Q1)**\n\n Thanks for your suggestion.\n\n(1) We carefully reviewed the literature you provided and found your suggestions valuable for improving the quality of our paper. In the revised manuscript, we will include the following discussion in the related work section (due to character limitation, we omit some references but will include them in the revised manuscript):\n\n\"Spiking Neural Networks (SNNs) employ several coding methods to encode input information, each offering unique advantages. Direct coding \\[1\\]\\[2\\], the simplest form and widely-used in image tasks, directly associates spikes with specific values or events, providing straightforward and interpretable outputs but often lacking efficiency for complex tasks. Rate coding \\[3\\]\\[4\\], where the input is represented by the frequency of spikes within a given timeframe, is more robust and widely used but can be less precise due to its reliance on averaged spike rates. Temporal coding \\[5\\]\\[6\\] (a.k.a latency coding) encodes information based on the timing of individual spikes, allowing for high temporal precision and efficient representation of dynamic inputs, though it can be computationally demanding. In addition, Delta coding \\[7\\] represents changes in input signals through spikes, focusing on differences rather than absolute values, which can enhance efficiency and response times but may introduce complexity in decoding. Each of these methods contributes to the versatility and applicability of SNNs in various domains, from neuroscience to artificial intelligence. The SNNs we considered in this paper should fall into the category of rate coding since back-prop is conducted on spike rate. Meanwhile, CPG-PE can be considered converting temporal information into spike rate of a group of neurons (Equations 11-12), and this is why CPG-PE can improve performance for sequential data. It is possible to introducing learning algorithms of temporal coding for the CPG neurons to tackle more complex sequence structure, which remains as future work.\"\n\n(2) In Related Works Section, we have discussed the positional encoding used in SNNs.\n\n###  **3.Typo (Q3)**\n Thanks for your reminding. We will add commas after formulas 4-10 to ensure readers are not misled.\n\n### Reference\n\n[1] Zhou Z, Zhu Y, He C, et al. Spikformer: When Spiking Neural Network Meets Transformer[C]//The Eleventh International Conference on Learning Representations (ICLR). 2023.\n\n[2] Yao M, Hu J, Zhou Z, et al. Spike-driven transformer[J]. Advances in Neural Information Processing Systems (NeurIPS), 2023.\n\n[3] Kim Y, Park H, Moitra A, et al. Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks?[C]//ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 71-75.\n\n[4] Lv C, Xu J, Zheng X. Spiking Convolutional Neural Networks for Text Classification[C]. // The Eleventh International Conference on Learning Representations (ICLR). 2023.\n\n[5] Han B, Roy K. Deep spiking neural network: Energy efficiency through time based coding[C]//European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 388-404.\n\n[6] Comsa I M, Potempa K, Versari L, et al. Temporal coding in spiking neural networks with alpha synaptic function[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 8529-8533.\n\n[7] Yoon Y C. LIF and simplified SRM neurons encode signals into spikes via a form of asynchronous pulse sigma–delta modulation[J]. IEEE transactions on neural networks and learning systems, 2016, 28(5): 1192-1205.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response

We express our gratitude to all the reviewers for the valuable insights and acknowledging our contributions to advance the sequential modeling ability of SNNs through central pattern generators. We are encouraged by the comments highlighting the strengths of our work:

- Clear motivation, novelty, and innovation (Reviewer 3Lmi, s3xa)
- Effectiveness / consistent performance improve (Reviewer 3Lmi, s3xa, Pjez)
- Both Hardware-friendliness and biologically-plausibility (Reviewer 3Lmi, s3xa, Pjez)
- Comprehensive experiments and analysis (Reviewer 3Lmi, s3xa, Pjez)
- Well-organized and clearly-written (Reviewer P6YU, s3xa)

A major concern most reviewers share is the evaluation on large-scale image classification benchmark, i.e., ImageNet. We agree with the comment and to address this issue, we have conducted experiments with Spikformer without positional encoding (PE), Spikformer with relative positional encoding (RPE), and Spikformer with our proposed CPG-PE on the ImageNet dataset. The results are as follows:

| Model                       | Param(M) | ImageNet Acc (Top 1)|
| --------------------------- | -------- | -------- |
| Spikformer w/o PE           |    $15.50$   |   $69.46$%    |
| Spikformer w/ RPE       |    $16.81$     |   $70.24$%    |
| Spikformer w/ CPG-PE \[ours\] |    $15.66$   |    $71.17$%   |

In particular, we set the depth to 8 and the dimension of representation to 384. From the table, we can see that CPG-PE performs well on large-scale image datasets. However, due to time constraints, we were unable to conduct experiments on Random PE or Float PE. We believe that the above results demonstrate the effectiveness of our proposed CPG-PE in positional encoding.

One comment on weakness is about our contributions. We would like to highlight several key contributions that distinguish our work from existing literature: **(1) Pioneer in the exploration of positional encoding for SNNs.** To the best of our knowledge, our proposed CPG-PE is the first work on spike-version positional encoding in SNNs, which are both brain-inspired and hardware-friendly. We corrected the inappropriate positional encoding designs previously used in SNNs. **(2) Consistent performance gain with a simple method.** CPG-PE enhances the performance of SNNs across a wide range of tasks, including time-series forecasting, text classification, and image classification. **(3) Insights on neuroscience and neuromorphic hardware.** We mathematically proved that the traditional positional encoding in Transformers is a particular solution of the membrane potential variations in a specific type of CPG. Additionally, we demonstrated that CPG-PE can be implemented with 2 LIF neurons, ensuring it does not introduce any burden of redesigning hardware.


We will address the concerns and polish our paper in the revised version to enhance its clarity and accessibility for the wider community. To summarize the updates:

1. Extensive experiment results on large-scale dataset, specifically ImageNet.
2. Further discussion on spike encoding methods, including temporal coding and rate coding, in the related work section.
3. Revision of typos and proofreading for better language.

We are confident that our work contributes to the NeurIPS community by advancing neuromorphic AI and potentially computational neuroscience. We are happy to answer follow-up questions from the reviewers if anything remains unclear.

Meta-review: All reviewers agree that this is an interesting work that can be accepted for NeurIPS. The paper proposes biologically inspired central pattern generators (CPGs) as a novel method for position encoding in spiking neural networks (SNNs).
The reviewers noted that the work
-    covers new ground for sequential processing with SNNs,
-    has a strong theoretical foundation,
-    provides a comprehensive empirical evaluation on a variety (and also large) data sets,
-    shows significantly improved performance over previous SNN models,
-    is well written and has insightful analysis.
I therefore recommend to accept the paper.
Decision: Accept (spotlight)