Title: Advancing Spiking Neural Networks for Sequential Modeling with Central Pattern Generators
Venue: NeurIPS 2024
Accepted: yes
Keywords: Spiking Neural Networks

Central Pattern Generators

Positional Encoding
Abstract: Spiking neural networks (SNNs) represent a promising approach to developing artificial neural networks that are both energy-efficient and biologically plausible.
However, applying SNNs to sequential tasks, such as text classification and time-series forecasting, has been hindered by the challenge of creating an effective and hardware-friendly spike-form positional encoding (PE) strategy.
Drawing inspiration from the central pattern generators (CPGs) in the human brain,  which produce rhythmic patterned outputs without requiring rhythmic inputs, we propose a novel PE technique for SNNs, termed CPG-PE.
We demonstrate that the commonly used sinusoidal PE is mathematically a specific solution to the membrane potential dynamics of a particular CPG.
Moreover, extensive experiments across various domains, including time-series forecasting, natural language processing, and image classification, show that SNNs with CPG-PE outperform their conventional counterparts.
Additionally, we perform analysis experiments to elucidate the mechanism through which SNNs encode positional information and to explore the function of CPGs in the human brain.
This investigation may offer valuable insights into the fundamental principles of neural computation.

Advancing Spiking Neural Networks for Sequential Modeling with Central Pattern Generators

Abstract

Spiking neural networks (SNNs) represent a promising approach to developing artificial neural networks that are both energy-efficient and biologically plausible. However, applying SNNs to sequential tasks, such as text classification and time-series forecasting, has been hindered by the challenge of creating an effective and hardware-friendly spike-form positional encoding (PE) strategy. Drawing inspiration from the central pattern generators (CPGs) in the human brain, which produce rhythmic patterned outputs without requiring rhythmic inputs, we propose a novel PE technique for SNNs, termed CPG-PE. We demonstrate that the commonly used sinusoidal PE is mathematically a specific solution to the membrane potential dynamics of a particular CPG. Moreover, extensive experiments across various domains, including time-series forecasting, natural language processing, and image classification, show that SNNs with CPG-PE outperform their conventional counterparts. Additionally, we perform analysis experiments to elucidate the mechanism through which SNNs encode positional information and to explore the function of CPGs in the human brain. This investigation may offer valuable insights into the fundamental principles of neural computation. Our code is available at https://github.com/microsoft/SeqSNN .

1 Introduction

Spiking neural network (SNN) [1] has increasingly attracted research interests in recent years, primarily due to its energy efficiency, event-driven paradigm, biological plausibility, and other distinctive properties. The spiking neurons in SNN are dynamical systems that generate binary signals (spike or non-spike) and communicate these signals like artificial neural networks (ANNs) for computation [2-9]. Many advanced architectures and methodologies developed for ANNs are also applicable to SNNs, enhancing their capabilities. Notable among these are backpropagation [10], batch normalization [11, 12], and Transformer architecture [4, 13, 5, 6], which collectively broaden the functional scope of SNNs.

Despite the promising advances in SNNs, several challenges persist when adapting them to diverse tasks. A fundamental challenge is that SNNs, which are event-triggered, lack robust and effective mechanisms to capture indexing information, rhythmic patterns, and periodic data. This limitation can adversely affect SNNs' ability to process and analyze different data modalities, including natural language, and time series. Meanwhile, while SNNs aim to emulate the neural circuits of the brain,

their reliance on spike-based communication imposes limitations. Consequently, not all deep learning techniques applicable to ANNs can be directly transferred to SNNs. For instance, methods like HiPPO [14] or trigonometric positional encoding [15] are not readily compatible with the spike format used in SNNs. Moreover, even the most state-of-the-art ANNs still lag significantly behind human capabilities in many tasks [16, 17]. Therefore, to enhance the functionality of SNNs, one promising approach is to draw further inspiration from biological neural mechanisms. In this regard, we propose the analogy of central pattern generators (CPGs) [18], a kind of neural circuit well-known in neuroscience, with positional encoding (PE), a technique extensively utilized in deep learning. This analogy is designed to operate within the SNN framework, potentially bridging the gap between biologically inspired models and modern deep learning techniques.

In neuroscience, a CPG (See Figure 2 for an illustration) is a group of neurons capable of producing rhythmic patterned outputs without requiring rhythmic inputs [19, 20]. These neural circuits are found in the spinal cord and brainstem and are responsible for generating the rhythmic signals that control vital activities such as locomotion, respiration, and chewing [21].

On the other hand, PE is an important technique for ANNs, particularly within models tailored for sequence processing task [15, 22, 23]. By endowing each element of the input sequence with positional information, typically achieved through diverse mathematical formulations or learnable embeddings, neural networks acquire the capability to discern the order and relative positions of the elements within the sequence.

We argue that these two concepts, despite seemingly unrelated, can be connected profoundly. Intuitively, CPG and PE both generate periodic outputs (with respect to time for CPG and with respective to position for PE). Moreover, in this paper, we reveal a deeper relationship between these two concepts by showing that the widely used sinusoidal PE is mathematically a particular solution of the membrane potential dynamics of a specific CPG .

However, current SNN architectures exhibit a notable deficiency in implementing an effective and biologically plausible PE mechanism. Existing so-called positional encoding methods for SNNs [4, 5] rely on input data, often resulting in non-spike and repetitive outputs for different positions. Furthermore, incorporating PE techniques designed for ANNs necessitates the calculation of membrane potentials, which is incompatible with the spike format of SNNs. To address these issues, we draw inspiration from the spiking properties of the CPGs and propose a straightforward yet versatile PE technique for SNNs, termed CPG-PE. This method encodes positional information with multiple neurons with various patterns of spike trains. To summarize the highlights of our study:

Novel Positional Encoding for SNNs. We introduce a bio-plausible and effective PE approach tailored specifically for SNNs. This innovative strategy draws inspiration from the central pattern generator found in the human brain. Additionally, we propose a straightforward implementation of CPG-PE in SNNs, which is also compatible with neuromorphic hardware as it can be realized using circuits of leaky integrate-and-fire neurons.

Consistent Performance Gain. Our proposed methods significantly and consistently enhance the performance of SNNs across a wide range of sequential tasks, including timeseries forecasting and text classification.

Insightful Analysis. Our research represents one of the pioneering efforts to comprehensively analyze (1) the mechanism by which SNNs capture positional information and (2) the role of CPGs in the brain. This analysis provides valuable insights into the underlying principles of neural computation.

2 Preliminaries

2.1 Spiking Neural Networks

The basic unit in SNNs is the spiking neuron, such as the leaky integrate-and-fire (LIF) neuron [1], which operates based on an input current I ( t ) and influences the membrane potential U ( t ) and the

(a) PE in Transformer

(b) PE in Spike Transformer

(c) CPG-PE (Ours)

Addition

𝑿

Conv2d

𝑿+𝑿′

Depends on the Whole

𝑿

Depends on the

Sequence Length of

𝑿

Uniqueness of

Each Position

Non-spike Output

Addition

𝑿

𝑿+𝑿′

Depends on the

Sequence Length of

𝑿

Spike Output

Uniqueness of Each

Position in Spike-form

Concatenation

𝑿

𝑿

𝑿′

CPGs

𝜽

…

Sin / Cos

Time-series Forecasting

Text Classification

Image Classification

(d) Performance

w/o PE

CPG-PE

CPG-PE

CPG-PE

𝟕𝟕. 𝟐𝟓

𝟕𝟗. 𝟒𝟏

𝟎. 𝟕𝟏𝟗

𝟎. 𝟕𝟒𝟒

𝟖𝟏. 𝟐𝟓

𝟖𝟑. 𝟑𝟖

w/o PE

w/o PE

Non-uniqueness of Each

Position in Spike-form

spike S ( t ) at time t . The dynamics of the LIF neuron are described by the following equations:

Here, I ( t ) is the spatial input to the LIF neuron at time step t , calculated using the function f with x as input and θ as learnable parameters. ∆ t is the discretization constant that determines the granularity of LIF modeling, and H ( t ) is the temporal output of the neuron at time step t . The spike S ( t ) is defined as a Heaviside step function based on the membrane potential. When U ( t ) reaches the threshold U thr , the neuron fires, emitting a spike, and the temporal output H ( t ) resets to V reset . If the membrane potential U ( t ) does not reach the threshold, no spike is emitted, and U ( t ) decays to H ( t ) at a decay rate of β .

In this paper, we choose direct training with surrogate gradients as our method to train SNNs. we follow [24] to choose the arctangent-like surrogate gradients as our error estimation function when backpropagation, which regards the Heaviside step function as: S ( t ) ≈ 1 π arctan( π 2 αU ( t )) + 1 2 , where α is a hyper-parameter to control the frequency of the arctangent function. Therefore, the gradients of S are ∂S ( t ) ∂U ( t ) = α 2 1 (1+( π 2 αU ( t )) 2 ) and thus the overall model can be trained in an end-to-end manner with back-propagation through time (BPTT) [25].

2.2 Positional Encoding

In the field of sequential tasks, PE is crucial for models like Transformers to understand the sequential order of input tokens. Absolute PE and relative PE are two prominent methods used to incorporate positional information into these models. Absolute PE [15] assigns fixed embeddings to each position in the input sequence using trigonometric functions like sine and cosine. These embeddings are based solely on the position index and are not influenced by the token content, which are predefined and are generated as follows:

Here, pos is the position and d is the dimension. In contrast, relative PE [26-28] captures the relationships between tokens by considering their relative distances. This dynamic approach allows models to learn position-specific patterns and dependencies, which is beneficial for tasks requiring different sequence lengths or hierarchical structures.

However, existing SNN architectures reveal a notable deficiency in the integration of an effective and biologically plausible PE mechanism. As shown in Figure 1, current Transformer-based SNNs [4, 5] are primarily tailored for image classification and predominantly rely on a convolutional layer to capture the relative positional information of image patches. However, this approach resembles more of a spike-element-wise (SEW) residual connection [2] rather than a classic PE module, as it does not ensure that each image patch has a unique spike-form positional representation. Furthermore, the addition between positional spikes and the original input spikes within these models may yield hardware-unfriendly non-binary integers (i.e., neither 0 nor 1 ), resulting from the addition of ' 1 ' and ' 1 '. Additionally, our investigation reveals that even SNNs designed for sequential tasks, such as SpikeBERT [29, 30], SpikeGPT [31], and SpikeTCN [32], also exhibit a notable absence of an effective spike-form PE mechanism for capturing positional information.

We think that an effective PE strategy should possess the following characteristics: uniqueness of each position and the capacity to capture positional information from the input data . Furthermore, an optimal PE designed for SNNs should be hardware-friendly and in spike-form .

2.3 Central Pattern Generators

Central Pattern Generators (CPGs) are neural networks capable of producing rhythmic patterned outputs without sensory feedback [18, 20]. These networks are fundamental for understanding motor control in vertebrates and invertebrates and are often applied to robotics and neural control systems. Mathematically, CPGs can be modeled using systems of coupled nonlinear oscillators, and the general form can be written as:

where x and y are the state variables (can be seen as membrane potential) of two coupled oscillators, F and H are intrinsic dynamics of the oscillators, and G and K are the coupling functions.

3 Methods

In biological systems, CPGs as well as other neurons do not transmit information directly through membrane potential but through spikes. A burst of spikes will be generated only when the membrane potential of a CPG exceeds a certain threshold. Therefore, we introduced the Heaviside step function in SNN, selecting only the part that exceeds the threshold, to design the CPG-PE. In this section, we will first reveal the relationship between CPGs and PE. Then we will introduce our proposed CPG-PE and its implementations.

3.1 Relationship between Central Pattern Generators and Positional Encoding

Consider one of the simplest CPGs with the following assumptions:

The CPG is a coupled nonlinear oscillator with 2 neurons whose states are represented as x ( t ) and y ( t ) .

Both neurons are autonomic neurons and will gain membrane voltage with constant speed, i.e., F ( x ) = b > 0 , H ( y ) = d > 0 .

Neuron represented by x will inhibits y while y excites x . And the influence is proportional to the other neuron's state. Formally, G ( x , y ) = a y , K ( x , y ) = -c x where a > 0 , c > 0 .

Now the coupled oscillators can be represented as:

The general solution of this differential equation system is:

where k 1 and k 2 are arbitrary constants. To simplify, we can further re-parameterize t with t ′ = t +arctan( k 1 /ak 2 ) as is to choose another start point, then we can rewrite Equations (7) and (8) as:

Comparing Equations (9) and (10) and Equation (4), we are astonished to find that the PE in Transformers [15] is a particular solution of the membrane potential variations in a specific type of CPG with properly chosen a, b, c, d . This finding suggests that the use of sinusoidal PE in Transformers is actually a bio-plausible choice that could possibly advance the model's ability to learn indexing and periodic information.

Spikes

Central Pattern Generators

Spikes

Inhibit Each Other

Inhibit

Inhibit

(a)

(b)

neuron#1

0

5

10

15

20

25

30

35

40

45

50

neuron#2

neuron #3

0

5

10

15

20

25

30

35

40

45

50

01

0

5

10

15

20

25

30

35

40

45

50

neuron #4

0

5

1

10

15

20

25

30

35

40

45

50

time step

3.2 CPG-based Positional Encoding

Consider a system with N pairs of CPG neurons, resulting in a total of 2 N cells. Then for i = 1 , 2 , ..., N , the equations governing the CPG-PE are as follows:

where η is a constant to control the period, τ represents the base period, and v thres denotes the membrane potential threshold. Note that this threshold is different from the U thr of spike neurons described in Equation (3). The Heaviside step function H reflects a spike when the membrane potential exceeds the threshold.

It is important to clarify that the t in Equation (11) and 12 is neither the time step in SNNs nor the position index. Suppose the input spike matrix X ∈ { 0 , 1 } T × B × L × D , where T is the time step in SNNs, B is the batch size, L is the sequence length of the input sample, D is the feature dimension. To ensure the uniqueness of each position at every time step, we flatten the dimensions T and L into a new dimension T × L . Therefore, t ranges from 0 to T × L . Notably, the entire CPG-PE operates in spike-form and is parameter-free. To better understand CPG-PE, we draw a simple approximation of the resulting CPG spiking patterns under the assumption of a sequence length of L = 128 and N = 20 pairs of CPG neurons, illustrated in Figure 2 (b).

3.3 Implementations

We design a simple implementation to apply CPG-PE to SNNs in a pluggable and hardware-friendly manner, shown in Figure 3. Before diving into the details, we want to emphasize that the data transmitted in SNNs should always be in spike-form. Therefore, the direct addition operation between two spike matrices, as used in [4, 5], should be forbidden.

𝑿

𝒊 = 𝟎

𝒊 = 𝟏

𝒊 = 𝟐𝑵

𝒊 = 𝟐

…

…

CPG-PE

𝑿

𝑿′

𝑿

𝑿′

Positional Encoding

𝑻

Time Steps in SNNs

𝑳

Sequence Length of

𝑿

Linear

𝑻 × 𝑳

𝑵

Number of CPG Pairs

𝒊

Index of CPG Neurons

BN

𝑿𝒐𝒖𝒕𝒑𝒖𝒕

Spiking Neuron Layer

(𝑫 + 𝟐𝑵)

To

𝑫

Initially, CPG-PE encodes the positional information of the input spike matrix X , resulting in X ′ . Then, to maintain binary values and avoid introducing non-binary elements, we opt to concatenate X and X ′ along the feature dimension. Lastly, a linear layer is employed to map the feature dimension from D +2 N back to D , where D is the feature dimension of X , and N is the number of CPG pairs. This effectively neutralizes the dimensional increase caused by concatenation. The whole process can be formalized as follows:

where BN represents batch normalization and SN is a spike neuron layer. Furthermore, CPG-PE necessitates that input samples be sequential data, making it directly applicable to time series data and natural language. For image data, however, an adaptation is required: images must be segmented into patches similar to the approach used in the Vision Transformer [23]. Considering the compatibility with neuromorphic hardware, we also (1) implement CPG-PE with LIF neurons, and (2) integrate CPG-PE into a classic linear layer. Please refer to Appendices C and D for details.

4 Experiments

In this section, we conduct experiments to investigate the following research questions:

RQ1 : Is our design of CPG-PE strategy effective and robust in sequential tasks?

RQ2 : Can CPG-PE work well on image patches that have no inherent order?

RQ3 : How will CPG's inner properties influence CPG-PE?

RQ4 : Does our CPG-PE satisfy the requirements of a good PE tailored for SNNs?

4.1 Datasets

To assess the PE capabilities of the compared models and answer RQ1 , we conduct two sequential tasks: time-series forecasting , and text classification . Following [32], we choose 4 real-world datasets for time-series forecasting: Metr-la [33]: This dataset contains the average traffic speed data collected from the highways in Los Angeles County. Pems-bay [33]: It consists of average traffic speed data from the Bay Area. Electricity [34]: This dataset captures hourly electricity consumption measured in kilowatt-hours (kWh). Solar [34]: It includes data on solar power production. For text classification, we follow [29] to conduct experiments on 6 benchmarks including: Movie Reviews [35], SST2 [36], SST5 , Subj, ChnSenti, and Waimai. In addition, to answer RQ2 , we also conduct image classification experiments on 1 static datasets CIFAR and 1 neuromorphic datasets CIFAR10DVS [37]. The dataset details and metrics are provided in Appendix A.

4.2 Time-Series Forecasting

As discussed in Section 3.3, our proposed CPG-PE can be seamlessly integrated into any SNN capable of sequence processing. Consequently, we applied CPG-PE to the SNN counterparts of Temporal Convolutional Networks (TCN) [38], Recurrent Neural Networks (RNN) to assess the efficacy of our method in enabling SNNs to capture positional information. The results for TCN, SpikeTCN w/o PE, RNN, and Spike-RNN w/o PE are sourced from the previous study by [32]. In addition, we

deliberately conducted experiments on PE in Spikformer to explore whether our specially designed CPG-PE is truly more suitable for SNNs than all previous PEs. Notably, we also investigated the modularization of CPG, i.e., replacing all Linear layers with CPG-Linear layers (See Appendix D), and its impact on the Spikformer model for time-series forecasting, i.e., Spikformer w/ CPG-Full. We report the results on 4 time-series forecasting benchmarks with various prediction lengths in Table 1. We also list results from ANNs for reference.

In summary, the results presented in Table 1 indicate that SNNs equipped with the CPG-PE module significantly outperform their counterparts lacking the PE feature. This finding effectively addresses RQ1 from a time-series analysis perspective. Detailed findings include:

(1) CPG-PE enables SNNs to successfully capture positional information . SNNs, including models such as Spike-TCN, Spike-RNN, and Spikformer, when integrated with CPG-PE, show superior performance compared to those without PE. Notably, CPG-PE also reduces the performance disparity between SNNs and traditional ANNs in time-series forecasting tasks, evidenced by an average increase of 0 . 013 in R 2 and a decrease of 0 . 022 in RSE.

(2) CPG-PE is the most suitable position encoding strategy for Spikformer . In addition to CPG-PE, other encoding strategies such as Float-PE (the original PE in Transformer) and RPE (the original PE in Spikformer) were also evaluated. The Spikformer equipped with CPG-PE emerged as the top-performing variant, confirming CPG-PE as the most suitable PE strategy for SNNs.

(3) CPG-Full module can also effectively model the positional information of time series data . The CPG-Full module's performance in modeling positional information of time-series data is comparable to that of CPG-PE, with average R 2 values nearly identical to those of Spikformer with CPG-PE and significantly better than those of other models.

4.3 Text Classification

In addition to time-series forecasting, natural language processing (NLP) serves as another critical domain to assess the efficacy of the CPG-PE module in encoding positional information. Following the pioneering work of [29], who first employed Spikformer for text classification tasks, we extended this application to 6 benchmark datasets. We also include results from fine-tuned BERT for reference.

The results presented in Table 2 shows that Spikformer enhanced with CPG-PE achieves the state-ofthe-art performance across 6 benchmarks, effectively addressing RQ1 . Meanwhile, we conducted a set of ablation experiments to eliminate the effects of increased parameter counts on model performance. Specifically, we replaced the spike-form positional encoding matrix obtained from CPG with a randomly generated spike matrix (See 'Spikformer w/ Random PE' Row). By comparing these two configurations, we confirmed the effectiveness of our proposed CPG-PE.

4.4 Image Classification

In this section, we aim to answer RQ2 . To adapt the CPG-PE for image classification, it is essential to conceptualize the array of image patches as sequential data. Consequently, some SNN models that do not incorporate a concept of 'sequence length' in their spike matrices, such as SEW-Resnet [2], are incompatible with the integration of a CPG-PE module. Therefore, we only consider ViT-liked SNN, i.e. Spikformer, in this experiment. We also include results from ViTs for reference.

We report the parameter counts and classification accuracy in Table 5. To elaborate, Spikformer with CPG-PE outperforms other variants, demonstrating the effectiveness of CPG-PE even when the sequence is an array of image patches lacking inherent order. Notably, owing to our streamlined implementation, the parameter count for Spikformer with CPG-PE is significantly reduced compared to the original Spikformer w/ RPE [4], with a reduction of 1 . 16 M. What's more, we conducted ablation experiments on model parameters by reducing the parameter count of Spikformer with CPG-PE to be comparable to Spikformer w/o PE, allowing for a more direct performance comparison, as shown in the last line in Table 5. The results on ImageNet are reported in Appendix E.

However, it is essential to acknowledge that the improvements in image classification are relatively modest compared to those observed in time series and text applications. This phenomenon can largely be attributed to the intrinsic non-ordered nature of image patches. Unlike text or time series data, where sequential order is crucial and inherently informative, image patches do not possess a natural or fixed sequence. This lack of order means that traditional methods of positional encoding, which significantly benefit ordered data by providing contextual positioning, are less effective. Thus, the application of our positional encoding techniques, optimized for data with inherent sequential order, does not translate as effectively to the domain of image classification.

4.5 Sweeping CPG properties

In this section, we investigate the influence of CPG properties on the ability to model positional information, addressing RQ3 . To this end, we evaluated the Spikformer model with CPG-PE by varying the base period τ and the number of CPG pairs N (see Equations (11) and (12)) in time-series forecasting and image classification tasks.

From Figure 4 (a) and (b), we observe that CPG-PE is insensitive to the base period τ (in biological neurons, τ is affected by the physiological properties of the CPG circuit such as RC constant and synaptic delay). The sequence lengths ( T × L ) of the time series and image patches are no larger than 672 (4 × 168) for all benchmarks, preventing repetitions in CPGs. Therefore, when N = 20 , sweeping τ ∈ { 100 , 1000 , 5000 , 10000 } makes minor influence on performance. Furthermore, Figure 4 (c) and (d) demonstrate that when τ = 10000 , increasing the number of CPG pairs N enhances Spikformer's performance. This is reasonable because more CPG neurons reduce repetitions in positional representations of X ′ .

100 1000

5000

10000

Base Period

0.5

0.6

0.7

0.8

0.9

1.0

R

2

Spikformer w/ CPG-PE

Metr-la

Pems-bay

Solar

Electricity

(a)

100 1000

5000

10000

Base Period

60

70

80

90

100

Acc

%

Spikformer w/ CPG-PE

CIFAR10

CIFAR10-DVS

CIFAR100

(b)

5

10

20

Number of CPG Pairs,

N

0.65

0.70

0.75

0.80

0.85

R

2













Spikformer w/ CPG-PE

Pems-bay

Solar

(c)

5

10

20

Number of CPG Pairs,

N

75

76

77

78

Acc

%













Spikformer w/ CPG-PE

CIFAR100

CIFAR10_DVS

(d)

4.6 Positional Encoding Analysis

In this section, we want to address RQ4 . As mentioned in Section 2.2, an ideal PE method for SNNs should include the following characteristics: (1) Uniqueness of each position ; (2) Compatibility with neuromorphic hardware ; (3) Formulation in spike-form . Our implementations ensure compatibility with neuromorphic hardware (2), and the CPG-PE is inherently formulated in spikeform, satisfying (3). Therefore, in order to assess (1), the uniqueness of each position, we would like to compare the CNN-based RPE in [4, 5] and our proposed CPG-PE, focusing specifically on their capacity to provide distinct positional signals. This analysis was conducted using the CIFAR10-DVS dataset, where we calculated the repetition rate of spike positional representations across all positions. Our findings were notable: the positional spike matrices produced by RPE exhibited a repetition rate as high as 12 . 19 % , which significantly undermines its effectiveness for PE. In contrast, our proposed CPG-PE exhibited no repetition, demonstrating that our CPG-PE is well-suited for serving as the PE module in SNNs. Please refer to Appendix B for details.

5 Related Work

5.1 Spike Encoding Methods

Spiking neural networks employ several coding methods to encode input information, each offering unique advantages. Direct coding [5, 40], the simplest form and widely-used in image tasks, directly associates spikes with specific values or events, providing straightforward and interpretable outputs but often lacking efficiency for complex tasks. Rate coding [8, 41], where the input is represented by the frequency of spikes within a given timeframe, is more robust and widely used but can be less precise due to its reliance on averaged spike rates. Temporal coding (a.k.a latency coding) [42, 43] encodes information based on the timing of individual spikes, allowing for high temporal precision and efficient representation of dynamic inputs, though it can be computationally demanding. In addition, delta coding [44] represents changes in input signals through spikes, focusing on differences rather than absolute values, which can enhance efficiency and response times but may introduce complexity in decoding. Each of these methods contributes to the versatility and applicability of SNNs in various domains, from neuroscience to artificial intelligence. The SNNs we considered in this paper should fall into the category of rate coding since back-prop is conducted on spike rate. Meanwhile, CPG-PE can be considered converting temporal information into spike rate of a group of neurons (Equations 11 and 12), and this is why CPG-PE can improve performance for sequential data. It is possible to introducing learning algorithms of temporal coding for the CPG neurons to tackle more complex sequence structure, which remains as future work.

5.2 Positional Encoding in SNNs

Currently, few works have demonstrated the importance of PE approaches in SNNs. Spikformer [4] and Spike-driven Transformer [5] utilize a combination of 'one convolutional layer + one batch normalization layer + one spiking neuron layer ' to generate learnable 'relative positional encoding'. From our perspective, this strategy is more like a spike-element-wise residual connection [2], rather than a classic PE module. The unique representation of each position is a fundamental requirement for a robust PE module. However, the spike position matrix generated by their method may result in the same spike representation for different positions. Additionally, the addition of the input spike matrix and the position spike matrix will result in the occurrence of non-binary numbers (i.e., 2 ) due

to the addition of 1 and 1 . For spiking graph neural networks, [45] proposed learnable positional graph spikes, aiming to capture neighbor information within graphs rather than sequences. Therefore, drawing inspiration from the periodic automatic spike generation pattern of CPGs, we propose a biologically plausible and effective spike-form absolute position encoding method called CPG-PE.

6 Rethinking the Role of CPGs in Neuroscience

Our study also provides novel insights into neuroscience on understanding the role of CPG in nervous systems. While traditionally CPG is believed to play a crucial role in producing the rhythmic motor patterns necessary for locomotion and other repetitive movements [18, 20], the analogy to PE in this work reveals that CPG can make a significant contribution in processing sequential data by encoding the positional information into unique spiking patterns at different times. This does not only work for time-series sensory input like auditory signals but also for visual sensory data: e.g. when a person looks at an image, saccades (eye movements) allow retinal neurons to receive different parts of the image at different times. This indicates that CPG neurons could potentially be utilized to encode positional information. Another extensive thought is that as PE can be learnable in ANNs, CPG may also benefit from adaptability to the data [46]. The hypothesis, however, remains to be examined through neuroscientific experiments [47].

7 Conclusion

In conclusion, inspired by central pattern generators, we introduce a pioneering position encoding approach termed CPG-PE, specifically tailored to mitigate the constraints associated with current PE techniques within SNNs. We mathematically prove that abstract PE in the Transformer is a particular solution of the membrane potential variations in a specific type of CPG. Furthermore, through comprehensive empirical investigations across diverse domains including time-series forecasting, natural language processing, and image classification, we demonstrate that the CPG-PE satisfies all the requirements of PE tailored for SNNs. The limitations and future work are discussed in Appendix F.

References

Wofgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural Networks , 14:1659-1671, 1997.

Wei Fang, Zhaofei Yu, Yanqing Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In Neural Information Processing Systems , 2021.

Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ANN-SNN conversion for fast and accurate inference in deep spiking neural networks. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021 , pages 2328-2336, 2021.

Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 , 2023.

Man Yao, JiaKui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, XU Bo, and Guoqi Li. Spikedriven transformer. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Man Yao, JiaKui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, Bo XU, and Guoqi Li. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. In The Twelfth International Conference on Learning Representations , 2024.

Wei Fang, Zhaofei Yu, Yanqing Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2641-2651, 2020.

Changze Lv, Jianhan Xu, and Xiaoqing Zheng. Spiking convolutional neural networks for text classification. In The Eleventh International Conference on Learning Representations (ICLR) , 2023.

Yuhang Li, Tamar Geller, Youngeun Kim, and Priyadarshini Panda. Seenn: Towards temporal spiking early exit neural networks. Advances in Neural Information Processing Systems , 36, 2024.

Yujie Wu, Lei Deng, Guoqi Li, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience , 12:323875, 2018.

Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11062-11070, 2021.

Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. Advances in Neural Information Processing Systems , 35:34377-34390, 2022.

Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Han Zhang, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual learning for transformer-based spiking neural network. arXiv preprint arXiv:2304.11954 , 2023.

Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems , 33:1474-1487, 2020.

Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv , abs/1706.03762, 2017.

Anthony M Zador. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature communications , 10(1):3770, 2019.

Melanie Mitchell. Why ai is harder than we think. In Proceedings of the Genetic and Evolutionary Computation Conference , pages 3-3, 2021.

Eve Marder and Dirk Bucher. Central pattern generators and the control of rhythmic movements. Current biology , 11(23):R986-R996, 2001.

Eve Marder and Ronald L Calabrese. Principles of rhythmic motor pattern generation. Physiological reviews , 76(3):687-717, 1996.

Sten Grillner. Biological pattern generation: the cellular and computational logic of networks in motion. Neuron , 52(5):751-766, 2006.

Ole Kiehn. Decoding the organization of spinal circuits that control locomotion. Nature Reviews Neuroscience , 17(4):224-238, 2016.

Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In European Conference on Computer Vision , pages 531-548. Springer, 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Yonghong Tian, and other contributors. Spikingjelly, 2020.

Paul J. Werbos. Backpropagation through time: What it does and how to do it. Proc. IEEE , 78:1550-1560, 1990.

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pages 464-468. Association for Computational Linguistics, 2018.

Vighnesh Shiv and Chris Quirk. Novel positional encodings to enable tree-based transformers. Advances in neural information processing systems , 32, 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020.

Changze Lv, Tianlong Li, Jianhan Xu, Chenxi Gu, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Spikebert: A language spikformer learned from bert with knowledge distillation. 2023.

Malyaban Bal and Abhronil Sengupta. Spikingbert: Distilling bert to train spiking language models using implicit differentiation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 10998-11006, 2024.

Rui-Jie Zhu, Qihang Zhao, and Jason K Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint arXiv:2302.13939 , 2023.

Changze Lv, Yansen Wang, Dongqi Han, Xiaoqing Zheng, Xuanjing Huang, and Dongsheng Li. Efficient and effective time-series forecasting with spiking neural networks. In Forty-first International Conference on Machine Learning (ICML) , 2024.

Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. arXiv preprint arXiv:1707.01926 , 2017.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , pages 95-104, 2018.

Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL , 2005.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP , 2013.

Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs: An eventstream dataset for object classification. Frontiers in Neuroscience , 11, 2017.

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 , 2018.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics , 2019.

Zhaokun Zhou, Kaiwei Che, Wei Fang, Keyu Tian, Yuesheng Zhu, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer v2: Join the high accuracy club on imagenet with an snn ticket. arXiv preprint arXiv:2401.02020 , 2024.

Youngeun Kim, Hyoungseob Park, Abhishek Moitra, Abhiroop Bhattacharjee, Yeshwanth Venkatesha, and Priyadarshini Panda. Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks? In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 71-75. IEEE, 2022.

Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In European conference on computer vision , pages 388-404. Springer, 2020.

IM Comsa, K Potempa, L Versari, T Fischbacher, A Gesmundo, and J Alakuijala. Temporal coding in spiking neural networks with alpha synaptic function: Learning with backpropagation. IEEE Transactions on Neural Networks and Learning Systems , 33(10):5939-5952, 2022.

Young C Yoon. Lif and simplified srm neurons encode signals into spikes via a form of asynchronous pulse sigma-delta modulation. IEEE transactions on neural networks and learning systems , 28(5):1192-1205, 2016.

Han Zhao, Xu Yang, Cheng Deng, and Junchi Yan. Dynamic reactive spiking graph neural network. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 16970-16978, 2024.

Rafael Yuste, Jason N MacLean, Jeffrey Smith, and Anders Lansner. The cortex as a central pattern generator. Nature Reviews Neuroscience , 6(6):477-483, 2005.

Adam H Marblestone, Greg Wayne, and Konrad P Kording. Toward an integration of deep learning and neuroscience. Frontiers in computational neuroscience , 10:94, 2016.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.

Acknowledgments

The authors would like to thank the anonymous reviewers for their valuable comments. This work was supported partially by National Natural Science Foundation of China (No. 62076068).

Broader Impact

This work aims to advance the field of spiking neural networks (SNNs). Unlike artificial neural networks (ANNs) which have been applied widely in people's lives, SNNs are still undergoing fundamental research. We do not see any negative societal impacts of this work.

Reproducibility Statement

The authors have diligently worked to ensure the reproducibility of the empirical results presented in this paper. The datasets, experimental setups, evaluation metrics, and hyperparameters are thoroughly described in Appendices A and B. Furthermore, the source code for the proposed PE method has been available at https://github.com/microsoft/SeqSNN .

A Datasets

A.1 Time-series Forecasting

Detailed statistical characteristics and distribution ratios for each dataset are provided in the following:

A.2 Text Classification

Here are the datasets we used in text classification experiments:

MR : MR, which stands for Movie Review, is a dataset containing movie-review documents labeled based on their overall sentiment polarity (positive or negative) or subjective rating [35].

SST5 : SST5 includes 11 , 855 sentences from movie reviews for sentiment classification across 5 categories: very negative, negative, neutral, positive, and very positive [36].

SST2 : SST2 is the binary version of SST5 , containing only 2 classes: positive and negative.

Subj : The Subj dataset is designed to classify sentences as either subjective or objective * .

ChnSenti : ChnSenti consists of approximately 7 , 000 Chinese hotel reviews, each annotated with a positive or negative label † .

Waimai : This dataset contains around 12 , 000 Chinese user reviews from a food delivery platform, intended for binary sentiment classification (positive and negative) ‡ .

A.3 Image Classification

Here are the datasets we used in image classification experiments: CIFAR dataset comprises a collection of 60 , 000 images, partitioned into 50 , 000 training and 10 , 000 testing images, each with a

resolution of 32 × 32 pixels. The CIFAR10-DVS dataset represents a neuromorphic adaptation of this original set, where static images have been transformed to accommodate the recording capabilities of a Dynamic Vision Sensor (DVS) camera. This conversion results in a dataset consisting of 9 , 000 training samples and 1 , 000 test samples with 128 × 128 resolution.

B Experiment Settings

B.1 Time-series Forecasting

Metrices The metrics we used in time-series forecasting are the coefficient of determination (R 2 ) and the Root Relative Squared Error (RSE).

In these formulas, M symbolizes the size of the test sets, C denotes the number of channels, and L signifies the length of predictions. ¯ Y represents the average of Y m . The term Y m c,l refers to the l -th future value of the c -th variable for the m -th sample, while ¯ Y c,l indicates the mean of Y m c,l across all samples. The symbols ˆ Y m and ˆ Y m c,l are used to represent the ground truth values. Compared to Mean Squared Error (MSE) or Mean Absolute Error (MAE), these metrics offer greater resilience against the absolute values of the datasets, making them particularly useful in the time-series forecasting setting.

Model Architecture All SNNs take 4 time steps. For SpikeTCNs and SpikeRNNs, we follow the same settings as [32]. We construct all Spikformer as 2 blocks, setting the feature dimension as 256 , and the hidden feature dimension in FFN as 1024 . For CPG-PE settings, we set τ = 10000 . 0 , N = 20 , η = 1 , and v thres = 0 . 8 .

Training Hyper-parameters we set the training batch size as 64 and adopt Adam [48] optimizer with a cosine scheduler of learning rate 1 × 10 -4 . An early stopping strategy with a tolerance of 30 epochs is adopted. We conducted time-series forecasting experiments on 24G-V100 GPUs. On average, a single experiment takes about 1 hour under the settings above.

B.2 Text Classification

Model Achirecture All models are with 12 encoder blocks and 768 feature embedding dimension. It is important to note that the original implementation of [29] incorporates a layer normalization module that poses challenges to hardware compatibility. To address this, we have substituted layer normalization with batch normalization in our directly-trained Spikformer models for text classification tasks. For CPG-PE settings, we set τ = 10000 . 0 , N = 20 , η = 1 , and v thres = 0 . 8 .

Training Hyper-parameters We directly trained Spikformers with arctangent surrogate gradients on all datasets. We use the BERT-Tokenizer in Huggingface § to tokenize the sentences to token sequences. We pad all samples to the same sequence length of 256 . We conducted text classification experiments on 4 RTX-3090 GPUs, and set the batch size as 32 , optimizer as AdamW [49] with weight decay of 5 × 10 -3 , and set a cosine scheduler of starting learning rate of 5 × 10 -4 . What's more, in order to speed up the training stage, we adopt the automatic mixed precision training strategy. On average, a single experiment takes about 1 . 5 hours under the settings above.

B.3 Image Classification

Model Architecture For all Spikformer models, we standardized the configuration to include 4 time steps. Specifically, for the CIFAR10 and CIFAR100 datasets, the models were uniformized

with 4 encoder blocks and a feature embedding dimension of 384 . For the CIFAR10-DVS dataset, the models were adjusted to have 2 encoder blocks and a feature embedding dimension of 256 . For CPG-PE settings, we set τ = 10000 . 0 , N = 20 , η = 2 π , and v thres = 0 . 8 .

Training Hyper-parameters We honestly follow the experimental settings in [4], whose source code and configuration files are available at https://github.com/ZK-Zhou/spikformer . As the training epochs are quite big ( 300 epochs) in their settings, we choose to use one 80G-A100 GPU, and it takes about 3 hours to conduct a single experiment, on average.

B.4 Details about Positional Encoding Analysis

We conducted positional encoding analysis experiments on the CIFAR10-DVS dataset. For the original Spikformer with relative positional encoding (RPE) as described by [4], the input and output channels of Conv2d are both set to 384 . In our Spikformer with CPG-PE, the parameters are set to τ = 10000 . 0 and N = 20 . Given the time step T = 4 and the sequence length L = 160 for the image patches in CIFAR10-DVS samples, the total "length" T × L in CPG-PE is 640 . We then calculated the repetition rate of positions. The results showed that the repetition rate for RPE is 12 . 19% , whereas for CPG-PE, it is 0 . 00% .

C Implement CPG-PE with LIF Neurons

In this section, we demonstrate that CPG-PE is a hardware-friendly design. While implementing the sinusoidal potential on the neuromorphic chips is not challenging (e.g., by maintaining additional LC circuits), we show how a CPG-PE neuron can be physically implemented with only 2 LIF neurons defined by Equations (1) to (3) and thus introducing no extra efforts on chip designs.

A CPG-PE neuron, after discretization, can be viewed as an autonomic neuron that will emit a burst of K spikes after resting for R time steps. The key idea is to set two LIF neurons, namely the Emitter and the Resetter . The emitter will draw constant current from the source, and as soon as its membrane potential reaches the threshold after R time steps, it will start emitting spikes constantly until receiving the reset signal from the resetter. The resetter, which will remain at the resting potential until it receives signals from the emitter, will count the number of spikes and emit a reset signal (inhibition signal) to the emitter after receiving K spikes.

We first prove the following Lemma, which establishes the relationship between the start time of the first spike and a constant input current.

Lemma 1. Given an LIF neuron defined by Equations (1) to (3) with decay rate β and threshold U thr , starting with resting potential U (0) = 0 , if fed with the constant current I ( t ) = I c > 0 , the first spike will emit at:

Proof. By definition, before the time to emit the first spike, we have S ( t ) = 0 . Thus Equation (1) can be rewrite as:

Simplifying the recurrence relation, we can obtain:

The first spike is generated when U ( k ∆ t ) ≤ U thr , thus we have:

that is to say,

Now we can implement the CPG-PE with LIF neurons:

Theorem 1. Given 2 LIF neurons, the emitter and the resetter, with decay rate β , threshold U thr , and reset potential V reset , starting with resting potential U e (0) = U r (0) = 0 . If

then the system will have the period of T = ( R + K )∆ t , and ∀ i ∈ N ∩ [0 , R + K -1] , k ∈ N :

Proof. Assuming the first spike generated by the emitter emits at time step T 1 . For every 0 ≤ t < T 1 , we have:

Since the input current of the emitter is a constant, by Lemma 1, we immediately get:

Starting from time step R , let's assume the first spike generated by the resetter emits at time step T 2 . Then for every T 1 ≤ t < T 2 , we have:

Starting from T 1 , for the emitter, the input current allows a spike event for every time step. And the input current of the resetter is a constant. Again, by applying Lemma 1, we can get:

Now Consider the state at time step R + K :

This is the same as the membrane potential at time step 0. Therefore, the system will behave periodically with period T = ( R + K )∆ T .

Theorem 1 gives a possible CPG-PE design with 2 LIF neurons, with the emitter generating K consecutive spikes every R + K time steps. This demonstrates that incorporating CPG-PE into the current SNN architecture is completely bio-plausible and will not introduce any burden of redesigning hardware.

D Implement CPG-Linear

We have developed a simple modularization implementation to integrate our proposed CPG-PE with original linear layers, as depicted in Figure 3 (b). Consider the original linear layer's input and output dimensions as D in and D out , respectively. Our objective is to incorporate CPG-PE within this framework. Following the application of the CPG-PE module, the modified input X ′ is obtained. X is then input into Linear 1 and X ′ into Linear 2 , resulting in outputs X 1 and X 2 , respectively. Both

(b) Modularization as a CPG-Linear

CPG-Linear

𝑿

CPG-PE

Spiking Neuron Layer

𝑿′

Concatenation

𝑿

Element-wise Addition

𝑿′

Positional Embedding

𝑻

Time Steps in SNNs

𝑳

Sequence Length

Integrate CPG-PE

into Linear Layers

Original Linear

Extra

Linear

(a) Original Linear

𝑿

Original Linear

Linear

Linear 1 and Linear 2 maintain an output dimension of D out . The final step involves summing X 1 and X 2 to produce X 3 , which is subsequently processed through batch normalization ( BN ) and spike normalization ( SN ). We term this implementation as 'CPG-Linear' and formulize as follows:

where + denotes element-wise addition. This implementation described above is fundamentally identical to Figure 3, within the context of a single linear layer. However, the CPG-Linear can seamlessly replace any linear layer in SNNs.

E Results on ImageNet

We have conducted experiments with Spikformer without positional encoding (PE), Spikformer with relative positional encoding (RPE), and Spikformer with our proposed CPG-PE on the ImageNet dataset. The results are as follows:

Specifically, we set the depth to 8 and the dimension of representation to 384 . From the table, we can see that CPG-PE performs well on large-scale image datasets. We believe that the above results demonstrate the effectiveness of our proposed CPG-PE in positional encoding.

F Limitations and Future Works

In this section, we will discuss the limitations and future works of our paper.

F.1 Limitations

As mentioned in Section 3.3, our CPG-PE can not be directly applied to those SNNs where spike matrices do not have a 'sequence length' dimension. Our CPG-PE is optimized for processing sequential data, making it ideal for applications involving time series or natural language. This intrinsic design, however, does not naturally extend to image data, which typically benefits from direct convolutional operations that capture spatial relationships across the entire image dimensions-height and width. In contrast, CPG-PE requires the segmentation of images into patches, a method inspired by the Vision Transformer. This adaptation contrasts with approaches like the Convolutional 2D layer, which applies convolution operations directly across the height and width of an image without requiring segmentation into smaller, discrete patches. The necessity to adapt CPG-PE for image data through patching can introduce complexities and potential performance bottlenecks, as it may not effectively capture the continuous spatial relationships and local features in the image, which are crucial for tasks such as object recognition and scene understanding.

F.2 Future Works

To enhance the applicability of the CPG-PE model to a broader range of data types, especially image data, future research could focus on developing a hybrid model that integrates the strengths of CPG-PE with traditional convolutional layers. This integration could potentially allow the model to handle both sequential and spatial data efficiently without the need for extensive pre-processing or adaptation. Specifically, integrating direct convolution operations that work across the entire spatial dimensions of an image within the CPG-PE architecture could help preserve spatial relationships and improve feature extraction capabilities. Additionally, exploring the use of adaptive patch sizes or dynamically adjusting the patching mechanism based on the nature of the input data could also provide a more flexible and performance-optimized approach. These advancements would make the model more versatile and capable of tackling a wider array of tasks across different domains.

Additionally, considering that CPG-PE is an absolute positional encoding designed for SNNs, it could be beneficial to explore the potential of implementing learnable relative positional encodings in SNNs. Such encodings would need to be developed to meet specific criteria: they must maintain the spikeform characteristic essential to SNNs and ensure the uniqueness of each position's encoding. This approach could significantly enhance the model's ability to capture and utilize the temporal dynamics of input data more effectively, potentially leading to more nuanced and context-aware processing capabilities. Exploring adaptive patch sizes or dynamically adjusting the patching mechanism based on the nature of the input data could also provide a more flexible and performance-optimized approach. These advancements would make the model more versatile and capable of tackling a wider array of tasks across different domains.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clarified our claims in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations and future work in Appendix F.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided the full set of assumptions and a complete (and correct) proof in the Method Section.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have shown our experiment results in the Experiment Section. We have submitted our source code in Supplementary Material. We will upload our code and data to Github upon acceptance.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have submitted our source code in Supplementary Material. We will upload our code and data to Github upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have shown our experimental settings and implementation details in Appendices A and B respectively.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our reported results are all averaged over several random seeds. We have reported the error bar of the results in Table 2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided the compute resource in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed both potential positive societal impacts and negative societal impacts of the work in Broader Impact Section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The datasets we used in the paper are all public datasets. Please refer to Appendix A for details of datasets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new position encoding technique CPG-PE for SNNs, which is inspired by the central pattern generator in the human brain and improves the ability of SNNs to process sequence data. Experimental results show that CPG-PE outperforms traditional SNNs in multiple fields such as time series prediction, text classification and image classification.\n\nSTRENGTHS:\n1. The bio-inspired CPG-PE technique is proposed to enhance the sequence processing capability of SNNs.\n2. The effectiveness of CPG-PE is verified in multiple fields, including time series, text classification, and image classification.\n3. The performance of SNNs in various tasks is significantly improved, outperforming traditional models.\n4. CPG-PE is designed with compatibility with neuromorphic hardware in mind, facilitating deployment in practical applications.\n\nWEAKNESSES:\n1. No experiments were conducted on the large-scale image classification Imagenet dataset.\n2. Other encoding methods in SNN such as rate coding [1] and temporal coding [2;3], were not discussed.\n\n[1] Kim Y, Park H, Moitra A, et al. Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks?[C]//ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 71-75.\n\n[2] Han B, Roy K. Deep spiking neural network: Energy efficiency through time based coding[C]//European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 388-404.\n\n[3] Comsa I M, Potempa K, Versari L, et al. Temporal coding in spiking neural networks with alpha synaptic function[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 8529-8533.\n\nQUESTIONS:\n1. I suppose the traditional positional encoding in Spikformer can be seen as direct encoding in SNNs. In contrast, the CPG-PE proposed in this paper is more like a temporal encoding with dynamic expressions related to time series. Could the authors discuss the differences between other encoding schemes [1;2;3] and positional encoding in SNNs within the related work?\n\n2. Can the authors provide experimental results of CPG-PE on the ImageNet image classification dataset?\n3. Some tyops: Formulas 4, 5, 6, 7, 8, and 10 are missing commas. There should be a comma after each formula.\n\nI would be pleased to raise the score if the authors address my concerns.\n\n[1] Kim Y, Park H, Moitra A, et al. Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks?[C]//ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 71-75.\n\n[2] Han B, Roy K. Deep spiking neural network: Energy efficiency through time based coding[C]//European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 388-404.\n\n[3] Comsa I M, Potempa K, Versari L, et al. Temporal coding in spiking neural networks with alpha synaptic function[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 8529-8533.\n\nLIMITATIONS:\nThe author has discussed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-14"}

comments: SUMMARY:
This paper introduces a novel positional encoding method for SNNs called CPG-PE, inspired by central pattern generators (CPGs) in biological neural systems. The authors demonstrate both theoretically and empirically that CPG-PE can effectively capture positional information in sequential data while maintaining the spike-based nature of SNNs. The approach is evaluated on a range of tasks including time-series forecasting, text classification, and image classification.

STRENGTHS:
1. Strong theoretical foundation: The authors mathematically demonstrate how CPG-PE relates to conventional sinusoidal positional encoding used in transformers.
2. Comprehensive empirical evaluation across time-series forecasting, text classification, and image classification tasks shows consistent performance improvements when incorporating CPG-PE.
3. The method is biologically plausible and potentially compatible with neuromorphic hardware, as it can be implemented using leaky integrate-and-fire neurons.
4. Well-structured paper with clear motivation, methodology, and insightful analysis of CPG-PE properties and their relationship to biological CPGs.

WEAKNESSES:
1. Typographical error on line 158 where X is incorrectly specified as belonging to the real number domain instead of the binary domain {0,1} for spike data.
2. The authors should clarify the similarities and differences between the decay mechanism in CPG-PE and that of LIF neurons.

QUESTIONS:
See my weakness part.

LIMITATIONS:
The authors have addressed limitations and future work in Appendix E, which is commendable. They discuss the challenges of applying CPG-PE to non-sequential data like images and propose potential solutions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
The lack of an effective and hardware-efficient spike-frm position encoding strategy in  SNNs has been a consistent motivation for this study.
Drawing inspiration from the central pattern generators (CPGs) in the human brain, which produce rhythmic patterned outputs without requiring rhythmic inputs, this work proposes a novel PE technique for SNNs, termed CPG-PE. 
Extensive experiments across various domains show the superior performance with CPG-PE.

STRENGTHS:
1.To the best of my knowledge, this is the first work on position encoding in SNNs, laying the foundation for efficient sequence modeling in SNNs.

2.The approach utilizes the coupling of multi-neuron pulse signals as position encoding, which is innovative.

3.The authors demonstrated the effectiveness of this method across various tasks.

4.The spike-position encoding generating method through mutual inhibition between two groups of spiking neurons is brain-inspired and hardware-friendly.

WEAKNESSES:
1.This type of positional encoding, through aggregation, introduces a small number of additional parameters and computational overhead. Please provide ablation experiments demonstrating that the performance improvement is not solely due to these factors.
2.I suggest the authors include results on ImageNet to demonstrate the effectiveness on large-scale datasets.

QUESTIONS:
1.In ANNs, positional encoding is typically added to features. Please analyze the similarities and differences of this category aggregation-based positional encoding compared to ANNs.

LIMITATIONS:
I believe the authors' discussion on limitations is comprehensive.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces central pattern generators (CPGs) from neuroscience into the SNN framework as a novel method for position encoding. Through mathematical derivation, it is proven that the existing abstract PE methods in transformers are actually a particular solution for a specific type of CPG. The effectiveness of CPG is validated through experiments across several domain benchmarks.\n\nSTRENGTHS:\n1.\tThis article connects existing abstract Positional Encoding (PE) methods in transformers with Central Pattern Generators (CPGs) in the human brain through mathematical derivation, showing that the former can be viewed as a specific mathematical solution to the membrane potential dynamics of the latter. This presents an interesting viewpoint.\n\n2.\tThe paper is well-organized and clearly written, offering high readability. Readers can effortlessly grasp the authors' intentions, supported by both the textual explanations and accompanying illustrations.\n\nWEAKNESSES:\n1.\tThe authors have not convinced me why the problem addressed in this paper is very important, i.e., why the existing PE methods in SNNs are such a big problem that they need to be improved by CPG.\n\n\n2.\tThe method proposed in the paper is simple and does not provide enough inspiring insights; the contribution is relatively limited.\n\n3.\tThe implementation of CPG-PE on hardware involves coupled nonlinear oscillators that require frequent updates of neuron membrane potentials, entailing floating-point computations and memory read-write operations, which result in additional energy expenditures. The paper should scrutinize and analyze whether the performance enhancements afforded by this encoding method justify the additional energy costs. This trade-off demands a detailed examination to assess its viability in practical applications.\n\n4.\tIn Sec3.1, why is \"F(x)=b<=0, H(y)=d<=0\" followed after “...gain membrane voltage with constant speed” instead of “F(x)=b>0, H(y)=d>0”?\n\n5.\tIn Sec4.2, does the CPG-Full method replace all linear layers in the model with CPG-Linear layers? Why is its performance not as good as that of CPG-PE? Can you provide further analysis and explanation?\n\nQUESTIONS:\nPlease refer to weakness 3, 4, 5\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}