SPLR: A Spiking Neural Network for Long-Range Temporal Dependency Learning

Authors: Biswadeep Chakraborty, Saibal Mukhopadhyay
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: spiking neural networks, long range dependencies, event data modelling, hippo matrix, state space models

Abstract:
Spiking Neural Networks (SNNs) offer an efficient framework for processing event-driven data due to their sparse, spike-based communication, making them ideal for real-time tasks. However, their inability to capture long-range dependencies limits their effectiveness in complex temporal modeling. To address this challenge, we present a **SPLR (SPiking Network for Learning Long-range Relations)**, a novel architecture designed to overcome these limitations. The core contribution of SPLR is the **Spike-Aware HiPPO (SA-HiPPO)** mechanism, which adapts the HiPPO framework for discrete, spike-driven inputs, enabling efficient long-range memory retention in event-driven systems. Additionally, SPLR includes a convolutional layer that integrates state-space dynamics to enhance feature extraction while preserving the efficiency of sparse, asynchronous processing. Together, these innovations enable SPLR to model both short- and long-term dependencies effectively, outperforming prior methods on various event-based datasets. Experimental results demonstrate that SPLR achieves superior performance in tasks requiring fine-grained temporal dynamics and long-range memory, establishing it as a scalable and efficient solution for real-time applications such as event-based vision and sensor fusion in neuromorphic computing.

Decision: reject
Reviewer scores: 5, 5, 8, 3, 5

Reviews and discussion:
<thread>
creation date:
1729485684906

thread_type:
official_review

<thread_object>
creator: reviewer_tvp3

<content>
summary:
This paper integrates state-space models (SSMs) and spiking neural networks (SNNs), and proposes the Spiking Network for Learning Long-Range Relations (SPLR) to enhance the ability of SNNs to capture long-range dependencies. Theoretical proofs and experimental results support the performance advantages of SPLR.

strengths:
1. The integration of SA-HiPPO and SPLR convolution enhances the model's ability to model long-range dependencies.
2. The SPLR model designed in this paper introduces a dendrite-based pooling layer, which further improves the performance using the DH-LIF neuron model.
3. The theoretical and experimental results in this paper confirm the effectiveness of the proposed method.

weaknesses:
1. The core innovation of this paper is the SPLR convolution layer to enhance the long-range temporal modeling capability of SNNs. However, while this improves the performance of the SNN, the integration of the SSM and the SNN requires significant computational overhead, which counteracts the power consumption advantage of the SNN. In addition, as the authors mention SPLR is difficult to implement in hardware, making the significance of this work seem small.

2. Can the authors clarify the key differences between the DH-LIF model in this paper and the one presented in [1]? How does the DH-LIF used in the Dendrite Attention Layer relate to attention? Is the author's claim of dendrite-based spatio-temporal pooling just spatial pooling after the output of DH-LIF? Where is there temporal pooling?

3. I suggest that the authors compare their method with other optimized spiking neural networks capable of long-range temporal modeling, such as [2,3,4].


[1] Zheng H, Zheng Z, Hu R, et al. Temporal dendritic heterogeneity incorporated with spiking neural networks for learning multi-timescale dynamics. Nature Communications, 2024.

[2] Wang L, Yu Z. Autaptic synaptic circuit enhances spatio-temporal predictive learning of spiking neural networks. ICML, 2024.

[3] Zhang S, Yang Q, Ma C, et al. Tc-lif: A two-compartment spiking neuron model for long-term sequential modelling. AAAI, 2024.

[4] Shen S, Zhao D, Shen G, et al. TIM: An efficient temporal interaction module for spiking transformer. IJCAI, 2024.

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730542224998

thread_type:
official_review

<thread_object>
creator: reviewer_ticg

<content>
summary:
This work proposes a spike SSM method named Spiking Network for Learning Long-Range Relations (SPLR). The proposed SPLR convolutional layer leverages state-space dynamics to enhance feature extraction while retaining the efficiency of sparse, event-based
processing, and incorporates a Spike-Aware HiPPO (SA-HiPPO) matrix that allows SPLR to effectively maintain long-range memory by adapting the HiPPO framework for discrete, spike-driven inputs. The authors tested their method on several datasets, such as Celex HAR, DVS128 Gesture, Sequential CIFAR-10/100

strengths:
1. This work explores the combination of Spiking Neural Networks (SNN) and State Space Models (SSM), which is an interesting direction. Using state-space methods to improve SNNs' ability to model long-term dependencies holds promise.
2. The experimental comparisons in this work are extensive.

weaknesses:
This work requires comprehensive improvements, with the main weaknesses outlined as follows.

1. Writing: The writing in this work requires careful and comprehensive improvement, covering the overall organization of the paper, paragraph structure, and numerous details that need refinement. (1) The authors placed the related work section in the supplementary materials and omitted citations to many key works, which can confuse readers. For instance, the authors did not cite relevant papers when HIPPO was first mentioned (in fact, there are almost no citations in paragraphs 3 and 4 of the introduction). (2) The methodology section is not clearly explained. What type of spiking neurons does this work use? How does SPLR integrate with spiking neurons? The authors repeat content introduced in the main text within the supplementary materials. Lines 147-150 reiterate the significance of SPLR, but readers are likely more interested in the methodological details and the rationale behind the proposed approach’s significance. Unfortunately, these critical details are missing. (3) In the theoretical discussion, the authors present several theorems but do not clarify why these are necessary. While sparse properties and FLOPS reduction are mentioned, the evaluation details are not provided. 
(4) The supplementary materials contain excessive repetition and are overly lengthy, making it difficult for readers to stay engaged. (5) Overuse of Abbreviations: The excessive use of abbreviations makes the paper difficult to follow. For instance, I cannot understand why "Spiking Network for Learning Long-Range Relations" is abbreviated as "SPLR"—what does "P" represent here? Is there a difference between "Spiking Network" and "Spiking Neural Networks (SNNs)"? Additionally, what does HIPPO stand for? Shouldn’t the authors explain these abbreviations first? (6) Overstatements: The paper is filled with terms like "spike-driven," "asynchronous," and "real-time." As I understand it, "spike-driven" implies a purely additive network[2], yet the pink section in Figure 1 seems unable to achieve this. Regarding "asynchronous," the authors’ explanation in lines 92-95 is too brief, making it difficult to discern what kind of preprocessing the network applies to the data.

2. Motivation. The authors repeatedly state that the proposed SPLE can address the challenge of modeling both short- and long-term dependencies in SNNs. However, they fail to analyze why SNNs have limitations in this area and why their proposed method can solve this issue. For instance, this is mentioned in lines 58-67, 147-149, and 1846-1850 without providing the necessary analysis.

3. Innovation.The originality of this work is limited. The dendrite modeling directly uses DH-LIF, and the SSM modeling is simply an SNN combined with HIPPO. I did not observe any standout contributions in this approach.

4. Experiments. The datasets chosen by the authors, DVS128 Gesture and Sequential CIFAR-10/100, do not effectively test the model's ability to handle long-range dependencies. The authors could consider more challenging datasets, such as LRA. Additionally, the fact that no one in the SNN field has addressed the Celex HAR dataset does not imply that SNNs cannot handle it. The authors have not even provided a complete description of the size and scope of the Celex HAR dataset. If the authors aim to compare with other SNNs on challenging DVS datasets, they might try HAR-DVS[1]. Furthermore, the authors overlook comparisons with many recent SOTA methods on the Gesture dataset, where SNN performance has already surpassed 99.5%[2].

---
[1] Hardvs: Revisiting human activity recognition with dynamic vision sensors. In AAAI 2024.
[2] Spike-driven transformer. In NeurIPS 2023.

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714283802

thread_type:
official_review

<thread_object>
creator: reviewer_lqnp

<content>
summary:
This paper presents a Spiking Network for Learning Long-Range Relations (SPLR). The proposed SPLR model comprises the dendrite attention layer, the Spike-Aware HiPPO (SA-HiPPO) layer, and the SPLR convolution layer. These modules enhance the long-range temporal dependency learning capability of SPLR. Experimental results demonstrate that SPLR outperforms prior methods in tasks requiring both fine-grained temporal dynamics and the retention of long-range dependencies.

strengths:
1. This paper is well-written and technically solid. Each module in SPLR is introduced in detail and highlighted in different colors. This paper presents a detailed theoretical analysis, including the long-range dependency capability and stability of SPLR.
2. The proposed SPLR model achieves competitive accuracy with less computational overhead than other state-of-the-art models on the Celex-HAR dataset.

weaknesses:
The proposed SPLR model incorporates several non-spike operations, including the NPLR decomposition and FFT convolution. It makes SPLR a hybrid architecture instead of a pure spiking neural network. The hybrid nature may compromise its hardware compatibility and make it difficult to deploy on neuromorphic hardware.

questions:
1. This paper only compares FLOPs vs. accuracy between the proposed SPLR and other models. Does SPLR have an advantage over other methods in terms of inference latency?
2. The ablation studies only examine the effects of removing the dendrite attention layer and replacing SA-HiPPO with LIF. What if we replace NPLR decomposition and FFT convolution with standard convolution?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717294986

thread_type:
official_review

<thread_object>
creator: reviewer_arz5

<content>
summary:
This work introduces SPLR (Spiking Network for Learning Long-Range Relations), designed to efficiently capture long-range temporal dependencies while maintaining the hallmark efficiency of spike-driven architectures. SPLR integrates a state-space convolutional layer and a Spike-Aware HiPPO (SA-HiPPO) layer, addressing the limitations of conventional SNNs in complex temporal modeling. The SPLR convolutional layer leverages state-space dynamics to enhance feature extraction, capturing spatial and temporal complexities in event-driven data while preserving the efficiency of sparse spike-driven processing. The SA-HiPPO layer adapts the HiPPO framework to spike-based formats, enabling efficient long-term memory retention. Through dendrite-based spatiotemporal pooling and FFT-based convolution techniques, SPLR demonstrates scalability when processing high-resolution event streams and outperforms traditional methods across various event-driven tasks.

strengths:
1. The authors propose SPLR which effectively captures long-range temporal dependencies, addressing limitations in traditional SNNs and enhancing temporal modeling capabilities for complex event-driven tasks.
2. The experiments show good results. SPLR achieves both computational efficiency and scalability.

weaknesses:
The primary weakness of this paper lies in its writing, which significantly hinders clarity and understanding. A reorganization is recommended to improve readability and logical flow.

1. Writing and Structure. For instance, Section 2 presents the SPLR components sequentially but lacks an overview that connects each part to the overall model structure, making it difficult for readers to understand how the parts interact. Section 3 is dense with theoretical content and proofs but does not clearly convey the main ideas, making it hard to follow the section’s intended focus.

2. Lack of Citations. The paper frequently omits citations in crucial areas. For example, although modifications to the HiPPO framework are proposed, no supporting references are provided. Furthermore, DH-LIF is introduced without citation, and the reference for this component is missing from the bibliography, weakening the academic rigor of the paper.

3. Confusions and Errors. There are several errors and confusions throughout the paper, such as the incorrect abbreviation of the Spiking State-Space Model as SPLR in line 855. Such errors further impact the readability and precision of the work.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730853646825

thread_type:
official_review

<thread_object>
creator: reviewer_y4sn

<content>
summary:
The manuscript introduces SPLR, a spiking neural network model designed to capture long-range temporal relationships by integrating state-space dynamics with spiking neuron models and augmenting the HiPPO framework to handle spike-driven inputs. The proposed model reportedly achieves high performance comparable to other models on event-based datasets.

strengths:
The manuscript presents an innovative approach by augmenting spiking dynamics with state-space model dynamics, potentially enabling spiking models to tackle more challenging tasks that require capturing long-range temporal dependencies. This direction could be of significant interest in broadening the applications of spiking models in complex temporal tasks.

weaknesses:
While the proposed method is intriguing and addresses the relevant challenge of enabling spiking models to capture long-term dependencies, the manuscript has several critical weaknesses. 

Firstly, the presentation lacks clarity, making it difficult to fully grasp how the method works, interpret the experimental results, or potentially reproduce the findings. Essential details expected in a research paper, such as a discussion of related works, are missing (e.g., [1]). Additionally, fundamental concepts necessary to understand the work are not well-introduced; although state-space models (SSMs) have gained popularity recently, they are not widely understood in machine learning, so a brief overview would be beneficial.

The manuscript also omits essential citations, including the original HiPPO framework, which is central to this work, and does not offer a proper explanation of how it functions. The equations are unclear; while convolutions are frequently mentioned, no equations illustrate how or where convolutions are applied. Variable definitions are sometimes confusing or incomplete; for instance, on line 187, $\Delta t$ is described as the time difference between spikes $i$ and $j$, but it is unclear what $i$ and $j$ refer to in the context of the matrix $F_{ij}$, as it operates over the hidden state rather than directly on spikes.

Regarding the experiments, the manuscript lacks details about the setup, hindering the interpretability of the results. For example, it’s unclear what “Sequential CIFAR-10” entails, such as the sequence length or frame generation process. Similarly, for the DVS Gesture dataset, it's ambiguous whether the processing was done for independent events or if events were accumulated into event frames.

[1] Stan, MI., Rhodes, O. Learning long sequences in spiking neural networks. Sci Rep 14, 21957 (2024). https://doi.org/10.1038/s41598-024-71678-8

questions:
- How does the dendritic attention layer differ from a current-based (CUBA) leaky integrate-and-fire (LIF) neuron model?
- How are spikes produced between the SPLR convolution layers?
- How are convolutions applied within the proposed model?
- In equation (1), is the variable $u(t)$ a binary vector representing input spikes?
- How does the inclusion of a decay matrix in the HiPPO framework enhance memory retention?
- Could you clarify the setup for the Sequential CIFAR-10 and CIFAR-100 tasks? How are frames sequenced? Similarly, could you elaborate on the experimental setup for the other datasets?
- For clarification, could you specify what spikes $i$ and $j$ refer to in line 187?
- Is the manuscript proposing a new type of spiking neuron, or an entire network architecture?
- Since the manuscript emphasizes improving SNNs' capacity to handle long-term dependencies, could you elaborate on why simple LIF models face challenges with this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732077039931

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Changes

comment:
We thank the reviewers for their valuable feedback and have made significant revisions to the manuscript to address the major concerns raised. Below, we outline the key changes and improvements:

1. **Improved Clarity, Organization, and Methodological Depth**

    - **Expanded Model Explanations:** Enhanced the description of the SPLR model in Section 2, providing detailed explanations of key components, including the SA-HiPPO layer, NPLR decomposition, and FFT-based convolutions.

    - **Revised Methodology Section:** Improved the logical flow of the Methods section by introducing an overview that connects all components of SPLR to the overall model structure.

   - **Intuitive Theoretical Explanations:** Added intuitive explanations for key theorems in Section 3 to clarify their relevance to SPLR's design and improve readability.

    - **Supplementary Material Enhancements:** Updated the supplementary sections to reduce redundancy, include pseudocode for reproducibility, and detail experimental setups, dataset processing pipelines, and hyperparameter configurations.

   - **Citations and Related Work:** Added citations for foundational works (e.g., the original HiPPO framework and DH-LIF model) and expanded the Related Work section in the supplementary material. Comparisons now include:
        - Spike-Driven Transformer (Figure 3a), STC-LIF for DVS Gesture-128 (Figure 4c), TCLIF in SHD and SSC (Figures 4a, 4b), and TIM in SHD (Figure 4a).


2. **New Experiments, Results, and Comparisons**

     - **New Benchmarks:** Evaluated SPLR on the Long Range Arena (LRA) benchmark (Table 1) to demonstrate its ability to handle long-range temporal dependencies.

    - **HAR-DVS Results:** Included HAR-DVS results, validating SPLR’s capability to process high-resolution event-based data and demonstrating its robustness (Figure 2).
    - **Ablation Studies:** Added new ablation studies to evaluate the impact of replacing NPLR decomposition and FFT-based convolution with standard convolution.
    - **Highlighted Results and Trade-offs:**
      - **Tables:** Added comparisons of SPLR's performance against recent state-of-the-art methods on DVS Gesture, HAR-DVS, and LRA datasets, emphasizing FLOPs vs. accuracy trade-offs.
      - **Figures:** Revised Figure 3 to include methods exceeding 99% accuracy on DVS Gesture for proper contextualization, and added new visualizations summarizing SPLR’s performance on HAR-DVS and Celex-HAR datasets.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces SPLR, a Spiking Network for Learning Long-Range Relations. The SPLR model consists of three main components: the Dendrite Attention Layer, the Spatial Pooling Layer, and the SPLR Convolution Layer. The proposed SPLR Convolution Layer combines SA-HiPPO, FFT, and NPLR to enhance SPLR's capability to learn long-range temporal dependencies. Experimental results show that SPLR outperforms previous methods in tasks that require both fine-grained temporal dynamics and the ability to retain long-range dependencies.

After the rebuttal period, 5 reviewers rate 3, 5, 5, 5, 8, respectively. Most of the reviewers agree that SPLR outperforms prior methods in tasks requiring both fine-grained temporal dynamics and the retention of long-range dependencies, which is the strength of SPLR. However, there are some criticisms. Reviewer Y4sN believes a more detailed discussion of the SA-HiPPO should be included. Reviewer tvp3 finds it difficult to identify the core novelty and strengths of this paper and the writing logic of the paper needs improvement. I think these criticisms are justified. I suggest that the authors further clarify the proposed mechanisms and highlight the key contributions.

As for the concerns regarding Reviewer TiCG's evaluation, I agree with the author's concern that Reviewer TiCG is too harsh in tone and too arbitrary in judgment. I have taken these issues into consideration. However, some of Reviewer TiCG's criticisms are still justified. As Reviewer TiCG states, "'Written a lot' does not mean 'well written'". I agree that the writing in this work is redundant. The main text should highlight the contributions and innovations of the paper rather than listing all the details of the proposed methodology. For example, I suggest the authors put the "Input Representation", "Normalization", and "Readout Layer" in Sec. 3 into the supplementary, since these are not the key contributions of this paper. However, the related work section should be placed in the main text as it helps to highlight the innovations and contributions of this paper. I believe it is not impractical to seek more details with limited pages; the key is to highlight the main points.

Overall, I believe this paper would benefit from further revision. Therefore, the final decision is to reject this paper.