Title: FLAME: Fast Long-context Adaptive Memory for Event-based Vision
Venue: NeurIPS 2025
Accepted: yes
Keywords: state space models

event based vision

Adaptive Memory

sequence modeling
Abstract: We propose Fast Long-range Adaptive Memory for Event (FLAME), a novel scalable architecture that combines neuro-inspired feature extraction with robust structured sequence modeling
to efficiently process asynchronous and sparse event camera data. As a departure from conventional input encoding methods, FLAME presents Event Attention Layer, a novel feature extractor that leverages neuromorphic dynamics (Leaky Integrate-and-Fire (LIF)) to directly capture multi-timescale features from event streams. The feature extractor is integrates with a structured state-space model with a novel Event-Aware HiPPO (EA-HiPPO) mechanism that dynamically adapts memory retention based on inter-event intervals to understand relationship across varying temporal scales and event sequences. A Normal Plus Low Rank (NPLR) decomposition reduces the computational complexity of state update from $\mathcal{O}(N^2)$ to $\mathcal{O}(Nr)$, where $N$ represents the dimension of the core state vector and $r$ is the rank of a low-rank component (with $r \ll N$). FLAME demonstrates state-of-the-art accuracy for event-by-event processing on complex event camera datasets.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a SSM-based framework for event-based processing applied to vision tasks. It introduces 3 main contributions:\n\n-\tAn event attention layer, made with multi-timescale LIF neurons ;\n-\tAn event-aware HIPPO, with a memory retention moduled with inter-events intervals ;\n-\tA further optimization of the HIPPO with Normal-Plus-Low Rank (NPLR) decomposition.\n\nThe paper exhibits impressive results over the state-of-the-art notably by looking at the accurary vs computation cost ratio.\n\nSTRENGTHS AND WEAKNESSES:\nWhile the paper is well and extensively positionned against the state-of-the-art, both in methodology and in results, I found several issues about the methodology presented and its implementation:\n\n-\tThe connectivity of the event attention layer is not clearly explained, nor how its learnable timing factor are learned. It is therefore not clear as well how the spatial pooling works, as it is defined relative to the input but applied after the event attention layer. Furthermore, I could not see where spatial pooling is actually implemented in the FLAMES_model.py available in the supplementary materiel package.\n-\tIt is not clear what brights the coupling strength parameters (of branch to soma), as it seems it could be fused in the weights of the branches current ;\n-\tNot much detail is given about the NPLR decomposition used: it is not explained how it is obtained and with what parameter (notably rank). Furthermore, I failed to see any implementation of this mechanism in the provided FLAMES_model.py. The NPLR decomposition is presented as being fundamental to the computation efficiency of the method, yet no impact analysis or ablation study is provided (a single number is given in the supplementary material l.208, without any detail on the setup of the NPLR).\n-\tIt is not specified how the HIPPO A matrix is initialized and learned such that the condition on its eigen values (<0) remains true.\n\nHowever, a lot of extra information is provided through the supplementary materiel, which includes detailed approximation and numerical stability proofs, as well as more details on the implemented method. This is definitely a strong point.\n\nQUESTIONS:\nMy main question is about the provided FLAMES_model.py in the supplementary material: this implementation does not reflect at all everything described in the paper, yet seems to be provided as a way to reproduce the experiments. Is it on purpose? Do you plan to release a complete open source version of your framework? As is, this script only cast doubts about the implementation of your method, unfortunately.\n\nDespite the extensive supplementary material, I believe some critical information on how to choose or learn hyperparameters is missing, making results reproductability potentially very difficult. I would have appreciated a section dedicated to the training of the model.\n\nAlso, I would very much appreciate if the issues mentioned in the previous section would be answered as well.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n1. The paper focuses on a basis and important topic for event-bases vision. The motivation is good.\n2. The proposed FLAME is technically sound. The methods seems effectively combines the event modality and the SSM structure. From the results, the FLAME shows its powerful  capability on balancing the compuatational efficiency and the task performance. There are plenty of theoretical analysis for proving the feasibility of the method.\n3. The paper presents good writing and formatting.\n4. The experiments are thorough and effective.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper focuses on a basis and important topic for event-bases vision. The motivation is good.\n2. The proposed FLAME is technically sound. The methods seems effectively combines the event modality and the SSM structure. From the results, the FLAME shows its powerful  capability on balancing the compuatational efficiency and the task performance. There are plenty of theoretical analysis for proving the feasibility of the method.\n3. The paper presents good writing and formatting.\n4. The experiments are thorough and effective. \n\nWeakness:\n1. The length of each section and the formatting of some experimental results can be refined. For example, the section 4 would benefit from tighter editing, and some important results of ablation studies should be moved to the main manuscripts.\n2. Using LIF neurons as processing unit in the event attention layers raises my doubts. the leaky neurons is more difficult for model training compared with IF neurons. Besides, considering that the event data are sparse, especially under in  small motion environments or dark environments, the LIF neuron may leads to stronger sparsity. \n3. It seems that the paper lacks results of CIFAR-100 dataset. If the results are presented, please highlight the results. The FLAME presents its ability in some common datasets. I am wondering if the FLAME can be scaled to more difficult tasks like object detection/trakcing, motion estimation, depth estimation.\n\nQUESTIONS:\nAs listed in the \"Strengths and Weaknesses\".\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FLAME, as claimed, a novel, event-by-event neural architecture that combines neuromorphic feature extraction with efficient structured state-space models to handle asynchronous, sparse data from event cameras. As claimed, the core contributions include:\n- An event attention layer leveraging leaky integrate-and-fire dynamics to extract multi-timescale features from raw event streams.\n- An event-aware HiPPO state-space core, which dynamically modulates memory based on inter-event intervals.\n- Integration of normal plus low-rank decomposition to reduce computation from $O(N^2)$ to $O(Nr)$ where $r<<N$.\n\nIt seems like FLAME achieves state-of-the-art accuracy across multiple event datasets while maintaining low latency and computational cost, demonstrating its efficiency for real-time event vision applications.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- As claimed, the FLAME allows for event-by-event processing with a lower inference latency; the proposed method is promising in low-latency applications.\n- The proposed event attention layer could be a valuable contribution in this research direction, given its ability to capture features over variable timescales using LIF dynamics, while the EA-HiPPO mechanism adapts memory retention to event sparsity, offering dynamic temporal scaling (with achieved efficiency via NPLR).\n- The authors provide clear theoretical guarantees (and proofs) on computational complexity, memory retention over long sequences, and numerical stability of FLAME while providing strong empirical results. \n\nWeaknesses\n- Even though the event attention layer can take event-by-event, to pass a meaningful extraction, the spatial pooling layer (i.e., the second step) needs an event map, which is hard to construct through one event (or even several events) at least at the first instance (after the first construction, the map can be updated by event-by-event; if this is method, I feel like most of the event-by-event processing is redundant due to lesser scene change over one event). Therefore, as per my understanding, from this step onwards, the claim for event-by-event should be limited.\n- I am not fully convinced about the assumption presented in lines 158 to 160 (\"... preserving crucial spatio-temporal details that might be lost if pooling raw, unprocessed input\"). As per this reviewer's experience, even in the presence of noise, the raw input presents more information on spatio-temporal distribution and trends than any other efficient or accurate representation/encoding. Additionally, what will happen if the spatial pooling layer is omitted in the pipeline (i.e, will the results be improved or decreased) or a hierarchical pooling strategy is introduced? Is the pooling introduced solely to reduce the computational complexity?\n-   Given that a number of state (transition) matrices need to be stored for capturing (recent) history context (even with reduced complexity via NPLR decomposition; since $N$ is typically a large number if the intended history window is significant -- which it needs to be, to capture long-range dependancies) , I feel like FLAME would leave a considerable memory footprint than other SNN methods. This needs to be explored and reported in the paper as well. Additionally, what will happen if NPLR is omitted?\n- Even though FLAME works comparatively with the action recognition tasks, it seems to be performing poorly on classification, which may be due to its higher dependence on long-range temporal relationships (which are not trivial in classification tasks) and lesser absorption of spatial details. Authors may need to explicitly address this observation. \n- Ablations can be further expanded to explore different configurations of the decay modulation function \n$F(\\Delta t)$ or its learned parameters. \n- I believe that the paper needs a proper restructuring before getting published, if accepted, since most of the results are referred to the appendix, and in general, the paper, in its current form, seems to be imbalanced. Additionally, the readability of the paper should be improved as well. \n- It seems like the paper interchangeably uses both the terms \"FLAME\" and \"FLAMES\" on numerous occasions (as an example, Fig. 1, line 76, 83, 89, etc.) to refer to the same proposed method, which needs to be rectified. \n- It is a requirement to complete the NeurIPS checklist with proper justifications, which this paper does not comply with. \n- Minor concerns:\n  - Figures, especially Fig. 2, need to be improved since the y-axis labels and markers are cropped. \n  - Figure and text references are missing: as an example, lines 282, 404, etc. in the appendix\n\nQUESTIONS:\nSee the \"weaknesses\" section.\n\nLIMITATIONS:\nEven though the authors have addressed some limitations as future work in section 6 of the paper, I feel like the following should also be acknowledged. \n- Even though the latency on GPUs is better, it is to be acknowledged that it is even beyond real-time implementation in CPUs (Table 8 in the appendix), let alone edge devices. \n- The dependence on event-by-event processing may expose the model to overly depend on noise, which would ultimately degrade the downstream performance.\n- The current framework lacks mechanisms (or adaptations) for on-device continual learning, which can be crucial for many real-world applications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FLAME, a novel architecture for event-based vision that combines neuromorphic feature extraction with adaptive state-space modeling. It introduces an Event Attention Layer based on multi-timescale LIF dynamics and an Event-Aware HiPPO mechanism to capture long-range temporal dependencies efficiently. FLAME processes input events asynchronously and achieves competitive accuracy on several event-based benchmarks with reduced computational cost.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proposed module is built on a solid mathematical model, complete with detailed derivations that lend it strong credibility.\n- The new method achieves a dramatic speed-up while maintaining accuracy at a level comparable to existing approaches.\n\nWeakness\n- The description of the Event Attention Layer is too vague. It should be expanded to explain exactly how it enhances the raw event stream, i.e., which features it extracts and what benefits these sparse, multi-scale representations provide over the original signal.\n- The evaluation does not include a thorough comparison with leading baselines on some datasets such as EventMamba.\n\nQUESTIONS:\n- I suggest a deeper discussion of the physical meaning of the operation of Event Attention Layer, and it would enhance the readability of the paper to make corresponding visualization of the enhanced feature. It should clearly discussed which aspects of the event stream the resulting sparse features enhance, and whether this process effectively amounts to a form of noise filtering.\n- The current design of the Event Attention Layer appears to respond to both long-interval and short-interval events. Would this lead to cross-talk between the two?\n- Explain why EventMamba outperforms the proposed method on small datasets but becomes less efficient on high-resolution datasets.\n- On the DVS Gesture dataset, the proposed method underperforms EventMamba, and this discrepancy should be discussed in depth. Additionally, the paper omits EventMamba’s results on several datasets and these comparisons should be included to complete the experiment part.\n- The arrows in Fig. 1(b) are vague, I suggest revision.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}