Title: PASS: Path-selective State Space Model for Event-based Recognition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Event Camera

Object Recognition

Arbitrary Temporal Frequency
Abstract: Event cameras are bio-inspired sensors that capture intensity changes asynchronously with distinct advantages, such as high temporal resolution. Existing methods for event-based object/action recognition predominantly sample and convert event representation at every fixed temporal interval (or frequency). However, they are constrained to processing a limited number of event lengths and show poor frequency generalization, thus not fully leveraging the event's high temporal resolution. In this paper, we present our PASS framework, exhibiting superior capacity for spatiotemporal event modeling towards a larger number of event lengths and generalization across varying inference temporal frequencies. Our key insight is to learn adaptively encoded event features via the state space models (SSMs), whose linear complexity and generalization on input frequency make them ideal for processing high temporal resolution events.  Specifically, we propose a Path-selective Event Aggregation and Scan (PEAS) module to encode events into features with fixed dimensions by adaptively scanning and selecting aggregated event presentation. On top of it, we introduce a novel Multi-faceted Selection Guiding (MSG) loss to minimize the randomness and redundancy of the encoded features during the PEAS selection process. Our method outperforms prior methods on five public datasets and shows strong generalization across varying inference frequencies with less accuracy drop (ours -8.62% v.s. -20.69% for the baseline). Moreover, our model exhibits strong long spatiotemporal modeling for a broader distribution of event length (1-10^9), precise temporal perception, and effective generalization for real-world scenarios. Code and checkpoints will be released upon acceptance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PASS, a framework for event-based recognition that handles varying event lengths and sampling frequencies. The authors introduce a PEAS module for adaptive event frame selection and an MSG loss to guide this process. By utilizing state space models for efficient spatiotemporal modeling, PASS achieves strong performance and generalization across multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The paper is overall well-written and easy-to-read.\n2. The paper is well-motivated and technically sound, addressing key limitations in event-based recognition, such as handling highly variable event lengths and poor generalization to different sampling frequencies. The proposed PASS framework effectively addresses these issues by adaptively selecting informative event frames.\n3. The method outperforms prior work by significant margins (e.g., +8.31% on HARDVS) across object and action recognition tasks.\n4. The authors provide three new datasets (ArDVS100, TemArDVS100, Real-ArDVS10), supporting broader evaluation of long and realistic event sequences.\n\n### Weaknesses\n1. While the PEAS module is designed to select informative frames via MSG loss, the interpretability of the selected paths remains unclear. The paper would benefit from qualitative analyses or visualizations showing which frames are selected and why, in order to justify that the selections align with semantically meaningful moments in the event stream.\n2. The PEAS module selects a fixed number of frames from the event stream, but the temporal coherence of the selection is not guaranteed. In particular, selected frames can be concentrated in the early part of the event sequence or capture redundant moments, potentially missing critical parts or temporal dependencies. \n3. Table 4 effectively evaluates the components of the proposed PASS model. To fully justify the individual contribution of the two components, it would be helpful to include an experiment where frames are selected via random sampling instead of using the PEAS module, and only the MSG loss is applied.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nThe authors mentioned the limitations in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses event-based recognition in long event streams. Leveraging efficient state space models, this work introduces a Path-selective Event Aggregation and Scan (PEAS) module for adaptive event encoding and a novel Multi-faceted Selection Guiding (MSG) loss to minimize feature redundancy. The approach outperforms prior methods on multiple datasets and establishes new benchmarks for long-duration event recognition.\n\nSTRENGTHS AND WEAKNESSES:\nWhile prior work typically uses fixed temporal intervals for event representation, this work leverages state-space models (SSMs) to enable an adaptive sampling frequency of long video streams and thus handle ultra-long sequences. It achieves SOTA performance while efficiently processing event streams of length up to 10^9, setting new performance benchmarks. \n\nWeaknesses:\n- Clarity can be significantly improved, especially the presentation of experimental results. It is not clear \n\n- Evaluation is insufficient. The  ablation studies fail to quantify contributions of PEAS/MSG versus SSM-ViT backbones. Comparison against scaled baselines (e.g., S5-ViT-M-K(16)) is insufficient, as the baselines seem to use different scaling parameters than this approach. This needs to be clarified to support the claims of superior performance (please see the details in Questions).\n\nQUESTIONS:\n1) Question about fair baseline scaling: Does PASS outperform SSM baselines when model size (params) and frame count (K) are equalized?\n\nSuggestion:\n- Benchmark against S5-ViT-M-K(16) (matching PASS-M-K(16)'s 74M params and K=16).\n- Compare with EventDance+SSM (EventDance upgraded with S5-ViT-M).\n\n2) Please clarify: Does PASS maintains an accuracy advantage over *equally scaled* SSM baselines on N-Caltech101/PAF (object/action recognition)? If  SSM baselines are not equally scaled, the performance gains could be attributable to the larger ViT or SSM.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript introduces PASS, a Path-Selective State Space Model framework designed for event-based object and action recognition. \n\nThe primary objective is to address limitations in existing methods that inadequately leverage the high temporal resolution of event data and suffer from poor generalization across varying event lengths and sampling frequencies. \n\nPASS utilizes a Path-Selective Event Aggregation and Scan (PEAS) module for adaptively aggregating and encoding event frames. It further employs a Multi-Faceted Selection Guiding (MSG) loss to reduce randomness and redundancy among the selected features. \n\nExtensive evaluations across five public datasets and three newly introduced datasets (ArDVS100, TemArDVS100, Real-ArDVS10) demonstrate that PASS outperforms state-of-the-art methods, exhibiting robust generalization across diverse event lengths and inference frequencies.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n(+) The manuscript aims to address known challenges in existing event-based recognition methods, particularly the limitations in handling diverse event lengths and sampling frequencies. The proposed PEAS module aggregates events with adaptive selection and scanning, improving representation efficiency. The MSG loss tries to address redundancy and randomness, enhancing model generalization.\n\n(+) Comprehensive experiments conducted on both existing benchmarks and newly introduced datasets convincingly demonstrate the superior performance and generalization capabilities of the PASS framework.\n\n(+) The manuscript is overall clear, organized, and effectively communicates the methodology, findings, and observations.\n\n---\n### Weaknesses\n(-) While extensive experiments are presented, detailed ablations specifically dissecting the relative contributions of the PEAS module versus MSG loss across diverse datasets would strengthen the manuscript's insights. Also, the idea of conducting frequency-adaptive event-based recognition has been studied in existing literature, while there is no discussions on the relevant works [R1] and [R2].\n\n(-) The introduction of multiple new components (PEAS and MSG loss) makes the model complex. Additional analysis regarding interpretability or insights into the internal decision-making process would help clarify their roles.\n\n(-) The complexity of the PEAS module, coupled with SSM layers, may pose computational challenges. Explicit analyses or discussions regarding computational overhead, runtime, and memory consumption would be beneficial.\n\n---\nReferences:\n- [R1] Daniel Gehrig and Davide Scaramuzza. Low-latency automotive vision with event cameras. Nature, 629(8014):1034–1040, 2024.\n\n- [R2] Dongyue Lu, Lingdong Kong, Gim Hee Lee, Camille Simon Chane, and Wei Tsang Ooi. FlexEvent: Towards flexible event-frame object detection at varying operational frequencies. arXiv preprint arXiv:2412.06708, 2024.\n\nQUESTIONS:\n1. Based on Weakness #1: Can you clarify which component—PEAS module or MSG loss—plays a more significant role in improving generalization across varying inference frequencies? Can you add discussions on the setting formulation with existing works [R1] and [R2]? Meanwhile, please discuss on the differences and similarities with [R4].\n\n---\n2. Based on Weakness #2: Can you provide further interpretability analyses or visualizations that elucidate how the PEAS module selects frames, and how MSG loss reduces redundancy?\n\n---\n3. Based on Weakness #3: Have you performed detailed computational complexity or runtime analyses under realistic deployment scenarios? How might these findings affect practical applications?\n\n---\nAdditional questions (minor):\n\n4. Clearly define acronyms (PEAS, MSG, SSM, etc.) upon first usage for better readability.\n\n---\n5. Provide detailed computational resource requirements (e.g., GPU type, inference time) explicitly.\n\n---\n6. Do your proposed datasets sufficiently cover diverse real-world scenarios? How might the model perform under drastically different event scenarios not explicitly tested in your experiments?\n\n---\n7. Consider additional qualitative visualizations of intermediate model outputs to improve interoperability.\n\n---\n8. Discuss potential scalability issues and future steps to address them (e.g., model compression techniques).\n\n---\n9. Please add more discussions on tasks beyond recognition, such as event-based object detection [R3, R4] and semantic segmentation [R5, R6].\n\n---\nReferences:\n- [R1] Daniel Gehrig and Davide Scaramuzza. Low-latency automotive vision with event cameras. Nature, 629(8014):1034–1040, 2024.\n\n- [R2] Dongyue Lu, Lingdong Kong, Gim Hee Lee, Camille Simon Chane, and Wei Tsang Ooi. FlexEvent: Towards flexible event-frame object detection at varying operational frequencies. arXiv preprint arXiv:2412.06708, 2024.\n\n- [R3] Mathias Gehrig and Davide Scaramuzza. Recurrent vision transformers for object detection with event cameras. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13884–13893, 2023.\n\n- [R4] Nikola Zubíc, Mathias Gehrig, and Davide Scaramuzza. State space models for event cameras. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5819–5828, 2024.\n\n- [R5] Zhaoning Sun, Nico Messikommer, Daniel Gehrig, and Davide Scaramuzza. ESS: Learning event-based semantic segmentation from still images. In European Conference on Computer Vision, pages 341–357, 2022.\n\n- [R6] Lingdong Kong, Youquan Liu, Lai Xing Ng, Benoit R. Cottereau, and Wei Tsang Ooi. OpenESS: Event-based semantic scene understanding with open vocabularies. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15686–15698, 2024.\n\nLIMITATIONS:\n- The performance of the proposed framework might heavily rely on the tuning of hyperparameters such as the number of selected frames (K) or aggregation frequency, potentially requiring considerable tuning effort for new datasets.\n\n- Larger variants of the model reportedly show signs of overfitting. While acknowledged, explicit strategies for mitigating this, beyond general suggestions, are not provided in sufficient detail.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles two existing problems in event-based action/object recognition: restricted distribution of event sequence lengths and generalization across varying frequencies, by creating a new kind of event information representation method. It proposes the Path-selective Event Aggregation and Scan (PEAS) module to encode event streams of varying length into fixed length representations, and presented Multi-faceted Selection Guiding (MSG) Loss, which contains 2 kinds of information loss, to optimize the event representation generated by PEAS module. The method is evaluated on a series of datasets, and demonstrates superior performance against the existing baselines. Ablation studies shows the effectiveness of PEAS module and the newly created loss functions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n(1) This work proposes a kind of representation learning method, which is capable of effectively encoding frame streams of varying event lengths and frequencies into fixed length representations, and achieves superior experimental results.\n(2) Through reduction of event length, it partially avoids the problem of SSMs easily forgetting initial information when processing long event sequences, and provides inspiration for effectively shortening the sequence length that SSMs need to handle.\nWeaknesses:\n(1) The information loss caused by PEAS module has not been quantitatively studied.\n(2) The proposed method in this paper, as a lossy representation approach, performs effectively on relatively simple tasks such as action/object recognition, but it is unclear whether it would be effective on more complex tasks.\n\nQUESTIONS:\nQuestions*\n(1) For the fixed length event frames generated by PEAS, when using SSMs as the backbone for object/action recognition, does the SSM actually utilize the event frames, or is its utilization of the initial information more limited than the later information?\n(2) The limited-length event streams generated by PEAS inevitably lead to information loss. How should this information loss be evaluated, and in which application scenarios is such loss acceptable, and in which cases might it have a significant impact?\n(3) What’s the authors’ thoughts on improving the spatiotemporal event modeling module?\n\nLIMITATIONS:\nLimitations are presented in the above Weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}