Title: Rethinking Scale-Aware Temporal Encoding for Event-based Object Detection
Venue: NeurIPS 2025
Accepted: yes
Keywords: Event-based object detection

temporal feature extraction
Abstract: Event cameras provide asynchronous, low-latency, and high-dynamic-range visual signals, making them ideal for real-time perception tasks such as object detection. However, effectively modeling the temporal dynamics of event streams remains a core challenge. Most existing methods follow frame-based detection paradigms, applying temporal modules only at high-level features, which limits early-stage temporal modeling. Transformer-based approaches introduce global attention to capture long-range dependencies, but often add unnecessary complexity and overlook fine-grained temporal cues. In this paper, we propose a CNN-RNN hybrid framework that rethinks temporal modeling for event-based object detection. Our approach is based on two key insights: (1) introducing recurrent modules at lower spatial scales to preserve detailed temporal information where events are most dense, and (2) utilizing Decoupled Deformable-enhanced Recurrent Layers specifically designed according to the inherent motion characteristics of event cameras to extract multiple spatiotemporal features, and performing independent downsampling at multiple spatiotemporal scales to enable flexible, scale-aware representation learning. These multi-scale features are then fused via a feature pyramid network to produce robust detection outputs. Experiments on Gen1, 1 Mpx and eTram dataset demonstrate that our approach achieves superior accuracy over recent transformer-based models, highlighting the importance of precise temporal feature extraction in early stages. This work offers a new perspective on designing architectures for event-driven vision beyond attention-centric paradigms. Code: https://github.com/BIT-Vision/SATE.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript proposes an CNN-RNN hybrid architecture tailored specifically for event-based object detection, highlighting the importance of early-stage, scale-aware temporal modeling. \n\nThe authors introduce recurrent modules at lower spatial scales to capture fine-grained temporal patterns inherent to sparse event streams. Central to their approach is the Decoupled Deformable-Enhanced Recurrent Layer (DDRL), a module that separates per-pixel motion estimation from feature fusion. \n\nThe architecture incorporates independent temporal downsampling branches at multiple resolutions, enabling flexible multi-scale spatiotemporal representation learning. \n\nExperiments conducted on two standard event-based object detection benchmarks, Gen1 and 1Mpx, demonstrate good performance over existing Transformer- and SNN-based methods, particularly in scenarios involving fast-moving or small-scale objects.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n(+) The manuscript identifies a limitation in existing temporal modeling strategies for event-based object detection and convincingly motivates the need for early-stage temporal feature preservation. The introduction of recurrent modules at lower spatial scales is with a good motivation, aiming at addressing a previously under-explored aspect in event-based detection.\n\n(+) The proposed method achieves state-of-the-art results on the Gen1 and 1Mpx benchmarks. The performance improvements over recent Transformer-based event-only object detection methods underscore the efficacy of the proposed temporal modeling framework.\n\n(+) The manuscript is overall clearly structured, well-written, and technically sound.\n\n---\n\n### Weaknesses\n(-) Moderate novelty of individual components in the proposed framework. While the integration of recurrent modules at earlier stages is innovative for event-based detection, the individual components (ConvLSTM, deformable convolution, FPN-based feature fusion) have been well-established in other contexts. Thus, the technical novelty lies more in the thoughtful combination and adaptation rather than entirely new component inventions.\n\n(-) The evaluation is limited to two datasets, albeit popular and standard in event-based detection. Given the broad claims about scale-awareness and robust temporal modeling, experiments or at least discussions about generalizability to other event-based or event-frame fusion datasets could enhance the impact.\n\n(-) Although experiments report competitive inference speed, deformable convolutions and recurrent layers inherently add computational complexity. A deeper exploration of runtime implications, scalability to higher resolution event cameras, or computational trade-offs in real-world deployment scenarios would further strengthen the manuscript.\n\n(-) A more thorough discussion of the proposed method's limitations, failure modes, or scenarios where performance gains are minimal would provide valuable insights for future research.\n\nQUESTIONS:\n1. Based on Weakness #1: While your integration of recurrent modules at earlier stages is effective, the key building blocks (ConvLSTM, deformable convolution, and FPN) are well-established. Can you clearly elaborate on which aspects of these modules were specifically adapted or uniquely designed for event-based detection in your work?\n\n---\n\n2. Based on Weakness #2: You evaluated your method on two standard datasets (Gen1 and 1Mpx). How well do you expect your architecture’s advantages to generalize beyond these specific benchmarks? Have you considered or tested scenarios with significantly different event distributions or sensor characteristics? The recent trend has shifted to incorporating more RGB cues in event-based detection frameworks. Can you discuss and compare your method (perhaps in event-frame fusion mode) with existing methods, such as DAGr [R1], RENet [R2], FlexEvent [R3], and/or CAFR [R4]?\n\n---\n\n3. Based on Weakness #3: Given the inherent complexity of recurrent and deformable convolution modules, have you performed a detailed runtime or complexity analysis under varying conditions (e.g., higher resolution or event density)? How might these computational considerations affect practical deployments on resource-constrained platforms?\n\n---\n\n4. Based on Weakness #4: Can you provide a more thorough discussion of scenarios or cases where your proposed DDRL module may perform poorly or have limited benefits? Understanding these boundary conditions would significantly strengthen the clarity and practical applicability of your contributions.\n\n---\nReferences:\n- [R1] Daniel Gehrig and Davide Scaramuzza. Low-latency automotive vision with event cameras. Nature, 629(8014):1034–1040, 2024.\n\n- [R2] Zhuyun Zhou, Zongwei Wu, Rémi Boutteau, Fan Yang, Cédric Demonceaux, and Dominique Ginhac. RGB-event fusion for moving object detection in autonomous driving. In IEEE International Conference on Robotics and Automation, pages 7808–7815, 2023.\n\n- [R3] Dongyue Lu, Lingdong Kong, Gim Hee Lee, Camille Simon Chane, and Wei Tsang Ooi. FlexEvent: Towards flexible event-frame object detection at varying operational frequencies. arXiv preprint arXiv:2412.06708, 2024.\n\n- [R4] Hu Cao, Zehua Zhang, Yan Xia, Xinyi Li, Jiahao Xia, Guang Chen, and Alois Knoll. Embracing events and frames with hierarchical feature refinement network for object detection. In European Conference on Computer Vision, pages 161–177, 2024.\n\nLIMITATIONS:\n- Although the method aims to address fine-grained temporal information effectively, scenarios with extremely sparse events—such as low-motion or stationary objects—may still challenge performance. The manuscript could benefit from further discussion on this point.\n\n- Despite demonstrating competitive inference speeds, the practical deployment of deformable and recurrent modules in real-time and resource-limited environments could still pose challenges. Acknowledging potential trade-offs between accuracy and latency/compute would enhance transparency regarding deployment feasibility.\n\n- Given the experiments’ constraints in scale (resolution downsampling and sequence length), performance scalability on much larger datasets or continuous event streams has not been explicitly verified. How the proposed approach scales up in terms of training time, computational resources, and memory consumption could be a limiting factor for widespread adoption.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of effectively modeling temporal dynamics for event-based object detection. It proposes a CNN-RNN hybrid architecture that enhances temporal modeling by placing recurrent modules at lower spatial scales, which allows for the capture of fine-grained temporal features in sparse event streams. In addition, the authors introduce a Decoupled Deformable-enhanced Recurrent Layer to decouple per-pixel motion estimation and feature fusion, along with a multi-scale approach for spatio-temporal encoding. The experimental results demonstrate that this method outperforms recent transformer-based models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. A new hybrid architecture, CNN-RNN, is proposed, offering a new perspective on event-based object detection by prioritizing early-stage temporal modeling.\n\n2. A scale-specific spatio-temporal encoding approach allows for more flexible and comprehensive feature extraction.\n\nWeaknesses:\n1. The proposed methods do not present new insights or theoretical advancements in the context of event-based representation learning, particularly in areas like asynchronous data processing or novel feature extraction techniques. This approach primarily improves on existing RNN-based models without introducing a groundbreaking shift in understanding the underlying event-based object detection problem.\n2. (1) Lack of recent state-of-the-art event-based object detection methods, such as: SpikeYOLO [1], EGSST [2], EventPillars [3], HsVT [4], SNN-ANN [5], etc；(2) Compared to recent Sstate-of-the-art methods, the performance improvement is limited; (3) Has the proposed method been tested for other alternative event representations (such as time surfaces, histogram)? (4) Can the proposed method be tested on the EvDET200K [6] dataset?\n\n[1] Integer-valued training and spike-driven inference spiking neural network for high-performance and energy-efficient object detection (ECCV2024)\n\n[2] EGSST: Event-based Graph Spatiotemporal Sensitive Transformer for Object Detection (NeurIPS 2024) \n\n[3] EventPillars: Pillar-based Efficient Representations for Event Data (AAAI2025)\n\n[4] Hybrid Spiking Vision Transformer for Object Detection with Event Cameras (ICML2025)\n\n[5] Efficient Event-Based Object Detection: A Hybrid Neural Network with Spatial and Temporal Attention (CVPR2025)\n\n[6] Object detection using event camera: A moe heat conduction based detector and a new benchmark dataset (CVPR2025)\n\nQUESTIONS:\nSee Weakness\n\nLIMITATIONS:\nNo, it is recommended that the authors explore the limitations of the paper in detail", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a hybrid CNN–RNN architecture for event-based object detection that emphasizes fine-grained temporal encoding at early stages. Raw event streams are discretized into 5 temporal bins over 50 ms, then processed by a shared stem and three Decoupled Deformable-enhanced Recurrent Layers (DDRLs). Each DDRL combines a ConvLSTM with a modulated deformable convolution to align per-pixel motion before spatial downsampling. Multi-scale spatiotemporal features from three resolution branches are fused via an FPN and passed to a YOLOv6-based detection head. Experiments on the Gen1 (304×240) and 1 Mpx (1280×720) driving datasets show state-of-the-art mAPs of 52.7% and 49.1%, respectively, with comparable parameter count and FLOPs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The early-stage ConvLSTM effectively captures temporal cues before significant spatial pooling, preserving fine-grained motion information.\n2. Combining ConvLSTM with a modulated deformable convolution in each DDRL adaptively aligns pixel-wise motion, reducing blur in fast-moving regions.\n3. The divide-and-conquer architecture cleanly separates motion estimation from feature fusion, which suits the asynchronous nature of event data.\n4. The model delivers state-of-the-art mAP on both Gen1 and 1 Mpx driving benchmarks, demonstrating robustness across resolutions.\n5. Comprehensive ablation studies (e.g., DDRL placement, branch design) substantiate key architectural choices.\n\nWeaknesses:\n1. The fixed 5-bin, 50 ms voxelization may not capture the fastest or smallest event dynamics, yet no ablation evaluates finer temporal discretization.\n2. The related work omits key fine-grained temporal-encoding methods (e.g., SODFormer, Time-Surface Pyramid), leaving the paper’s positioning in this space unclear.\n3. There is no visualization or quantitative analysis of the learned offsets (∆xₖ), nor any comparison to optical-flow ground truth to validate that motion alignment is actually occurring.\n4. Combining deformable convolution with ConvLSTM could over-smooth spatial detail, but no effective-receptive-field or smoothing analysis is provided.\n5. Independently processed branches are fused via a standard FPN without any temporal-state synchronization mechanism, and no small/medium/large AP breakdown or branch-ablation confirms the benefits for small or rapid objects.\n6. The efficiency evaluation is incomplete: “Time (ms)” in Table 1 is ambiguous, and there are no hardware-specific end-to-end latency, peak memory, or throughput measurements to support the “compact yet accurate” claim.\n\nQUESTIONS:\n1. Have you conducted an ablation over voxelization granularity (e.g., bins ∈ {5, 8, 10}, window ∈ {20 ms, 50 ms, 100 ms}) to confirm that the chosen 5-bin, 50 ms discretization sufficiently captures the fastest event dynamics?\n2. Could you visualize the learned deformable offsets (∆xₖ), such as heat maps or flow fields, to validate that the deformable module aligns motion as intended?\n3. Please compare or discuss your approach relative to SODFormer (Li et al., 2023) and the Time-Surface Pyramid (Manderscheid et al., 2019). Benchmarking against these fine-grained temporal-encoding methods would help clarify the paper’s novelty and positioning.\n4. Can you report end-to-end inference latency, peak GPU memory usage (batch 1), and a small/medium/large object AP breakdown on both Gen1 and 1 Mpx inputs, measured on specified hardware?\n5. Would moving one or two key supplementary results (e.g., the Table S1/S2 ablations or a representative offset heat map) into the main manuscript improve transparency around your architectural choices?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers object detection using event-camera data. The authors argue that temporal information needs to be captured at low spatial scales before the extraction of high-dimensional convolutional features reduces the spatial resolution. The key idea is to introduce event-recurrent (RNN) modules at low spatial scales first, followed by scale-specific (CNN) feature extraction and a feature pyramid network (FPN) for object detection. The proposed RNN-CNN architecture operates over a 3D spatio-temporal voxel grid of event polarities, applies decoupled deformable convolution to fuse the current frame features with historical features, a ConvLSTM layer to capture spatio-temporal features from consecutive event frames, and finally an FPN to fuse multi-scale features. The method is evaluated on two of the widely used datasets for event-based object detection: Gen1 and 1Mpx in comparison to several recent Transformer, Transformer+RNN, GNN, and CNN+RNN architectures.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n + The proposed architecture is simple, well motivated, and performs well. I find the proposed design using several recurrent layers with deformable convolution and LSTM for the temporal events simple and effective. The consideration of both a hidden state and a memory cell in the LSTM model is nice.\n \n + The performance of the method on the Gen1 and 1Mpx datasets is very good in terms of both accuracy and time in comparison to strong recent baselines.\n\n + The ablation study results are interesting and, importantly in my opinion, underscore that the proposed design is effective, even without deformable convolution or Squeeze-Excite blocks in the model, and in comparison to Transformer-RNN [13].\n \n \n## Weaknesses:\n - The use of deformable convolution in the recurrent layer design raises questions about generalization. It is important that the authors evaluate the performance of the model when trained on one dataset and tested on a different one. \n\n - The presentation in Sec. 1-3 is somewhat repetitive regarding the proposed design. Reducing the repetitiveness will allow more space for presenting additional technical details, adding more quantitative (generalization) and qualitative results, and improving the qualitative results.\n \n - The qualitative results in Fig. 5 and 6 are hard to see and offer limited value. What are we supposed to observe? How does the visualization show anything about the feature processing and integration by the different models? The evaluation can be strengthened by providing better/additional qualitative results.\n\nQUESTIONS:\n1. My most critical suggestion is to add a generalization experiment, e.g., train on one dataset and test on the other. For example, this paper https://arxiv.org/abs/2505.12908 does this in their Table 2. It would be nice to see the performance of the proposed method in a similar experiment.\n \n2. The paper is repetitive about the high-level points that temporal patters in event data appear in low-level features and are best captured before spatial abstraction. This point and several related ones are made multiple times in the Intro, Related Works, Method sections. On the other hand, the review of related work is quite brief and very few qualitative results are presented. I recommend that the authors reduce the repetitiveness of the high-level motivation for the approach and instead add more of: technical details, review of additional related works, qualitative results.\n\n3. The authors should include the time and accuracy results for all methods in Table 1. Also, the best and second-best method should be highlighted for the time results as well.\n \n4. Minor comments:\n  - The meaning of hidden state, cell state, and BN are not clear the first time they are introducing in eq. (2), (3). These elements are not explained clearly until Sec. 3.4\n  - The references have multiple problems that need to be addressed, e.g., [7] is missing a publication venue, [10] includes et al. in the author list (not preferable), [17], [18], etc. include the publication year and IEEE twice (repetition is unnecessary), etc.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}