Event Camera Object Detection at Arbitrary Frequencies

Authors: Dongyue Lu, Lingdong Kong, Gim Hee Lee, Camille Simon Chane, Wei Tsang Ooi
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Event Camera Object Detection, Low-Latency Vision, Label-Efficient Learning

Abstract:
Event cameras offer unparalleled advantages for real-time perception in dynamic environments, thanks to their microsecond-level temporal resolution and asynchronous operation. However, existing event-based object detection methods are limited by fixed-frequency paradigms, which fail to fully exploit the high-temporal resolution and adaptability of event cameras. To address these limitations, we propose FlexEvent, a novel event camera object detection framework that enables detection at arbitrary frequencies. FlexEvent consists of two key components: FlexFuser, an adaptive event-frame fusion module that integrates high-frequency event data with rich semantic information from RGB frames, and FAL, a frequency-adaptive learning mechanism that generates frequency-adjusted labels to enhance model generalization across varying operational frequencies. This combination allows FlexEvent to detect objects with high accuracy in both fast-moving and static scenarios, while adapting to dynamic environments. Extensive experiments on large-scale event camera datasets demonstrate that our approach surpasses state-of-the-art methods, achieving significant improvements in both standard and high-frequency settings. Notably, FlexEvent maintains robust performance when scaling from 20 Hz to 90 Hz and delivers accurate detection up to 180 Hz, proving its effectiveness in extreme conditions. Our framework sets a new benchmark for event-based object detection and paves the way for more adaptable, real-time vision systems. The code will be made publicly available to facilitate future research.

Decision: reject
Reviewer scores: 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730274345512

thread_type:
official_review

<thread_object>
creator: reviewer_wfuc

<content>
summary:
The paper introduces FlexEvent, an object detection framework for event cameras that operates across varying frequencies. It features two main components: 1. FlexFuser: An adaptive event-frame fusion module that integrates the high temporal resolution of event data with the rich semantic information from RGB frames, enhancing detection accuracy. 2. FAL (Freq-Adaptive Learning): A mechanism that generates frequency-adjusted labels, enabling the model to generalize across different frequencies and improve high-frequency detection through training.

strengths:
The FlexEvent framework shows strong performance in handling varying frequencies, particularly in dynamic and fast-changing environments, as demonstrated in specific datasets that combine event data and RGB frames. Its advantage lies in the effective fusion of event cameras' high temporal resolution with the rich semantic information from RGB frames, which enhances detection accuracy in scenarios where both data modalities are available. The integration of frequency-adaptive learning allows the model to generalize well across different operational conditions, showing robust performance in high-frequency scenarios on multimodal datasets like DSEC-Det, DSEC-Detection, and DSEC-MOD.

weaknesses:
One weakness of FlexEvent is that the comparison with other models may not be entirely fair, as many of the compared methods are trained on datasets that use only event data, where the data distribution, camera setups, and number of modalities differ. This discrepancy in experimental setup can lead to significant performance differences, making it harder to evaluate FlexEvent's true advantage. Since most of the other comparison methods are trained on the 1Mpx and Gen1 datasets, the authors should clearly state in the main text whether the comparison methods are retrained on DSEC-Det, DSEC-Detection, and DSEC-MOD.  The authors should ensure consistency in the training modalities of all models, comparing them either exclusively in the Event-RGB mode or using purely event-based training.

Another potential weakness is that the FAL in FlexEvent closely resembles existing methods such as EventDrop [1] and Shadow Mosaic [2]. Fundamentally, all these techniques focus on altering the event data’s density during training. By manipulating the frequency or sparsity of event data, FlexEvent's approach may not provide significant innovation beyond these existing techniques. This similarity raises the question of whether the improvement in performance comes more from an established method of adjusting event density, rather than from a novel or more advanced detection strategy. The authors could provide more ablation studies and visualization results to compare whether FAL shows significant differences from other data augmentation methods.

The authors should provide the network's latency, parameter count, and computational cost, as other event camera object detection networks do, to enable a fairer performance comparison.

[1] Fuqiang Gu and Weicong Sng and Xuke Hu and Fangwen Yu. (2021). EventDrop: data augmentation for event-based learning
[2] Peng, Y., Zhang, Y., Xiao, P., Sun, X., & Wu, F. (2023). Better and Faster: Adaptive Event Conversion for Event-Based Object Detection.

questions:
It is a well-established fact that objects generally do not move significantly within a few milliseconds. Based on this premise, is changing the frequency and adjusting the event sparsity through data augmentation essentially the same? When addressing the issue of missed detections at low frequencies, is it more the RGB information that compensates for this, rather than the method you proposed?

Unlike other papers, this article does not provide the model’s inference time, particularly how it varies across different frequencies. I am curious to know whether the inference time can be kept within the range of the input event data.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690375587

thread_type:
official_review

<thread_object>
creator: reviewer_ulco

<content>
summary:
This paper introduces FlexEvent, an innovative joint object detection framework that utilizes both events and frames to enable object detection at arbitrary frequencies, leveraging the high temporal resolution of event cameras. The proposed FlexEvent incorporates two key modules. The first FlexFuser adaptively combines high-frequency events with rich frame textures. The second FAL generates frequency-adjusted labels to enhance model performance across various operational frequencies. The results show that FlexEvent outperforms current state-of-the-art methods, achieving significant gains in high-frequency detection settings.

strengths:
1) The relevant background knowledge of this paper is clearly explained.

2) The topic of object detection using events and frames at arbitrary frequencies is very interesting topic.

3) The authors conducted extensive experiments to validate the effectiveness of FlexEvent across multiple large-scale event-based object detection datasets. FlexEvent achieves significant improvements in mAP performance compared to existing SOTA methods and demonstrates the capability to detect objects at arbitrary frequencies.

weaknesses:
1. In the contributions section, the authors claim to be the first to address arbitrary-frequency object detection using event cameras. This phrasing could be more precise, and I recommend refining it in the camera-ready version. I recognize this as an important topic for practical applications of event cameras. Some prior works have explored high-frequency inference by combining events and frames, such as CovGRU for arbitrary-frequency depth estimation [1] and asynchronous fusion modules for arbitrary-frequency object detection [2].

[1] Combining events and frames using recurrent asynchronous multimodal networks for monocular depth prediction, IEEE RAL, 2021.

[2] SODFormer: Streaming object detection with transformer using events and frames, IEEE TPAMI 2023.

2. In Table 1, the authors report a high mAP of 0.574 on the DSEC-Det dataset. Could you clarify the number of classes used for this evaluation? Is it two classes or eight? The appendix states eight classes for DSEC-Det. Please provide clarification on this point.

3. The writing could be improved, particularly in Section 3.1, where the background and motivation would be more appropriate in the Introduction or Related Work sections.

questions:
Please see weakness. Now, I give a preliminary score based on the initial manuscript and will consider adjustments depending on the authors' responses and feedback from other reviewers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691581686

thread_type:
official_review

<thread_object>
creator: reviewer_fu5g

<content>
summary:
Summary: 

The authors present FlexEvent, a pioneering framework for event camera object detection that leverages the high temporal resolution and asynchronous operation of event cameras. By integrating an adaptive event-frame fusion module (FlexFuser) with a frequency-adaptive learning mechanism (FAL), FlexEvent overcomes limitations of existing methods that rely on fixed-frequency paradigms. This enables the model to detect objects with high accuracy across various operational frequencies, demonstrating robustness in both fast-moving and static scenarios.

strengths:
Strengths:

1 The FlexFuser module effectively combines the rich spatial and semantic information from RGB frames with the high temporal resolution of event data. The approach to sample event data at different frequencies during training is particularly commendable, allowing the model to retain the benefits of high-frequency data during inference while maintaining efficiency.

2 The FAL module enhances the framework by generating frequency-adaptive labels for unlabeled high-frequency data. This self-training approach improves model generalization and robustness across varying operational frequencies, which is a notable advancement in the field.

weaknesses:
Weaknesses:

1 The overall architecture in Figure 2 does not clearly illustrate the process of feature fusion at different frequencies, specifically the $(i) _ {hfuse}=(i) _ {hfuse}^a+(i) _ {hfuse}^b $ mentioned at the end of section 3.2 on Event-Frame Adaptive Fuser. Additionally, there seems to be an issue with the representation of Eq (2). Since $i$ represents different scales, the inputs to the network for all scales except the initial one which corresponds to $E^a$, $E^b$, $F$, should correspond to $(i-1) _ {hE}^a$, $(i-1) _ {hE}^b$, and $(i-1) _ {hF}^a$.

2 In section 3.3, the formula related to pre-training with low-frequency labels lacks numbering, which may confuse readers. This formula is critical as it corresponds to the FAL part of Figure 2.

3 The experiments presented in section 4.2 are somewhat disorganized. The evaluation methods across the three datasets (DSEC-Det, DSEC-Detection, DSEC-MOD) differ, and it is unclear how comparisons are made consistently. For instance, in DSEC-Det, only the proposed method and DAGr are tested, while the results in DSEC-Detection compare against CAFR, necessitating a clearer presentation.

4 There is a discrepancy in the reported number of training iterations between section 4.1 and supplementary material A.3, with one stating 100,000 iterations and the other citing 200,000. This inconsistency should be rectified to ensure clarity.

questions:
Could the authors clarify the rationale behind selecting specific datasets and methods for testing, and how this impacts the validity of their findings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732364493993

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank the reviewers for devoting time and effort and providing constructive feedback on this work. 

---

We are encouraged to see that our work is recognized as an innovative joint object detection framework leveraging both event and frame data to enable object detection at arbitrary frequencies. We particularly appreciate the reviewers’ recognition of the following strengths:
- Reviewer `FU5g` commended our approach of *"sampling event data at different frequencies during training as particularly effective"*.  
- Reviewer `uLco` found the topic of object detection using events and frames at arbitrary frequencies *"highly interesting"* and noted that the *"background knowledge in our paper is clearly explained"*.  
- Reviewers `uLco` and `wFUC` highlighted that FlexEvent *"demonstrates strong performance in handling varying frequencies"*, particularly "*in dynamic and fast-changing environments"*.

---

We would like to reiterate the key contributions of our work:
1. **FlexEvent Framework**: Designed to address the challenging problem of object detection using event cameras at arbitrary frequencies, making it one of the early explorations in this area.  
2. **FlexFuser Module**: Combines the strengths of event and frame data, enabling efficient and accurate detection in dynamic environments.  
3. **Frequency-Adaptive Learning (FAL)**: A novel mechanism that generates frequency-adaptive labels, significantly improving generalization across a wide range of motion frequencies.  

---

### **Summary of Changes**

We have revised our manuscript to incorporate the feedback from our reviewers. The modified content related to the comments from Reviewers `FU5g`, `uLco`, and `wFUC` are highlighted in **cyan**, **pink**, and **purple** colors, respectively.

Here, we summarize the changes as follows:

- ***Introduction***
  - As suggested by Reviewer `uLco`, we have refined the contribution claims.

- ***Related Works***
  - As suggested by Reviewer `uLco`, we have added a detailed comparison with earlier works addressing varying-frequency scenarios, such as CovGRU and SODFormer.

- ***Methodology***
  - As suggested by Reviewer `uLco`, we have refined the Preliminaries section.  
  - As suggested by Reviewer `FU5g`, we have improved Figures 2 and 3 for clarity and added a more detailed explanation of the method for handling different frequencies.

- ***Additional Experiments***
  - As suggested by Reviewer `FU5g`, we have corrected the explanation regarding training iterations in the Appendix.  
  - As suggested by Reviewer `wFUC`, we have added comparison results for inference time and parameter count in the Appendix.  
  - As suggested by Reviewer `wFUC`, we have included comparison results for data augmentation methods.

---

We sincerely hope that these revisions and additional clarifications address the concerns raised by the reviewers and further demonstrate the novelty and significance of our work.

---

Meanwhile, we are looking forward to hearing your feedback. Please let us know whether you have further questions on this work during this **Author-Reviewer Discussion** session. 
Thank you once again for your valuable feedback and insights, which have greatly helped us improve the quality of our paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733301928741

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal for Submission 589

comment:
We sincerely thank all the reviewers for their time, effort, and constructive feedback, which have greatly contributed to improving our manuscript. Below, we summarize our key contributions and the changes made to address the reviewers' concerns.

---

### **Key Contributions**

1. **FlexEvent Framework**: One of the early attempts to address object detection using event cameras at arbitrary frequencies, validated through extensive benchmarks across varying scenarios.

2. **FlexFuser Module**: Combines the high temporal resolution of event data with the spatial richness of frames, enhancing detection accuracy in dynamic conditions.

3. **Frequency-Adaptive Learning (FAL)**: A novel mechanism that improves generalization across motion frequencies by leveraging frequency-adaptive labels.

---

### **Concerns by reviewers**
1. Reviewer `FU5g` raised issues regarding the clarity of figures, equations, and experimental organization.
2. Reviewer `uLco` suggested refining the contribution claims for greater precision, clarifying evaluation setups, and reorganizing sections for improved structure.
3. Reviewer `wFUC` questioned the nature of our FAL mechanism, expressed concerns about fairness, and requested comparisons on efficiency and computational cost.
---
### **Revisions and Changes**

We have revised the manuscript based on the reviewers' feedback. Key changes are highlighted in **cyan** (Reviewer `FU5g`), **pink** (Reviewer `uLco`), and **purple** (Reviewer `wFUC`). Below is a summary of the changes:

1. **Introduction**
- Refined contribution claims to clarify FlexEvent’s novelty compared to related works as suggested by Reviewer `uLco`.

2. **Related Works**
- Expanded discussions to include comparisons with earlier works addressing varying-frequency scenarios (e.g., CovGRU and SODFormer) and data augmentation methods  (e.g., EventDrop and Shadow Mosaic), providing clearer distinctions (Reviewer `uLco` and Reviewer `wFUC`).

3. **Methodology**
- Improved clarity of Figures 2 and 3 and added detailed explanations of how FlexEvent handles varying frequencies (Reviewer `FU5g`).
- Refined the Preliminaries section for better structure and coherence (Reviewer `uLco`).

4. **Additional Experiments**
- Added comparison results for inference time and parameter count to address concerns of computational fairness in the Appendix (Reviewer `wFUC`).
- Included comparisons with data augmentation methods, such as EventDrop and Shadow Mosaic in the Appendix (Reviewer `wFUC`).
- Corrected discrepancies regarding training iterations in the Appendix (Reviewer `FU5g`).
- Detailed experiment setting regarding baselines and datasets in the Appendix (Reviewer `uLco`).

5. **Additional Clarifications**
- In our discussion with reviewer `wFUC`, we clarified the principles and novelty of our FAL mechanism, as well as the fairness of our experiments about FAL.
---

### **Conclusion**

We are confident that we have addressed the reviewers' concerns and strengthened the manuscript. The revisions clarify our contributions, provide additional experimental evidence, and highlight the novelty and significance of our work. 

Thank you for your time and consideration.

Sincerely,

The Authors of Submission 589
</content>
</thread_object>

</thread>


Metareview:
metareview:
Three reviewers have given scores of 5, 5, and 6. Among them, the reviewers believe that there are significant issues with the writing of the article, which may make it difficult for readers to truly understand the author's intentions. Although the author has given a strong response in this regard, it still cannot cover the shortcomings in the writing. In terms of innovation, the author's approach of modeling event data from the perspective of dynamic temporal resolution is relatively close to several existing works mentioned in the discussion. The author failed to give a direct response to the issue of the sliding window. Considering the problems in writing and innovation, the current version of the manuscript is not recommended for acceptance.