Labits: Layered Bidirectional Time Surfaces Representation for Event Camera-based Continuous Dense Trajectory Estimation

Authors: Zhongyang Zhang, Jiacheng Qiu, Shuyang Cui, Yijun Luo, Tauhidur Rahman
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: event camera, trajectory estimation, computer vision, event-based vision, dynamic vision sensor, feature tracking

Abstract:
Event cameras provide a compelling alternative to traditional frame-based sensors, capturing dynamic scenes with high temporal resolution and low latency. Moving objects trigger events with precise timestamps along their trajectory, enabling smooth continuous-time estimation. However, few works have attempted to optimize the information loss during event representation construction, imposing a ceiling on this task. Fully exploiting event cameras requires representations that simultaneously preserve fine-grained temporal information, stable and characteristic 2D visual features, and temporally consistent information density—an unmet challenge in existing representations. We introduce Labits: Layered Bidirectional Time Surfaces, a simple yet elegant representation designed to retain all these features. Additionally, we propose a dedicated module for extracting active pixel local optical flow (APLOF), significantly boosting the performance. Our approach achieves an impressive 49\% reduction in trajectory end-point error (TEPE) compared to the previous state-of-the-art on the MultiFlow dataset. The code will be released upon acceptance.

Decision: reject
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729269568418

thread_type:
official_review

<thread_object>
creator: reviewer_nuye

<content>
summary:
This paper introduces a simple and elegant representation to preserve fine-grained event features for event-based continuous dense trajectory estimation. Experiments conducted on the MultiFlow dataset demonstrate the effectiveness of the proposed solution. Extensive analyses and visualizations are presented. The source code will be provided to foster future research.

strengths:
1. A novel and effective synchronous event representation aware of the event camera's asynchronous characteristics is presented.
2. The paper is overall well-written and nicely structured.
3. The source code will be provided to foster research in this direction.
4. Rich interesting visualization results demonstrate the effectiveness of the proposed solution.

weaknesses:
1. If it is possible, it would be nice to investigate the potential of the proposed representation for optical flow estimation or mesh flow estimation.
2. In addition to the MultiFlow dataset, it would be nice to experiment on another dataset to show the generalization capacity of the proposed solution.
3. If it is possible, please consider presenting analyses to directly show the dense time-continuous prediction results by visualizing the results in different time steps within a sample window of the MultiFlow dataset.
4. More detailed ablation studies and parameter analyses could be conducted. For example,  the numbers of the TORE volume, the voxel grid, the Labits, and time bins could be experimented for analyzing the best parameter settings or helping understand the effects of different parameters.

questions:
1. Would you consider discussing some limitations of the presented work and pointing out some future research directions?
2. Would you consider directly comparing the proposed representation against some existing event representations to verify the superiority of your solution?
3. In Table 1, it is indicated that the voxel grid discards the event polarity. However, voxel grids actually contain the accumulation of event polarity information. Could you explain this?
4. Actually, for voxels, they can also support the training of a voxel-to-APLOF network by computing a voxel active event mask and computing voxels with accumulative time spans, similar to the unified voxel grid of EVA-Flow. Could you explain this?
5. What is r_start in equation(5)(6)? Does O(x) indicate the optical flow speed or displacement? Then how did you compute Or(x), Or+(x_start), and Or-(x_start)? This is somewhat misleading.
6. It is claimed that "Labits is well-suited for local motion-sensitive tasks and has the potential to standard". However, only the last two rows in Table 2 indicate the results of Labits. It would be nice to add more results to verify the superiority of Labits, e.g., by testing DCT-RAFT with Labits and by testing with Unified Voxel Grid.
7. While the proposed method achieves high accuracy on MultiFlow, it is a synthetic dataset. It would be nice to offer results on a real-world dataset like DSEC-Flow to verify the generalization capacity of the proposed method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729842099444

thread_type:
official_review

<thread_object>
creator: reviewer_8m1q

<content>
summary:
They introduced Labits, a novel event representation that simultaneously retains fine-grained temporal information, meaningful 2D visual patterns, and local speed cues. They also developed the Labits-to-APLOF net, which accuratelyconverts Labits into active pixel local optical flows. The results high-light that, beyond model architecture, event representations also have a transformative impact onthe final outcomes.

strengths:
1. a novel synchronous event representation that is aware of event camera's asynchronous characteristic and keeps rich local movement trend information

2.They achieved the SOTA performance on event-based dense trajectory estimation task.

3.trained a corresponding active pixel local optical flow estimator based on Labits layers, which significantly benefits movement-sensitive tasks.

weaknesses:
1.In Figure 4, there is less difference between (d) and (e). What’s the influence of APLOF? Could you give explanation for this phenomenon?

2.In Table3, you already give the model size and other metrics. Could you please compare your model with others in FLOPs and total parameters to demonstrate your model’s efficiency?

questions:
See weakness above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730473386387

thread_type:
official_review

<thread_object>
creator: reviewer_zwl1

<content>
summary:
This paper introduces Labits, a new representation method called "Layered Bidirectional Time Surfaces," designed for event cameras to enable precise dense trajectory estimation and optical flow calculation. Labits maximize the advantages of event cameras by capturing fine-grained temporal information, retaining stable 2D features, and maintaining consistent information density in spatiotemporal motion estimation. The authors also propose a specialized module, Labits-to-APLOF (Active Pixel Local Optical Flow), which uses Labits representations to enhance the accuracy of local optical flow estimation at active pixels. This paper demonstrates Labits' superior performance by reducing trajectory end-point error (TEPE) by 49% over previous state-of-the-art methods in the MultiFlow dataset, making it an effective and efficient tool for motion-sensitive tasks.

strengths:
- The motivation is compelling, addressing the need for more accurate and dense event representations that fully exploit the unique properties of event cameras. This need is crucial in dynamic environments where event cameras can provide high temporal resolution.
- The paper is organized logically, with a clear distinction between the problem background, the proposed solution (Labits), and the experiments.
- Labits and the Labits-to-APLOF module demonstrate notable improvements in trajectory estimation accuracy. By leveraging bidirectional time surfaces, Labits successfully addresses temporal occlusion and provides a stable representation of movement information, which is particularly useful for complex tasks such as trajectory prediction, event-based video interpolation, and high-speed object tracking.

weaknesses:
- Some technical justifications for the Labits representation could be expanded. For example, while the use of bidirectional time surfaces is novel, more in-depth analysis of why this bidirectional approach provides such significant accuracy improvements over unidirectional methods could make the claims stronger.
- The paper introduces the Labits-to-APLOF network as a lightweight and adaptable feature. However, additional evidence supporting its adaptability, such as visual examples of Labits in various scenarios, would clarify its generalizability.
- While Labits achieves efficiency gains, it is slower than simpler methods like the Voxel Grid. Additional experiments demonstrating the trade-off between computational cost and accuracy gains in real-time settings would strengthen the justification for using Labits in time-sensitive applications.
- The evaluation of Labits is confined to a single dataset (MultiFlow), limiting insight into its performance across different event-based vision tasks and datasets. Exploring Labits in various contexts and testing it on multiple datasets would offer a more comprehensive assessment of its versatility and robustness, providing a promising direction for future research.

questions:
- The authors claim that the bidirectional time surface design enables Labits to reduce estimation error significantly. Could they provide additional analysis or visualizations to show why this bidirectional approach better captures motion density and consistency compared to unidirectional methods?
- Could the authors clarify the Labits layer's adaptability? Specifically, how does Labits handle scenarios with minimal event data where forward and backward time data might be sparse?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644687368

thread_type:
official_review

<thread_object>
creator: reviewer_xbdn

<content>
summary:
This paper tackles the representation problem of asynchronous indefinite-length event data feeding into modern networks. The authors propose Labits, a new representation based on a hierarchical bi-directional event surface, to better preserve the millisecond time information in the raw data. The authors conduct experiments on the continuous point tracking task and verify the performance benefits over other representations. However, from my perspective, the contribution is still insufficient, with only a new representation that has not been fully experimentally validated.

strengths:
1. A new event representation Labits is proposed to preserve rich temporal information for motion-related downstream tasks.
2. The benefits of this new representation compared to the commonly used voxel grid are demonstrated on the nonlinear dense trajectory estimation task.

weaknesses:
1. Experimental comparisons only on simulated data are not convincing, especially since the compared methods [1] already did some experiments on real data. At least qualitative or quantitative experimental comparisons should be done on a real collected event data, e.g., the optical flow estimation evaluation on DSEC in [1].

2. The new representations Labits presented in this paper are one of the core contributions, and it is necessary to apply this to multiple tasks (both motion-related and motion-irrelevant) and perform comprehensive evaluations, e.g., two-frame optical flow estimation, depth estimation, object detection, motion segmentation, etc. The authors only experimented with one task and mentioned that it could be used for other motion-related downstream tasks, but not actually provided.

3. ``Since the number of time bins in Labits can be freely adjusted by the user, maintaining a single-channel input increases flexibility.'' Does this indicate that the pretrained Labits-to-APLOF Net can adaptively handle different time bins set by the user? In addition, ablation experiments with a larger number of channels are necessary.

4. The differences between the dense tracking model and [1] need to be clarified.

[1] Dense continuous-time optical flow from event cameras, TPAMI 2024.

questions:
1. Table 1 indicates that the Voxel Grid discards polarity information. How does Labits handle this?
2. ``Labits is a synchronous event representation.'' Does it support online processing?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732315671454

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Reviewers,

We are deeply grateful for your insightful feedback, which has significantly contributed to enhancing our research. Your detailed comments have enabled us to clarify key aspects and substantially strengthen our paper. We have provided separate and detailed responses to each of your comments and suggestions.

In response to your feedback, here are several important updates and clarifications, highlighted in **blue** in the revised draft:

- **Visualizations and Clarifications**: Updated Fig. 4, added Fig. 9 (sample window), Fig. 10 (Labits in various scenarios), and Fig. 12 (One-way Labits).

- **Bidirectional Structure Analysis**: Added Section A.3 in the supplementary material, along with updates to Table 4, Table 5, and Fig. 12.

- **Ablation Study on Time Bin Size**: Detailed in Section A.4 in the supplementary, reflected in Table 6.

- **Additional Comparative Experiments**: Described in Section A.5 of the supplementary, see Table 7 and Table 8.

- **More Efficiency Analysis**: Expanded in Section A.1 of the supplementary, see Table 3.

- **Tone and Overstatement Corrections**: Adjusted in the Abstract, Sections 1, 3, and 6.

- **Limitations and Future Directions**: Discussed in Section A.6 of the supplementary material.

We thank you again for your constructive reviews. We are committed to advancing this field and hope that our revisions meet your expectations and further demonstrate the value of our contributions.

Best regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732777889194

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Reviewers,


Thank you for your valuable insights and constructive feedback. We recognize the importance of validating Labits on diverse tasks and real-world datasets, including the two-frame optical flow estimation task with the DSEC dataset. We will work on this and provide detailed comparisons and results in our future updates, as time allows.

We acknowledge that while Labits has specific use cases and prerequisites, it is fundamentally designed to maximize the capabilities of event cameras. Our ambition is for Labits to act as a robust foundational basis that not only fosters further exploration but also encourages the broader research community to leverage its potential in various applications of event-based vision. For a detailed discussion of **the limitations and future directions**, please refer to **Section A.6** of the supplementary material. We trust that this work will inspire new insights and further research within the field.

We are grateful for your guidance and are committed to refining our work based on your insightful feedback.

Additionally, we have included a new table in our paper that compares Labits with more recent event representations (**Table 9**).

Sincerely,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
## Summary

The paper introduces Labits, a new representation method for asynchronous indefinite-length event data in modern networks. Labits preserves millisecond time information in raw data by capturing fine-grained temporal information, retaining stable 2D features, and maintaining consistent information density in spatiotemporal motion estimation. The authors also propose a specialized module, Labits-to-APLOF, which enhances local optical flow estimation accuracy at active pixels. Labits reduces trajectory end-point error by 49% over previous methods in the MultiFlow dataset, making it an effective tool for motion-sensitive tasks. Experiments on the MultiFlow dataset demonstrate its effectiveness.

## Strengths

* Presents a novel synchronous event representation, Labits, that preserves rich temporal information for motion-related tasks; demonstrating the benefits of this new representation over the commonly used voxel grid on the nonlinear dense trajectory estimation task.
* The paper is organized logically, distinguishing between the problem background, proposed solution (Labits), and experiments.
* Labits addresses temporal occlusion and provides a stable representation of movement information, useful for complex tasks like trajectory prediction, event-based video interpolation, and high-speed object tracking.
* Achieves SOTA performance on event-based dense trajectory estimation task.

## Weaknesses

* The differences between the dense tracking model and [1] need to be clarified.
* Technical justifications for the Labits representation could be expanded, including the use of bidirectional time surfaces. 
* Labits achieves efficiency gains but is slower than simpler methods like the Voxel Grid.
* The evaluation of Labits is confined to a single dataset (MultiFlow), limiting insight into its performance across different event-based vision tasks and datasets. Further experiments could be conducted to show the generalization capacity of the proposed solution.
* More detailed ablation studies and parameter analyses could be conducted.
* The proposed representation should be compared against some existing event representations to verify the superiority of the solution.
* The proposed method achieves high accuracy on MultiFlow, but it is a synthetic dataset.

## Conclusions
Based on the reviews and the author's feedback, the paper should include all the concerns addressed before accepting the paper. Mainly, as stated by three reviewers, they would still strongly suggest that the authors add some results on real-world datasets like DSEC-Flow. Even if it does not provide high-FPS ground truth, some qualitative results would be nice for enriching the analyses and helping understand the effectiveness of the proposed method. Some metrics like FWL which do not require ground truth could also be used. 
Moreover, there is a critical concern that were not discussed *Combining the responses to W1, W2 and W4, the core contribution of this paper is a new event representation, while the network structure applied to the trajectory estimation task is essentially the same as in [1]. The authors do not explicitly specify task-specific relevance of the proposed representation for the trajectory estimation task. Therefore, I argue that the new event representation should be validated on more tasks.* and the other two reviewers are also interested on this point.