Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles the problem of event data-based object tracking. Instead of using event frame/image representation as in the existing methods, the authors chose to process raw event clouds for event data-based object tracking. The proposed framework includes a downsampling strategy to extract key events from a raw event cloud, a graph-based network to embed spatio-temporal information of key-events, semantic and Motion-aware Target Likelihood Prediction, and a confidence-based object proposal. The authors conduct experiments on FE108 [49] and a new synthetic dataset Event LaSOT for tracking the evaluation of non-rigid objects. All the baselines and the proposed method are trained on FE108 and evaluated on the testing set of FE108 and Evt-LaSOT. The experiments prove the possibility of consuming raw event clouds for object tracking and achieving state-of-the-art performance on challenging datasets.
strengths_and_weaknesses: Strengths:
1. The authors focus on event-based object tracking, and we make the first attempt to construct a new end-to-end learning-based paradigm that directly consumes event clouds. The reviewer found the direction will interest those working on event-based cameras. Moreover, those who work on point cloud modeling might find a new problem to test the applicability and generalization of their methods.
2. The work contributes a network architecture that can consume a raw event cloud and demonstrate that the network can outperform other event cloud representations for object tracking on two challenging datasets.
3. Convincing tracking results are demonstrated in the supplementary material.
4. The implementation of this framework has been released.

Weaknesses:
1. 3D point cloud tracking: The proposed framework shares similarities with existing 3D point cloud tracking literature, e.g., Qi et al., 2020 and Zhou et al., 2022. It will be essential to summarize the difference between the literature pools. Moreover, the authors could analyze how the proposed architecture harvests characteristics of event cloud (e.g., inherent motion trajectories) where methods built for 3D point cloud tracking do not consider these aspects. Moreover, the experimental section could be further improved by comparing the proposed framework with a state-of-the-art 3D point cloud tracker, which aims to show the importance of considering data characteristics.
- Qi et al., P2B: Point-to-Box Network for 3D Object Tracking in Point Clouds, CVPR 2020
- Zhou et al., PTTR: Relational 3D Point Cloud Object Tracking with Transformer, CVPR 2022

2. Ablative studies: The reviewer found the contribution of the two target likelihood predictors unclear. Specifically, based on the implementation of the proposed architecture, the authors could have removed one of the predictions and tested the corresponding model. However, in Table 3, the performance of removing semantic-aware target likelihood prediction is missing.

3. STOA comparison: In Table 1, the proposed method demonstrates inferior performance compared to E-PrDiMP on the HDR setting. The discussion in Sec. 4.2 does not explicitly discuss it. It would be essential for the authors to provide the corresponding insights as they mention that event-based generally surpass RGB-based methods under LL and HDR scenarios.
questions: The current recommendation is Borderline accept (5). Overall, the authors indeed prove the claim that they design an architecture that consumes raw event cloud and achieves promising object tracking performance on two challenging datasets compared with STOA. The paper could be improved by discussing the difference between the proposed methods with existing 3D point cloud tracking literature. Moreover, it would be interesting to demonstrate that existing 3D point cloud trackers could not outperform the proposed method as it considers the characteristics of the event cloud.
limitations: Yes, the authors discuss the limitations of the work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper make the first attempt to construct a new end-to-end learning-based paradigm that directly consumes event clouds. Considered the high temporal resolution and sparse characteristics in event clouds, it proposed key-event and graph-based network to extract irregular spatio-temporal information from raw data. Then this paper proposes a dual-path architecture to predict the possibility of a key-event of the search event cloud. The dual-path architecture captures the similarity of each search key-event to the template key-events and utilize the information of motion trajectories respectively. This paper demonstrates the superiority of the proposed framework over state-of-the-art methods in terms of both the tracking accuracy and speed.
strengths_and_weaknesses: (1) The paper has a clear structure and is easy to understand the motivation and innovation.
(2) It proposed key-event and graph-based network to extract irregular spatio-temporal information from raw data, Making better use of the characteristics of event data.
(3) It developing a new end-to-end learning-based paradigm that directly consumes raw event data , in which new modules considering the unique features of event clouds.

Weaknesses：
(1) The specific technical details are not clear enough
(2) This paper mentioned the high temporal resolution and sparse characteristics of event data. The biggest motivation of this paper is to design data feature extraction module and algorithm model according to the unique attributes of data. But I didn't see a targeted design of the unique attributes of the Event data.

questions: (1) Motivation mentioned the high temporal resolution and sparse characteristics of event data. Where is the specific design for these two characteristics the data processing and model design.
(2) The ablation experiment did not discuss how much does the use of raw data contributed to the final results. Does removing the uniquely designed data processing part of your method make a big difference to the final result ?
(3) When comparing with SOTA, why do you choose some RGB-based methods to train yourself instead of comparing with other event-based methods?
(4) “Considering that trajectories are usually continuous, the former event points should have relatively smaller moving offsets” . The offset is derived from the projection of the key-event in the last step, and the key-events in the last step is not the same timestamp either.  And the time difference between the same key-event before and after is not sure. Is it unreasonable to have a relationship between the displacement and the timestamp?

limitations: The motivation is very convincing, but targeted design and description with respect to motivation are needed. The comparison experiment should also be focused on the points emphasized in the motivation.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Instead of converting events into 2D event frames/images, this work first sample a subset of key events and use GNN to transform them into high dimensional feature embeddings. The resulting feature embeddings are used to compute target likelihood following siamese fashion. In addition, motion-aware target likelihood is also computed to strengthen the matching process. Experiments on both synthetic and real event datasets demonstrate the its effectiveness.
strengths_and_weaknesses: 1. A novel event processing method which first samples large amount of events into a subset of key events and then uses GNN to transform them into feature embeddings. the embedding representations could preserve more task-specific information than others.
2. Different from previous event-based trackers that, the proposed method explore to utilize the motion information contained within the event could, which is very interesting.
questions: 1. The description about key-event sampling is not very clear. For grids with empty events, do they also try to find spatially closest events as key-events?
In addition, what if there are two events that both are closest events to the grid? 
Does this operation heavily compress temporal information within event streams?
2. Too many details are moved to supplementary material, which cause difficulty in understanding the whole method. For example, GNN-based spatio-temporal embedding, target likelihood prediction via motion trajectory back-tracking, and confidence-based object proposals.
3. In the experiment on synthetic data, would low fps of videos make the simulation less realistic and bring more gap between simulation and real-world data?
4. What about comparison with other event representations such as voxel grid and time surface?
5. It is not clear how to effectively supervise the motion-aware branch to get the correct motion? Is there any intermediate supervision or regularization?
6. In Fig. 3, what is the relationship between (a) (b) (c) (d)? In addition, it seems the motion vectors are noisy. Do the motion vectors can really reflect the movement triggering the corresponding key events?
7. The performance on Ev-LoSOT (PRP 30+) is very low, while the top RGB trackers (e.g. OSTrack)  have achieve performances higher than 70+ with realtime speed. It implies the experiment on Ev-LaSOT tend to fail. The SiamBAN is mentioned but is not compared in the experiments.
8. The result of FENet in Table 1 seems to be worse than the one reported in original paper.
limitations: Yes, authors have discussed the limitation of this work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Unlike existing methods that usually reorganize raw event data to solve the unusual data structure problem, this work develops new designs tailored to the unique data structure to realize object tracking. To this end, the authors construct a new end-to-end learning-based paradigm that directly consumes event clouds. The proposed method featured key-event embedding and motion-aware target likelihood prediction to take advantage of the unique characteristics of event data. Extensive experiments on both synthetic and real event datasets demonstrate the excellent performance of the proposed method in terms of both tracking accuracy and speed.
strengths_and_weaknesses: This paper concentrates on the unusual data structure problem and presents the first end-to-end learning-based object tracking paradigm on raw event data. Experiments illustrate that the proposed framework achieves competitive performance and efficiency compared to SOTA methods. The article is well-structured and easy to read. However, due to space limitations, some experimental details are not fully elaborated, and the analysis of experimental results can also be further strengthened. Please refer to Questions for details.
questions: 1.	What is the correspondence between subfigures in the upper and lower lines of Figure 3? Can you further explain how to understand the green arrow in the second row? (Number of arrows, consistency of direction, the thickness of arrows, etc.)
2.	In Section 4.1, what is the sequence length of the datasets used in this experiment (mean frame, min frame, max frame)? Does the sequence length have a greater impact on the experimental results?
3.	As far as I know, LaSOT contains 1400 sequences, and the sequence length varies from 1K to 10K frames, so how does the Evt-LaSOT construct in this article filter data from the original LaSOT? Are there specific standards? Is the filtered sequence have the same length as the original sequence?
4.	Table 2 presents the proposed algorithm has a higher FPS on Evt-LaSOT. The authors point out that the main reason is this method directly consumes raw event data without pre-processing. Is there any other reason besides this? Can you design some ablation experiments to analyze the processing speed?
5.	The video demo's main screen intuitively shows this method's effect, but what do the three sub-screens on the right refer to? Please explain the specific meaning of each sub-screen.

limitations: The authors discuss the public privacy issues in the additional material and argue that the event-based vision puts a lower threat to public privacy than RGB image. However, I want to know how secure and robust the event-based vision is? If we deploy the event-based scheme on self-driving systems, will it be safer than the current RGB image-based plan?
review_title: null

Meta-review: metareview: The paper receives overall positive reviews and rebuttal has resolved the reviewer's concerns. The paper proposes a new framework that directly takes raw event clouds as inputs for object tracking. Reviewers agree that this innovation is inspiring. AC agrees and recommends accepting the paper.
confidence: Certain
Decision: Accept