Learning Graph-embedded Key-event Back-tracing for Object Tracking in Event Clouds
Authors: Zhiyu Zhu
Junhui Hou
Xianqiang Lyu
Venue: NeurIPS 2022 Accept (2022)
Keywords: event data
object tracking
deep learning

Abstract: Event data-based object tracking is attracting attention increasingly. Unfortunately, the unusual data structure caused by the unique sensing mechanism poses great challenges in designing downstream algorithms. To tackle such challenges,  existing methods usually re-organize raw event data (or event clouds) with the event frame/image representation to adapt to mature RGB data-based tracking paradigms, which compromises the high temporal resolution and sparse characteristics. By contrast, we advocate developing new designs/techniques tailored to the special data structure to realize object tracking. To this end, we make the first attempt to construct a new end-to-end learning-based paradigm that directly consumes event clouds. Specifically, to process a non-uniformly distributed large-scale event cloud efficiently, we propose a simple yet effective density-insensitive downsampling strategy to sample a subset called key-events. Then, we employ a graph-based network to embed the irregular spatio-temporal information of key-events into a high-dimensional feature space, and the resulting embeddings are utilized to predict their target likelihoods via semantic-driven Siamese-matching. Besides, we also propose motion-aware target likelihood prediction, which learns the motion flow to back-trace the potential initial positions of key-events and measures them with the previous proposal. Finally, we obtain the bounding box by adaptively fusing the two intermediate ones separately regressed from the weighted embeddings of key-events by the two types of predicted target likelihoods. Extensive experiments on both synthetic and real event datasets demonstrate the superiority of the proposed framework over state-of-the-art methods in terms of both the tracking accuracy and speed. The code is publicly available at https://github.com/ZHU-Zhiyu/Event-tracking.

Learning Graph-embedded Key-event Back-tracing for Object Tracking in Event Clouds
Zhiyu ZHU, Junhui HOU; Xianqiang LYU
Department of Computer Science, City University of Hong Kong zhiyuzhu2-cone comy ci cityu. hkk ho.uityiiyu.... eyu.edu.......eee edu, hk, xianqialv2- eiiialllcmmmy c@my cityu.edu. hk
Abstract
Event data-based object tracking is attracting attention increasingly. Unfortu- nately, the unusual data structure caused by the unique sensing mechanism poses great challenges in designing downstream algorithms. To tackle such challenges, existing methods usually re-organize raw event data (or event clouds) with the event frame/image representation to adapt mature RGB data-based tracking paradigms, which compromises the high temporal resolution and sparse charac- teristics. By contrast, advocate developing new designs/techniques tailored to the special data structure to realize object tracking. To this end, we make the first attempt construct consumes event clouds. Specifically, to process a non-uniformly distributed large- scale event cloud efficiently, we propose simple yet effective censity-insensiie downsampling strategy to sample a subset called key-events. Then, we employ a graph-based network embed the irregular spatio-temporal information of key- events into a high-dimenniioal feature space, and the resulting embeddings are utilized to predict their target likelihoods via semantic-driven stamese-matching Besides, we also propose motion aware target likelihood prediction, which learns the motion flow to back-trace the potential initial positions of key-events and measures them with the previous proposal. Finally, we obtain the bounding box by adaptively fusing the two intermediate ones separately regressed from the weighted embeddings of key-events by the two types of predicted target likeli- hoods. Extensive experiments on both synthetic and real event datasets demon- strate the superiority of the proposed framework over state-of-ther methods in terms both tracking accuracy and speed. The code publicly available at https: //g thub. Soom/HU--- 2H1--Z----- Whiyn/Evett----eeeeeeer -
end-to-end learning-basee paradigm that directly
Introduction
Event cameras, the bio-inspired silicon retinas, have been widely adopted in computer vision and robotics communities [57, 37, 18, 31, 28, 47]. Different from traditional RGB cameras, which synchronously record the light intensity of each pixel, event cameras [13] only sense the changes in dynamic scenes, i.e., if an event senor captures an intensity increment (resp. decrement) larger than threshold, it would trigger an event, composed the corresponding pixel location, the timestamp, and the polarity indicating brightness increase decrease. With the concise information flow, event cameras enjoy numerous advantages, such as high temporal resolution, low latency, high dynamic range, and low power consumption. Moreover, recent works show that even without being associated with RGB images, only event data could also realize various vision tasks, such as depth estimation [14], semantic segmentation [1], and object detection [32].
"ooresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Notably, owing to the thove-mentioned advantages, event cameras are desired to be promising object tracking solution [39, 18, 50], aiming to estimate the position of a specified target object. However, the unusual data structure of the raw event data poses great challenges in designing tracking algorithms. Facing such challenges, the existing non-learning-base models based on various representations, such as event frame [40, 17, 29], time surface [33, 55, 7], and event clouds [2, 32, 45J. In view of the great success of deep learning techniques in various fields, especially RGB data-based object tracking, recently several deep learning-basee methods for event data-based object tracking have been naturally proposed [51, 47J. Generally, they quantize and accumulate raw event data along the temporal dimension into event frames/images to adapt to mature RGB resolution and sparse characteristics of raw event data to some extent, thus compromising tracking accuracy.
methods usually build their
object tracking paradigms. However, such a manner drops the high temporal
Instead of following the research line of the existing learning-hased methods, we are interested in developing a new end-to-end learning-band based paradigm that directly consumes raw event data (or event clouds), in which new algorithns/modules considering the unique features of event clouds are involved. Specifically, to cope with the irregular structure, we employ a graph-based neural network to map an event cloud into high-dimensional feature space to embed rich spatio-temporal semantic information. Meanwhile, to process a large-scale event cloud efficiently and being aware of its non-uninee non-uniform distribution characteristic, we introduce simple yet effective censity-insensitie downsampling strategy to sample a limited number of key-events, on which the feature embedding process is performed. We then link the embeddings of template and search event clouds via Siamese- matching to predict semantic-aware target likelihoods of search key-events. Besides, observing the high temporal resolution characteristic of event data naturally preserve motion trajectories, propose to back-trace the possible initial positions of the search key-event and measure them with the previous proposal to predict their target likelihoods. Finally, we obtain the bounding box by adaptively fusing the two intermediate ones separately regressed from the weighted embeddings of key-events by the two types of predicted target likelihoods. We conduct extensive experiments both real and synthetic event data to demonstrate the superiority of our framework over state-of tottt----th--attt the-art methods in terms of both the tracking accuracy and speed, and comprehensive ablation studies to validate the effectiveness of the proposed modules.
also
In summary, the main contributions of this paper are two-fold:
we propose the first end-to-end learning- object tracking paradigm directly handling raw event data (or event clouds), featured with earefully-designed modules, c.g., key-event embedding and motion-aware target likelihood prediction, to take advantage of the unique characteristies of event data; and
The superiority performance of our paradigm certifies the great potential of designing new algorithms tailored to the special characteristics of event data to achieve object tracking. Besides, we will release our framework built from scratch contribute this community.
2
Related Work
Deep learning on event data. As kind of asynchronous bio-inspired vision sensors, event cam-
eras [13] have shown remarkable performance in various scenarios, such as quick motion and low
light environments, representing a paradigm shift in visual information collection [9]. Deep learning- based frameworks for modeling event data have been proposed. For example, Phased LSTM [36] extends the LSTM unit by adding a new time gate for classification on event data. EVDL [31] proposes an wvent-frame representation and constructs a CNN model for prediction of a vehicle's steering angle. Then, EV-SegNet [1] introduces the first baseline for semantic segmentation with event data. The emergence of MVSEC [57] dataset has aroused wide interest in the research of optical flow [56. 58, 16] and depth estimation [58, 20J. Unlike the previous works, Rebecq et al. [41, 42] used simulated events to train a convolutional recurrent neural network for image reconstruction.
RGB data-based object tracking. Recent methods could be roughly divided into two categories: spatio-only-base and spatio-temporull-assee
Most of offline Siamese-based tracking methods [3, 26, 25] fall into the first category. For example, SiamFC [3] enhances the tracking accuracy by first introducing a correlation layer as a fusion tensor. SiamRPN [26] employs a single-stage RPN [43] detector to redetect the template by comparing its features to that of the current frame.
2

---

SiamRPN++ [25] uses a spatial-aware sampling strategy to remove the influence factors, such as padding, and introduce ResNet [19] into the Siamese network-based visual trackers. However, the spatio-only methods are difficult to deal with fast-moving or similar objects in the scene.
Compared with spatio-only trackers, spatio-temporal- based methods [35, 23, 46, 49] additionally exploit temporal information to improve trackers' robustness. For example, MDNet [35] proposes CNN-based multi-domain learning framework, which separates Nomain-independen information from domain-speciin one, to capture shared representations. RT-MDNet [23] improves MDNet by employing the RoIAlign technique to extract more accurate representations of targets and candidates from a feature map. Recently, Wang et al. [46] converted a general object detector to achieve domain adaption and efficient online update. However, these methods are either difficult balance performance and speed or cannot handle real-time scenarios and low light scenes well.
Although various advanced techniques have been proposed for boosting the performance of RGB data-based object tracking, the essentially different data modality requires additional efforts to extend these techniques to event data-based object tracking.
Event data-based object tracking has been receiving increasing attention recently, and the proposed methods could be broadly divided into two categories: optimization-based and learning-based
For the optimization- -aase category, as the pioneering work, Mitrokhin er al. [32] proposed to approximate the 3D geometry of the event stream without feature tracking or explicit optical flow computation. Bryner al. [5] presented a method track the 6-DOF pose of an event camera in a known environment described by a photometric 3D) map.
Ramesh et al. [39] presented the first leamiig-bassed long-term object tracking algorithm for event cameras, based on a
framework. Then, Li et al. [27] utilized the features by VGG-Net-16 16 to represent the appearance of the event-stream object achieve tracking. Recently, Chae et al. [6] applied a Siamese-matching paradigm [3] to the event-based object tracking task for learning edge-aware similarity. However, the methods in [27, 6] mainly focus on applying well-developed RGB basa- tracking techniques to event data without well taking the special characteristics of event data into account.
3 Proposed Method
Notation. Denote by e [xr ys 1, R an event, where and y are the spatial coordinates on the image plane, whose values are normalized to j0, 1]. t is the timestamp, and is the 1-bit polarity. indicating brightness increase or decrease. Denote by E E R x4 an event cloud, where N events in short period of time [fo. to + A,] are randomly stacked in row-wise. Accordingly, for events in long period of time, we represent them with an event cloud sequence of U non-overlapping
J i=='' Let E, be template event cloud, and Bd E R x4 be the annotated bounding box indicating the 2-D position of the object captured by E, N.4, which is parameterized by the 2-D spatial locations and the corresponding size (i.e., width and height). The objective of event-based object tracking is predict the bounding box of target object described by Bd and E, from each of given search event cloud sequence (E E N., X4 U.
event clouds, i.c., [ E RNN444
U
1=1-
Overview. In contrast to existing learning-based methods usually re-organizing raw event data with the frame-based representation, which compromises the high temporal resolution characteristic to some extent and thus limits tracking performance, we directly deal with raw event data in its intrinsic 3-D aaatio-temprral space. Specifically, shown in Fig. 1, the proposed framework processes the event clouds of EE, L sequentially via the following three main steps:
Graph-based Key-event Embedding. This module aims project an event cloud into a high- dimensional feature space to embed rich ppatio-temporal semantic information (see Sec. 3.1). However, the irregular structure underlying event clouds, caused by the sparse and asynchronous sensing manner, makes it intractable to learn the embeddings efficiently and effectively. To this end, we employ GNN, which has demonstrated the effectiveness in modeling irregularly distributed signals, to embed input raw event data. Moreover, to process large-scale event clouds efficiently and being aware of their noisy and highly non-uniform characteristics, we propose a simple yet effective downsampling strategy, capable of sampling a few key-events evenly and robustly from an input
3

---

Frampler
Key-event embedding
Fe
Target likellhood prediction
MLP
Beem
Object proposal
Eseareh
Semantic-driven Siamese-matching
Sem
semm
f share
F
Fusion weight
My
Be
H
mon
Key-event embedding
Motion trajectory Back-tracking
mmon
Byon
(b)
(c)
Etemplet
Fesrrhh
Workflow
Object motion
Multiplication
Target key event
Initial position
Key event
Graph aggregation
Trajectory
Bounding box
Av Motion flow
Figure Flowchart of the proposed end-to-end toaming-baseed framework for tracking objects in event clouds. It mainly consists of three steps: GNN-based key-event embedding, semantic matching and motion-aware target likelihood prediction and confidence-based object proposal. Sub-figs. (a), (b), and (c) illustrate the detailed processes of the first two steps, marked with corresponding color.
event cloud. Accordingly, we perform the tracking process on key-events instead of the whole event cloud.
Semantic and Motion-aware Target Likelihood Prediction. This module, dtual-path architecture, aims to predict the possibility a key-event of the search event cloud belonging to the target object, i.e., the target likelihood. Basically, from the visual recognition perspective, propose a semantic- module, in which the similarity each search key-event to the template key-events is evaluated, based their embeddings (see Sec. 3.2). Besides, observing that the event cloud inherently preserves the motion trajectories of events, we also propose a motion-aware target likelihood prediction module, in which the initial position of each key-event is back-traced and compared with the previous proposal (see Sec. 3.3). We expect these two paths designed from different perspectives will compensate each other for robust and accurate target likelihood prediction.
driven Siamese-mater
Confidence-bate Object Proposal. In this module, we first regress two intermediate bounding boxes separately from the embeddings of key-events weighted by the two types of predicted target likelihoods and then fuse them adaptively under the guidance of the learned confidence scores, generating the final bounding box (see Sec. 3.4).
3.1 Graph-based Key-event Embedding
We first sample a limited number of key-events via a simple yet effective density-insensitvee down- sampling strategy, then embed them into a high-dimensional feature space via a GNN.
Key-event sampling. As shown in Fig. 2 (a), to sample n& key-events from typical event cloud, we first define a 2D regular grid that contains ng grid points and has the same scale as the event cloud in the spatial domain. Then, for each of the grid points, we find the spatially closest event in the Euclidean distance sense, which will be thought of as a key-event. Such a simple downsampling strategy could spread the receptive field of the neural network in more spatially uniform manner. We demonstrate its superiority against widely-used random sampling [21] in Sec. 4.3.
GNN-based spatio-temporal embedding. We utilize the graph convolutional layers encode the irregular 3-D ppatio-temporal information of an event. Specifically, as shown in Fig. 2 (a), for each key-event, we first figure out its nearest neighbours, then learn the high-dimensional spatio-temporal embeddings of such a set events via feeding the explicitly encoding" of event clouds to multi-layer perceptrons (MLPs). Meanwhile, adaptive pooling weights are also learned to aggregate neighbouring information in an unordered manner, finally producing embeddings F, E R XC and F, E RR, x for E, and E containing n4 and nt, key-events, respectively, where C' is the number of channels. We refer readers to Appendix for more details.
"The encoding refers to the concatenation of the central event point e., its neighbour e.. the difference Cn. and the Euclidean distance |/e, enll2.

---

Event intensity
Projected events
Object
Trajectory
Key events & its projection
KNN events
+ Sample grid
Motion flow
X
y
(a)
(b)
Figure 2: Illustration of (a) the density-insensitiv event downsampling strategy and local spatio- temporal aggregation pattern of the graph neural network, and (b) the key-event back-tracing process used in motion-aware target likelihood prediction.
3.2 Target Likelihood Prediction via Semantic-driven Siamese-matching
Let Prem E [0,1] be the target likelihood of the j-th search key-event, and Psem :Peem' :.mmi""..PPe'"""" former, which has shown its great potential in sparse point cloud processing [52], first apply swin-transformer [30] on the embeddings F, and F,, which respectively encode the semantics of the key-events of E, and E; equip them with abstract and distinguishable information, generating S, E IR and S. E RR,xCC (C, > C). To link E, and E,, we then design a cross-attention layer, which adaptively projects S. onto S.:
: :P"em] E R Encouraged by the learnable combination pattern of trans-
S M, x S., M. = G(),
where the associated matrix M. E R" is learned via another GNN G(:,)) and St, E R. x.. denotes the projected embeddings blending the information of both E, and E,. Finally, feed the concatenation of St, and S, into another MLP to regress P sem- We refer readers to Appendix for the detailed architecture.
3.3 Target Likelihood Prediction via Motion Trajectory Back-tracking
In this module, we leverage the inherent motion trajectories of event data [58, 34, 40] to predict the target likelihood of each key-event of E, The basic idea is linking the current search key-events to the previous target proposal to distinguish them. Specifically, we first back-trace the initial position of each key-event from its motion trajectory in 3D spatio-temporal space. Considering that trajectories are usually continuous, the former event points should have relatively smaller moving offsets. To explicitly model such characteristic, apply MIPs the embeddings of search key-events to learn their motion velocities V Ia, x2 which are further used to calculate the motion flow M E g i.c.
M x T (T), V ML.(F...
(2)
where T x records the timestamps of the search key-events, and max(-) is the operation selecting the maximum element of the input. With the motion flow and the positions of the search key-events, denoted K R can localize their initial positions, denoted by Kini g=.x2 as K,n- M + K. Finally, we predict the motion-aware target likelihood of each search key-event, denoted pinon E [0, 1] (accordingly, Prmon := Iphon Pimnn :Phoon E Rn.x1., from
Kini together with the previous bounding box and a normalized distance (#-xro) + [w--
As
independently via MLPs, where Th, and yh are the spatial coordinates of the box center, and wa and fú are the width and height of the box, respectively. We refer readers to Appendix for the detailed architecture.
3.4 Confidence-based Object Proposal
This module regresses a bounding box from the predicted two kinds of predicted target likelihoods P sem and P Pons well the embeddings F, encoding the rich spatio-temporal semantic informa-
5

---

tion of E', Specifically, taking the semantic- branch as an example, we formulate the bounding box regression as
Beem MLP(H sem), Hem
Mp Peem
x (F, DP semm). }), M Poem
S(MLP & OP sem )));
(3)
where Beem E RRxx denotes the intermediate bounding box, based on P Psem= H E Rixc encodes the embeddings after adaptive pooling. Mp.co. RRxxx is the pooling weights, denotes the hadamard product, D(-) is the duplication operation, and S(-) stands for the softmax function. We also apply the same procedures the motion-aware branch, leading to another intermediate bounding box Bmon Finally, we fuse the two intermediate bounding boxes under the guidance of learned confidence scores, i.e.,
B, = M2 X B sem: m.oll M, = ({MMPP(H ML
(4)
where B, denotes the final output bounding box, and M, E RIX2 refers to the learned confidence scores. We refer readers Appendix for the detailed architecture.
3.5 Training & Online Tracking
Loss function. To train the proposed framework end-to-end, we design the following loss function:
CB g: B.P,
P) g.
Σ
Las(P Px)
Σ (Creg (By. B gion (B +g))+ (5)
PLEP
BRE
where B- = [B., Beem; Bynon J. P= (Pomm P P. Bg and Pg are the ground-truth bounding box and target likelihood, respectively, (.) is the binary cross-entropy loss to supervise the target likelihood prediction, (reg('s) refers to the Li loss to supervise the regression of the bounding box, ( -): is the generalized intersection over union (GIOU) loss [44], and LOD computes the Chamfer distance between the back trace key-events and their potential locations to supervise motion flow estimation, defined
CCD
Σ Tsss K.EK.
1
min M(llk:-kell kiEK,..
Σ min M(lkk - -elli
(6)
The K.EK.nt
K,EK,
where Kini E 2 denotes the back-projecter key-events. We clip the event clouds within very tiny time interval K. Ra,x22 to t. S to + μd,, as the potential locations of Kini- Due to the inevitable noise events, we truncate the accidental errors by a threshold y via a minimization function M(-). we empirically set the syper-parameters a, 3, μ,, and to le". le 1,,, -e-1, and le-1, respectively.
Robust training strategy. In addition to E, and E;, we also have to feed the proposed framework with the previous bounding boxes. Accordingly, the motion-aware target likelihood prediction may highly rely on the accuracy of previous predictions. To mitigate the potential accumulated errors and improve the robustness of our framework, we disturb the input bounding boxes with random Gaussian noises during training.
Template updating strategy for online tracking. Considering that the appearance of an object may undergo large variance during a tracking process, we also introduce a strategy to update template key-events. Specifically, we first partition the embeddings of key-events of E,, i.e., S,, into two sets equally: the constant set Ste and the variant set S Then, after each search procedure, we stack in the embeddings of the key-events of E (i.e.. S, from) with high target likelihood, i.e., p > т, where r is empirically set to 9e-1, to S., and randomly pop out embeddings from St, to keep the total size of S, unchanged.
Experiments
4.1 Experiment Settings
Datasets. We evaluated the proposed method with both real and synthetic event datasets. The real event dataset FE108 (MIT) [51] provides event data of spatial dimensions 346x 260, dynamic range 120dB, and minimum latency time 20 jas, captured by the DAVIS 346 camera. The ground-truth bounding boxes were captured by the Vicon motion capture system. FE108 contains 108 sequenees

---

(c)
(d)
Ground truth
Ours
TransT-EF3
TransT-EF1
Amplification
Motion flow direction
Figure 3: Upper row: visual comparison of the tracking results of different methods on four event clouds randomly selected from the FE108 dataset. Bottom row: a local zoom-in view of the dashed box in corresponding upper sub-fignu sub-figures. The green arrows start from the spatial locations of key-events and point the direction of motion flow.
of total length 1.5 hours, recording the motion of 21 different types of objects under four challenging scenarios. There are 59K event clouds for validation and 140K event clouds for training. Considering that most objects in FE/08 are rigid, which may be insufficient to validate the performance on non- rigid objects, also constructed the widely-used RGB data-based object tracking dataset LaSOT (Apache-2.0) [12] via Vid2E[15]. Evi-LaSOT consists of eight sequences' randomly drawn from LaSOT and there are 18K event clouds in total.
synthetic dataset Event LaSOT' (Evt-LaSOT), simulated from
Compared methods. We compared the proposed method with SOTA RGB data-based object tracking algorithms, including SiamRPN [26], ATOM [10], DiMP [4]. SiamFC++ [48], and PrDiMP [11]. and the event data-based branch of FENet [51]. Moreover, we modified two SOTA RGB data-based object tracking methods, i.e., PrDiMP [8] and TransT [8], to adapt them to the event frame representation [51], leading two strong baselines, namely E-PrDiMP and E-TransT. For fair comparisons, trained all methods with the training set of FE108 and evaluated them on the testing set of FE/08 and EvS-L.
Implementation details. To train our framework a batch manner, we sampled 10K event points for each event cloud. We set the value of A the interval of annotations, i.e., 25 50ms for the FE108 dataset according to its 20/40 FPS annotations. We implemented the framework with Pytorch on Ubuntu 18.04 and trained it with 4 x NVIDIA-3090 and batch size of 22 samples in each GPU, about 300K iterations. We adopted the Adam optimizer [24] with the weight decay 1e 15- and the learning rate of 1 (resp. 1 for the backbone (resp. the two target likelihood prediction and object proposal modules). We set the spatial search region of the current event cloud as 1.2x the size of the previous proposal.
4.2 Results
Results on FE108. Table and Fig. (a) and (b) show the quantitative comparisons of different methods FE108, where it can be seen that the event-based methods generally surpass the RGB data- based methods under the LL and HDR scenarios, demonstrating the advantage event data. Besides, the significant improvement of E-TannTT with the input changed from EF-1 to EF-3 demonstrates that the high temporal resolution characteristic of event data of importance. Our method achieves the
'The eight sequences are "bird- 15", "bird-3", "crab-6", "crab- 18". "cat- 20", "cat-3", "crocodile-3" 3", and "bear-4".
7

---

Table 1: Quantitative comparison on the FE108 dataset in terms of four commonly-used metries, ie., representative success rate (RSR), representative precision rate (RPR), and overlap percision (OP) with the threshold equal to 0.5 (OP ) and 0.75 (OPo.75). For all metrics, the larger, the better. There are four challenging scenarios, i.e., low-light (LL). high dynamic range (HDR), fast motion with and without motion blur on image frame (FWB and FNB), and all testing dataset (ALL). "Img" (resp. "EF-n") indicates that the input of methods the RGB image/video (resp. the event frame/image), where n refers to the number of channels or the temporal resolution. "RawE" stands for raw event data (or event clouds). The best and second best results of event-based methods are highlighted in red and green, respectively.
Metrics
ATOM [10] DiMP SiamFC++ [48] PrDiMP [1] PrDiMP [11]
T-TanssT [8]
E-TransT [8]
FENet [51 Ours
Input
Img
Img
Img
Img
EF3
EF-1
EF-3
EF-3
RawE
RSR
32.3
41.8
15.3
44.3
42.4
50.7
50.0
50.9
OPo.s
36.9
50.0
15.0
52.8
62.5
51,4
60.8
57.8
60.7
HDR
OPo.
11.8
17.9
1.3
19.6
23.8
17.5
23.0
18.8
17.2
RPR
55.2
62.7
25.2
66.3
84.6
70.5
84.2
77.4
81.5
RSR
38.6
45.6
13.4
44.6
$8.0
46.8
58.8
56.6
60.6
OPo.sa
44.3
52.8
8.7
48.2
67.8
55,9
72.2
69.1
75.0
L..
OPo.7s
13.2
11.2
0.8
8.9
28.7
25.0
31.4
22.9
30.8
RPR
59.6
69.5
15.3
69.5
86.2
72.6
92.2
86.1
90.1
RSR
44.1
69.4
28.6
67.0
43.3
31.6
46.8
59.3
60.8
OPo.s0
50.9
94.7
36.3
89.9
52.9
34,3
57.9
74.2
76.3
FWB
OPo.rs
16.7
37.1
6.0
33.6
19.6
0.09
28.4
32.1
29.8
RPR
88.6
99.7
48.2
99.7
81.6
77.0
87.9
84.9
90.6
RSR
57.1
60.5
36.8
60.6
30.7
$1.8
50.2
54.3
OPo.s
68.2
75.6
42.7
75.8
51.0
35.7
60.2
56.3
61.8
OPo.rs
32.1
29.3
7.4
29.7
18.9
12.9
24.8
19.0
18.7
RPR
87.9
93.2
63.1
93.3
80.0
52,9
85.8
75.5
87.0
RSR
40.6
52.6
23.8
53.0
$2.0
39.1
52.4
52.3
54,9
ALL OPo.so a0
47.8
65.4
26.0
65.0
60.9
46.2
62.2
61.4
65.8
OPo.75
17.3
23.4
3.9
23.3
22.6
16.6
21.0
19.8
21.4
RPR
67.0
79.1
39.1
80.5
84.6
67.2
87.0
80.0
85.9
best or second best performance terms of different metrics under most scenarios, and especially, it significantly exceeds the compared methods under the fast motion scenario. The reason may be that the compared methods are disturbed by the large motion offset in the accumulated event frames, shown in Fig. 3 (b). Although the proposed method is slightly worse than or comparable to the compared event frame-based methods under very rare scenarios, we want to notice that those methods benefit lot from the RGB data-based tracking, long-standing field featured with great efforts devoted and mature techniques, while our new paradigm is still plain to some extent, and highly expected that its performance raises gradually with more advanced modules incorporated in the future. Results on Evt-LaSOT. Since these synthetic event data from RGB data do not have high temporal resolution and are driven from estimated optical- the real movement of an object, for the sake of fairness, we only compared the proposed method with event- based methods on this dataset. shown in Table 2 and Figs. (c) and (d), the proposed method trained with real event data from FE108 is well generalized on Evr-LaSOr and outperforms all the compared methods to a significant extent in terms of all the three metrics, which is credited to the new paradigm of directly modeling raw event data. Besides, since our method directly consumes raw event data without pre-processing required by existing event frame/image- based methods, more efficient, reflected by the higher FPS. We also refer readers to the video demo contained in Supplementary Material for the visual comparison different methods.
which may correctly reflect
Comparison with 3D point cloud-bssd trackers. Recently 3D point cloud- based object tracking achieves remarkable growth, which aims to predict the location of target object from a 3D point cloud (PC) sequence [38, 54, 22, 53]. Although event clouds could be thought of as 3D point clouds, they are still different. Specifically, 3D point clouds by the LiDAR sensor record the geometric information of 3D entities, which are generally collected by measuring the reflection time of the emitted pulses of infrared light hitting nearby objects/scenes. Only the part of the object/scene visible to the LiDAR sensor will be perceived (one emitted pulse will generate at most one 3D point). Thus, the captured points are distributed on the surface of the 3D entity. However, even data are acquired by measuring the pixel intensity variations in short time period, and for typical pixel, there may

---

Table 2: Quantitative comparison on the Evt-LaSOT dataset. FPS refers to the number of times of prediction per second processed during inference. We refer readers to Appendix for the metric details. The best results are highlighted in bold.
Metries
E-PrDiMP [11]
E-TransT [8]
FENer [31]
Ours
FPS
15.7
18.7
17.2
25.2
RSR
27.9
30.3
28.7
32.3
Evt-LaSOT
OPo.5
12.3
17.9
15.4
22.1
RPR
25.4
30.1
26.9
35.9
04
Eheap
Lod MP
TESt
Seaccmen Rutt
Locaoon Ecrn Throheld
Onerla Tlmnaad
Nanaled Saran Thrnkei
Omelap Thmakid
(A)
(4)
Figure 4: The precision and success plots of different methods, where sub-figure (a) and (b) shown for FE108, (c) and (d) for Evt-LaSOT.
be multiple events (i.e., the events have the same x and dimensions but different /. dimensions).
Thus, the events are distributed within a 3D cube. Such a data distribution difference may make it inappropriate to apply 3D point cloud-based trackers to event data. Here, also conducted experiments to compare the proposed method with SOTA 3D point cloud-based trackers on FE108 dataset. As listed in Table 3, the proposed method exceeds the 3D point cloud-bases trackers [38, 54] to a large extent, i.e., 7.4% and 15.5% improvements in terms of the success rate and precision respectively, demonstrating the demand of designing event-specialized tracking algorithms.
4.3 Ablation Studies
Event downsampling strategy. We constructed a baseline by replacing the proposed downsampling strategy with random sampling. By comparing rows 4) and 6) of Table 4, it can be seen that without using the proposed downsampling strategy, the values of RSR and RPR decrease by 4.8% and 8.4%, respectively, validating the effectiveness of this strategy. Moreover, such a downsampling process can accelerate the network convergence during training.
Motion-aware target likelihood prediction & Training strategy. To validate the effectiveness of this module, we constructed a baselines without using As shown rows 1). 2), and 3) of Table it can be seen that this module augmented by the training strategy achieves 2.7% gains in terms of RSR. Besides, without using the training strategy, this module even harms the accuracy, validating the effectiveness of the training strategy on mitigating the accumulated errors.
Flow estimation module. By comparing rows 3) and 6) of Table 4, it can be seen that this module improves the value of RSR by 1%, validating its effectiveness. Besides, we also visualized the flow
Table 3: Comparison of the proposed method and 3D point cloud-based trackers the FE108 dataset. We trained the 3D point dlud-based trackers with the same experiment settings as ours and also tried best to adapt those methods to event data for optimal performance.
method
RSR
OPo.50
OPo.75
RPR
P2B [38]
36.1
35.4
4.7
40.5
PTTR [54]
47.5
49.6
11.5
70.2
Ours
54.9
65.8
21.4
85.9

---

Table 4: Quantitative results of the ablation studies of the proposed method. "x" (resp. v) means the corresponding module is unused (resp. used). (a) Downsampling strategy (b) Injecting noise during the training phase (c) Flow estimation (d) Motion-awar target likelihood prediction (e) online template updating strategy (f) Semantic-aware target likelihood prediction. Note that (b) and (c) are applicable only if (d) is used.
(a)
(b)
(c)
(d)
(c)
(f)
RSR
O0
OP0.75
RPR
I)
V
X
X
52.2
61.5
14.2
85.0
2)
X
51.7
61.2
14.5
83.7
3) V
V
V
V
v
53.9
64.0
19.9
84.9
4) X
V
50.1
59.5
19.3
77.5
5)
V
V
v
54.5
65.2
21.0
85.7
6) V
V
54.9
65.8
21.4
85.9
of key-events Fig. 3, where it can be seen that although there are some noise events, the predicted flow could correctly present the motion of both the background and target object.
Online template updating strategy. The effectiveness of this strategy is validated by comparing rows 5) and 6) of Table 4, where it can be see that the value of RSR improves by 0.4% when including this strategy.
4.4 Discussions
Limitations: although the proposed method demonstrates the great potential of raw event data-based object tracking, there are still some promising directions that are worth studying to further boost tracking performance. First, noisy events inevitable, shown in Fig. 3, which may disturb the target likelihood prediction; hence, a typical event data denoising algorithm could be applied as pre-processing to remove the interference. Second, in addition the prior knowledge of the continuous motion trajectory preserved in event data, the velocity information of an object hidden in event data, which may also be continuous, could be taken into account. Third, for the long-term object tracking. it is necessary to investigate more delicate template updating strategies. Finally, according RGB data-based object tracking, the powerful pre-trained backbones are essential for improving tracking performance, and thus, it is highly desired to introduce pre-trained backbone on event data into event-based object tracking.
5
Conclusion
We have presented the first end-to-end learning-basd object tracking paradigm on raw event data. Owing critical modeling of the tracking process for exploring the special characteristics of raw event data, including the delicate downsampling method, the motion-aware module, and the noise injecting training strategy, the proposed framework achieves competitive performance and efficiency in comparison with state-of-the-ar methods.
Last but not least, despite that we demonstrate the great potential of designing new algorithms tailored to the special characteristics of event data to achieve object tracking, as beginning of this research line, our framework is still relatively plain, compared to the well-developed RGB data-based object tracking, and it could be expected that the performance will improve gradually with more efforts and advanced techniques incorporated in the future.
Acknowledgements. Funding direct support of this work: Hong Kong Research Grants Council under Grant 1202320, Grant 11219422, and Grant 11211121.
References
[1] Inigo Alonso and Ana C Murillo. Ev-segnet: Semantic segmentation for event-based cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
10

---

[2] Ryad Benosman. Charles Clereq, Xavier Lagorce, Sio-Hoi leng. and Chiara Bartolozzi. Event-based visual flow. IEEE Transactions Neural Networks and Learning Systems, 25(2):407-417, 2013.
[3] Luca Bertinetto, Jack Valmadre. Joao F Henriques. Andrea Vedaldi, and Philip HS Torr. collyyconvolutionall siamese networks for object tracking. In Proceedings of European Conference on Computer Vision, pages
850-865, 2016.
[4] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning discriminative model prediction for tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1191,1191, 2019.
[5] Samuel Bryner, Guillermo Gallego, Henri Rebecq. and Davide Scaramuzza. Event-based, direct camera tracking from a photometric 3d map using nonlinear optimization. In Proceedings of International Conference on Robotics and Automation, pages 325-331, 2019.
[6] Yujeong Chae, Lin Wang. and Kuk-Jin Yoon, Siamevent: Event-based object tracking via edge-aware similarity learning with siamese networks. arXiv preprint arXiv:2109. 13456, 2021.
[7] Haosheng Chen, Qiangqiang Wu, Yanjie Liang, Xinbo Gao, and Hanzi Wang. Asynchronous tracking- by detection on adaptive time surfaces for event-based object tracking. In Proceedings of the 27th ACM International Conference on Multimedia, pages 473-481, 2019.
[8] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan La. Transformer tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8126-8135, 2021.
[9] Zhiwen Chen, Jinjian Wu, Junhui Hou, Leida Li, Weisheng Dong, and Guangming Shi. Eesnet: Spatio- temporal feature learning for event camera. IEEE Transactions on Circuits and Systems for Video Technology, 2022.
[10] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. Atom: Accurate tracking by overlap maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4660 4669, 2019.
[11] Martin Danelljan, Luc Van Gool, and Radu Timofte. Probabilistic regression for visual tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7183-777777 7192, 2020.
[12] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In Proceedings of the IEEE/CVF Conference
Computer Vision and Pattern Recognition, pages 5374 5383, 2019.
[13] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. Event-base, vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):154- 180, 2020.
[14] Guillermo Gallego, Henri Rebecq, and Davide Scaramuzza. A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3867-3876, 2018.
[15] Daniel Gehrig, Mathias Gehrig, Javier Hidalgo-Carie
and Davide Scaramuzza. Video events: Recycling video datasets for event cameras, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1386-3595,, 2020.
[16] Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. End-to-end learning of representations for asynchronous event-based data, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5633-5643, 2019.
[17] Daniel Gehrig, Henri Rebeeq, Guillermo Gallego, and Davide Scaramuzza. Ekit: Asynchronous photomet- ric feature tracking using events and frames. International Journal of Computer Vision, 128(3);601-66 -618, 2020.
[18] Arren Glover and Chiara Bartolozzi. Robust visual tracking with a freely-moving event camera. In Proceedings of 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3769-3776, 2017.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[20] Javier Hidalgo-Carrió, Daniel Gehrig, and Davide Scaramuzza. Learning monocular dense depth from events. In Proceedings of 2020 International Conference on 3D Vision (3DV), pages 534-542. IEBE, 2020.
[21] Qingyong Hu, Bo Yang. Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang. Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. In Proceedings of the IEFF/CVF Conference
Computer Vision and Pattern Recognition, pages 11 11108-11117, 2020.
[22] Le Hui, Lingpeng Wang, Mingmei Cheng, Jin Xie, and Jian Yang. 3d siamese voxel-to-bev tracker for sparse point clouds. In Proceedings of Advances in Neural Information Processing Systems, volume 34, pages 28727. 2021.
[23] Ilchae Jung, Jeany Son, Mooyeol Back, and Bohyung Han. Real- time mdnet. In Proceedings of European Conference on Computer Vision, pages 33-98,22 2018.

---

[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint urXiv:1412.0 6980, 2014.
[25] Bo Li, Wei Wu, Qiang Wang. Fangyi Zhang, Junliang Xing, and Junjie Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4282-4291. 2019.
[26] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. High performance visual tracking with siamese region proposal network. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8971-8988 2018.
[27] Hongmin Li and Luping Shi. Robust event-based object tracking combining correlation filter and cnn representation. Frontiers in neurorobotics, 13:82, 2019.
[28] Jianing Li, Xiao Wang. Lin Zhu, Jia Li, Tiejun Huang. and Yonghong Tian. Retinomorphic object detection in asynchronous visual streams. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[29] Min Liu and Tobi Delbruck. Adaptive time-slice mlock-natching optical flow algorithm for dynamic vision sensors. In Proceedings of the British Machine Vision Conference, 2018.
[30] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[31] Anal I Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso Gareía, and Davide Scaramuzza. Event- based vision meets deep learning steering prediction for self-driving cars. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 5419-5427, 2018.
[32] Anton Mitrokhin, Cornlla Fermüller, Chethan Parameshwara, and Yiannis Aloimonos. Event-based moving object detection and tracking. In Proceedings of IEEE/RSJ International Conference Intelligent
Robots and Systems, pages 1-9, 2018.
[33] Elias Mueggler, Chiara Bartolozzi, and Davide Scaramuzza. Fast event-based corner detection. In Proceedings of the British Machine Vision Conference, 2017.
[34] Elias Mueggler, Guillermo Gallego, and Davide Scaramuzza. Continuous-ting trajectory estimation for event- event-based vision sensors. Technical report, 2015.
[35] Hyconseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for visual tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4293-4302 2016.
[36] Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased Istm: Accelcrating recurrent network training for long or event-based sequences. In Proceedings of Advances in Neural Information Processing Systems,
pages 3882-3890. 2016.
[37] Etienne Perot, Pierre de Tournemire, Davide Nitti, Jonathan Masci, and Amos Sironi. Learning to detect objects with megapixel event camera. In Proceedings of Advances in Neural Information Processing Systems, volume 33, pages 16639-16652, 2020.
[38] Haozhe Qi. Chen Feng. Zhiguo Cao, Feng Zhao, and Yang Xiao. P2b: Point-to-box box network for 3d object tracking in point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6329-6338, 2020.
[39] Bharath Ramesh, Zhang, Zhi Wei Lee, Zhi Gao, Garrick Orchard, and Cheng Xiang. Long-term object tracking with moving event camera. In Proceedings of the British Machine Vision Conference, 2018.
[40] Henri Rebecq, Timo Horstschäfer, Guillermo Gallego, and Davide Scaramuzza. Evo: geometric approach to event- based 6-dof parallel tracking and mapping in real time. IEEE Robotics and Automation Letters, 2(2):593-660 600, 2016.
[41] Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. Events-to-videoo Bringing modern computer vision to event cameras, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3857-38666 2019.
[42] Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. High speed and high dynamic range video with event camera. IEEE Transactions Pattern Analysis and Machine Intelligence, 43(6): 1964):994-109 1984-19800 2019.
[43] S Ren, K He. R Girshick, and J Sun. Faster r-can: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6): 137- 1149, 2016
[44] Hamid Rezatofighi, Nathan Tsoi, Jun Gwak, Amir Sadeghian. Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 658-666, 2019.
[45] Yusuke Sekikawa, Kosuke Hara, and Hideo Saito. Eventnet: Asynchronous recursive event processing. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3887-3896. 2019.
[46] Guangting Wang, Chong Luo, Xiaoyan Sun, Zhiwei Xiong, and Wenjun Zeng. Tracking by instance detection: A meta-learning approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6288 6297, 2020.
12

---

[47] Xiao Wang. Jianing Li, Lin Zhu, Zhipeng Zhang. Zhe Chen, Xin Li, Yaowei Wang, Yonghong Tian, and Feng Wu. Visevent: Reliable object tracking via collaboration of frame and event flows. arXiv preprint arXiv:2108. 05015, 2021.
[48] Yinda Xu, Zeyu Wang, Zuoxin Li, Ye Yuan, and Gang Yu. Siamfe+:: Towards robust and accurate visual tracking with target estimation guidelines. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 15549-1256,, 2020.
[49] Bin Yan, Houwen Peng. Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
10448 10457, 2021.
[50] Ju Hong Yoon, Chang- Baeng-Ryeolaae Lee. Ming-Hsuan Yang. and Kuk-Jin Yoon. Online multi-object tracking via structural constraint event aggregation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1392-140K 1400, 2016.
[51] Jiqing Zhang, Xin Yang. Yingkai Fu, Xiaopeng Wei, Baocai Yin, and Bo Dong. Object tracking by jointly exploiting frame and event domain. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13043-13052, 2021.
[52] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16259- 16268, 2021. [53] Chaoda Zheng, Xu Yan, Jiantao Gao, Weibing Zhao, Wei Zhang, Zhen Li, and Shuguang Cui. Box- aware feature enhancement for single object tracking on point clouds. In Proceedings of the IEEE/CVF International Conference
Computer Vision, pages 13199 1 13208, 2021.
[54] Changqing Zhou, Zhipeng Luo, Yueru Luo, Tianrui Liu, Liang Pan, Zhongang Cai, Haiyu Zhao, and Shijian Lu. Pttr: Relational 3d point cloud object tracking with transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8531-8540, 2022.
[55] Yi Zhou, Guillermo Gallego, Henri Rebecq, Laurent Kneip, Hongdong Li, and Davide Scaramuzza, Semi- dense 3d reconstruction with a stereo event camera. In Proceedings of European Conference on Computer Vision, pages 235-251, 2018.
[56] Alex Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Danilidiss Ev-flownet: Self-supervised optical flow estimation for event-based cameras. In Proceedings of Robotics: Science and Systems, 2018.
[57] Alex Zihao Zhu, Dinesh Thakur, Tolga Özaslan, Bernd Pfrommer, Vijay Kumar, and Kostas Danillidis. The multivehicle stereo event camera dataset: An event camera dataset for 3d perception. IEEE Robotics and Automation Letters, 33):22032-2039, 2018.
2018.
[58] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chancy, and Kostas Danillidis. Unsupervised event-based learning of optical flow, depth, and egomotion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 989-997, 2019.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You are strongly encouraged include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section 2.
Did you include the license the code and datasets? [No] The code and the data are proprictary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We have discussed the limitations in Sec. 4.4.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Existing visual object tracking algorithms may cause public privacy issues. However, our tracking algorithms is specially designed with event-based vision, which contains less visual patterns than traditional RGB images. Thus, we think would put lower threaten for public privacy.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A] No theoretical results.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] Our algorithm is robust to the random seed. We tested
our model with different random seeds and showed the results in Table
Table 5: Quantitative comparison on the FE108 dataset, with different random seeds.
Seeds
RSR
OPo.s
OP
OPo.75 0.75
RPR
50
54.9
65.8
21.4
85.9
100
54.9
65.9
21.6
85.7
200
54.9
65.8
21.4
85.8
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please refer to Sec. 4.1.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) your work uses existing assets, did you cite the creators? [Yes] We have correctly cited the FE108 dataset in Sec. 4.1.
(b) Did you mention the license of the assets? [Yes] Please refer Sec. 4.1.
14

---

(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We have included the source code and pre-trained model in the supplementary material. And we will also release the simulated Evt-LaSOT.
(d) Did you discuss whether and how consent obtained from people whose data you're ro using/curating? [Yes] Since there no random process for those image- based methods during inference. Their result is constant.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] There is no person related data in our dataset.
5. If you used crowdsourcing or conducted research with human subjects
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] No research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "YNkQdJ4Y9Is", "reviewer": "Reviewer_kchd", "summary": "Unlike existing methods that usually reorganize raw event data to solve the unusual data structure problem, this work develops new designs tailored to the unique data structure to realize object tracking. To this end, the authors construct a new end-to-end learning-based paradigm that directly consumes event clouds. The proposed method featured key-event embedding and motion-aware target likelihood prediction to take advantage of the unique characteristics of event data. Extensive experiments on both synthetic and real event datasets demonstrate the excellent performance of the proposed method in terms of both tracking accuracy and speed.", "questions": "1.\tWhat is the correspondence between subfigures in the upper and lower lines of Figure 3? Can you further explain how to understand the green arrow in the second row? (Number of arrows, consistency of direction, the thickness of arrows, etc.)\n2.\tIn Section 4.1, what is the sequence length of the datasets used in this experiment (mean frame, min frame, max frame)? Does the sequence length have a greater impact on the experimental results?\n3.\tAs far as I know, LaSOT contains 1400 sequences, and the sequence length varies from 1K to 10K frames, so how does the Evt-LaSOT construct in this article filter data from the original LaSOT? Are there specific standards? Is the filtered sequence have the same length as the original sequence?\n4.\tTable 2 presents the proposed algorithm has a higher FPS on Evt-LaSOT. The authors point out that the main reason is this method directly consumes raw event data without pre-processing. Is there any other reason besides this? Can you design some ablation experiments to analyze the processing speed?\n5.\tThe video demo's main screen intuitively shows this method's effect, but what do the three sub-screens on the right refer to? Please explain the specific meaning of each sub-screen.\n", "limitations": "The authors discuss the public privacy issues in the additional material and argue that the event-based vision puts a lower threat to public privacy than RGB image. However, I want to know how secure and robust the event-based vision is? If we deploy the event-based scheme on self-driving systems, will it be safer than the current RGB image-based plan?", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper concentrates on the unusual data structure problem and presents the first end-to-end learning-based object tracking paradigm on raw event data. Experiments illustrate that the proposed framework achieves competitive performance and efficiency compared to SOTA methods. The article is well-structured and easy to read. However, due to space limitations, some experimental details are not fully elaborated, and the analysis of experimental results can also be further strengthened. Please refer to Questions for details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XZnEKfIhyIP", "reviewer": "Reviewer_Vqd3", "summary": "Instead of converting events into 2D event frames/images, this work first sample a subset of key events and use GNN to transform them into high dimensional feature embeddings. The resulting feature embeddings are used to compute target likelihood following siamese fashion. In addition, motion-aware target likelihood is also computed to strengthen the matching process. Experiments on both synthetic and real event datasets demonstrate the its effectiveness.", "questions": "1. The description about key-event sampling is not very clear. For grids with empty events, do they also try to find spatially closest events as key-events?\nIn addition, what if there are two events that both are closest events to the grid? \nDoes this operation heavily compress temporal information within event streams?\n2. Too many details are moved to supplementary material, which cause difficulty in understanding the whole method. For example, GNN-based spatio-temporal embedding, target likelihood prediction via motion trajectory back-tracking, and confidence-based object proposals.\n3. In the experiment on synthetic data, would low fps of videos make the simulation less realistic and bring more gap between simulation and real-world data?\n4. What about comparison with other event representations such as voxel grid and time surface?\n5. It is not clear how to effectively supervise the motion-aware branch to get the correct motion? Is there any intermediate supervision or regularization?\n6. In Fig. 3, what is the relationship between (a) (b) (c) (d)? In addition, it seems the motion vectors are noisy. Do the motion vectors can really reflect the movement triggering the corresponding key events?\n7. The performance on Ev-LoSOT (PRP 30+) is very low, while the top RGB trackers (e.g. OSTrack)  have achieve performances higher than 70+ with realtime speed. It implies the experiment on Ev-LaSOT tend to fail. The SiamBAN is mentioned but is not compared in the experiments.\n8. The result of FENet in Table 1 seems to be worse than the one reported in original paper.", "limitations": "Yes, authors have discussed the limitation of this work. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. A novel event processing method which first samples large amount of events into a subset of key events and then uses GNN to transform them into feature embeddings. the embedding representations could preserve more task-specific information than others.\n2. Different from previous event-based trackers that, the proposed method explore to utilize the motion information contained within the event could, which is very interesting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I4ycaaUPqfN", "reviewer": "Reviewer_bkyP", "summary": "This paper make the first attempt to construct a new end-to-end learning-based paradigm that directly consumes event clouds. Considered the high temporal resolution and sparse characteristics in event clouds, it proposed key-event and graph-based network to extract irregular spatio-temporal information from raw data. Then this paper proposes a dual-path architecture to predict the possibility of a key-event of the search event cloud. The dual-path architecture captures the similarity of each search key-event to the template key-events and utilize the information of motion trajectories respectively. This paper demonstrates the superiority of the proposed framework over state-of-the-art methods in terms of both the tracking accuracy and speed.", "questions": "(1) Motivation mentioned the high temporal resolution and sparse characteristics of event data. Where is the specific design for these two characteristics the data processing and model design.\n(2) The ablation experiment did not discuss how much does the use of raw data contributed to the final results. Does removing the uniquely designed data processing part of your method make a big difference to the final result ?\n(3) When comparing with SOTA, why do you choose some RGB-based methods to train yourself instead of comparing with other event-based methods?\n(4) “Considering that trajectories are usually continuous, the former event points should have relatively smaller moving offsets” . The offset is derived from the projection of the key-event in the last step, and the key-events in the last step is not the same timestamp either.  And the time difference between the same key-event before and after is not sure. Is it unreasonable to have a relationship between the displacement and the timestamp?\n", "limitations": "The motivation is very convincing, but targeted design and description with respect to motivation are needed. The comparison experiment should also be focused on the points emphasized in the motivation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(1) The paper has a clear structure and is easy to understand the motivation and innovation.\n(2) It proposed key-event and graph-based network to extract irregular spatio-temporal information from raw data, Making better use of the characteristics of event data.\n(3) It developing a new end-to-end learning-based paradigm that directly consumes raw event data , in which new modules considering the unique features of event clouds.\n\nWeaknesses：\n(1) The specific technical details are not clear enough\n(2) This paper mentioned the high temporal resolution and sparse characteristics of event data. The biggest motivation of this paper is to design data feature extraction module and algorithm model according to the unique attributes of data. But I didn't see a targeted design of the unique attributes of the Event data.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "3-P75O81wmb", "reviewer": "Reviewer_mj7a", "summary": "The paper tackles the problem of event data-based object tracking. Instead of using event frame/image representation as in the existing methods, the authors chose to process raw event clouds for event data-based object tracking. The proposed framework includes a downsampling strategy to extract key events from a raw event cloud, a graph-based network to embed spatio-temporal information of key-events, semantic and Motion-aware Target Likelihood Prediction, and a confidence-based object proposal. The authors conduct experiments on FE108 [49] and a new synthetic dataset Event LaSOT for tracking the evaluation of non-rigid objects. All the baselines and the proposed method are trained on FE108 and evaluated on the testing set of FE108 and Evt-LaSOT. The experiments prove the possibility of consuming raw event clouds for object tracking and achieving state-of-the-art performance on challenging datasets.", "questions": "The current recommendation is Borderline accept (5). Overall, the authors indeed prove the claim that they design an architecture that consumes raw event cloud and achieves promising object tracking performance on two challenging datasets compared with STOA. The paper could be improved by discussing the difference between the proposed methods with existing 3D point cloud tracking literature. Moreover, it would be interesting to demonstrate that existing 3D point cloud trackers could not outperform the proposed method as it considers the characteristics of the event cloud.", "limitations": "Yes, the authors discuss the limitations of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors focus on event-based object tracking, and we make the first attempt to construct a new end-to-end learning-based paradigm that directly consumes event clouds. The reviewer found the direction will interest those working on event-based cameras. Moreover, those who work on point cloud modeling might find a new problem to test the applicability and generalization of their methods.\n2. The work contributes a network architecture that can consume a raw event cloud and demonstrate that the network can outperform other event cloud representations for object tracking on two challenging datasets.\n3. Convincing tracking results are demonstrated in the supplementary material.\n4. The implementation of this framework has been released.\n\nWeaknesses:\n1. 3D point cloud tracking: The proposed framework shares similarities with existing 3D point cloud tracking literature, e.g., Qi et al., 2020 and Zhou et al., 2022. It will be essential to summarize the difference between the literature pools. Moreover, the authors could analyze how the proposed architecture harvests characteristics of event cloud (e.g., inherent motion trajectories) where methods built for 3D point cloud tracking do not consider these aspects. Moreover, the experimental section could be further improved by comparing the proposed framework with a state-of-the-art 3D point cloud tracker, which aims to show the importance of considering data characteristics.\n- Qi et al., P2B: Point-to-Box Network for 3D Object Tracking in Point Clouds, CVPR 2020\n- Zhou et al., PTTR: Relational 3D Point Cloud Object Tracking with Transformer, CVPR 2022\n\n2. Ablative studies: The reviewer found the contribution of the two target likelihood predictors unclear. Specifically, based on the implementation of the proposed architecture, the authors could have removed one of the predictions and tested the corresponding model. However, in Table 3, the performance of removing semantic-aware target likelihood prediction is missing.\n\n3. STOA comparison: In Table 1, the proposed method demonstrates inferior performance compared to E-PrDiMP on the HDR setting. The discussion in Sec. 4.2 does not explicitly discuss it. It would be essential for the authors to provide the corresponding insights as they mention that event-based generally surpass RGB-based methods under LL and HDR scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept