Revisiting motion information for RGB-Event tracking with MOT philosophy
Authors: Tianlu Zhang
Kurt Debattista
Qiang Zhang
Guiguang Ding
Jungong Han
Venue: NeurIPS 2024 poster (2024)
Keywords: RGB-Event tracking; MOT philosophy; Candidate matching
Primary area: deep_learning_architectures
TL;DR: We propose a novel RGB-E tracking framework, i.e., CSAM, which first predicts the candidates by using an appearance model and then keeps track of both targets and distractors with an MOT philosophy.

Abstract: RGB-Event single object tracking (SOT) aims to leverage the merits of RGB and event data to achieve higher performance. However, existing frameworks focus on exploring complementary appearance information within multi-modal data, and struggle to address the association problem of targets and distractors in the temporal domain using motion information from the event stream. In this paper, we introduce the Multi-Object Tracking (MOT) philosophy into RGB-E SOT to keep track of targets as well as distractors by using both RGB and event data, thereby improving the robustness of the tracker. Specifically, an appearance model is employed to predict the initial candidates. Subsequently, the initially predicted tracking results, in combination with the RGB-E features, are encoded into appearance and motion embeddings, respectively. Furthermore, a Spatial-Temporal Transformer Encoder is proposed to model the spatial-temporal relationships and learn discriminative features for each candidate through guidance of the appearance-motion embeddings. Simultaneously, a Dual-Branch Transformer Decoder is designed to adopt such motion and appearance information for candidate matching, thus distinguishing between targets and distractors. The proposed method is evaluated on multiple benchmark datasets and achieves state-of-the-art performance on all the datasets tested.

Revisiting Motion Information for RGB-Event Tracking with MOT Philosophy
Tianlu Zhang EMIM
Kurt Debattista
Qiang Zhang* EMIM
Warwick Manufacturing Group University of Warwick
Xidian University tl tlzhang96@outloook.o com
Xidian University
K. Debattisttasarvickk ac uk
qzhang@@xidian edu.
Guiguang Ding
Jungong Han
School of Software Tsinghua University dingggêtsinghua edu. cn
Department of Automation Tsinghua University
j jungonghan770maiil. com
Abstract
RGB-Event single object tracking (SOT) aims to leverage the merits RGB and event data to achieve higher performance. However, cxisting frameworks focus exploring complementary appearance information within multi-modal data, and struggle to address the association problem of targets and distractors in the temporal domain using motion information from the event stream. In this paper, we introduce the Multi-Objec Tracking (MOT) philosophy into RGB-E SOT keep track of targets as well as distractors by using both RGB and event data, thereby improving the robustness of the tracker. Specifically, an appearance model is employed predict the initial candidates. Subsequently. the initially predicted tracking results, combination with the RGB-E features, are encoded into appearance and motion embeddings, respectively. Furthermore, a Teriirall Transformer Encoder is proposed model the spatial-temporal relationships and learn discriminative features for each candidate through guidance of the appearance motion embeddings. Simultaneously.
Transformer Decoder designed adopt such motion and appearance information for candidate matching, thus distinguishing between targets and distractors. The proposed method is evaluated on multiple benchmark datasets and achieves tfateof--the-ar performance on all the datasets tested.
1
Introduction
Single object tracking (SOT) aims to predict the position of a target in videos, by being given only the
position of the target in the initial frame. While traditional RGB-based trackers [ 8l can effectively capture comprehensive scene representations, including color and semantic information, they face significant performance degradation challenging conditions like fast motion, low illumination and distractions from similar objects.
To address such challenges associated with frame-based cameras, advantage of event cameras (39], which are characterized by high temporal resolution and high dynamic range, augment the RGB data for reliable object tracking. In the past few years, various methods have been proposed for RGB-Event (RGB-B) object tracking. Existing RGB-E trackers primarily concentrate on exploring complementary appearance information within RGB and event
researchers have taken
"Coresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

RGB Tamplite & Scarch
Appearance Model
RGBl Templane Saardh
Predictor
Matching
0.39
Eveest Templase & Search
Resilt
Histrrcal Videos
Scene
Information
Propagation
(8)
Historicl Vidzos
Historical Result
Candidate Matching
Tracker
CEM
STTE
Tomplee & Sizrch
Matching
Candidates
Appearance Model
CEM
Tarae
(c)
Figure Architectures of different RGB-E tracking frameworks. (a) RGB-E tracker based on appearance information. (b) RGB tracker based information propagation. (c) Our proposed CSAM framework.
data to enhance tracking performance with three typical approaches i.c., early fusion method [29], middle fusion method [37 38) and one-stream method [24] (4]. Despite achieving commendable
improvements, mainstream RGB-B E tracking algorithms still cannot solve the association problem of the targets and distractor objects in the temporal domain, as shown in Fig. (a).
Alternatively, some RGB trackers 3 [9] propagate valuable scene information through the sequence improve their discriminative ability, as shown in Fig. (b). These methods mine the information in two main ways. The first one relies implicitly transforming the scene information to locate the targets, and the scene information is generally represented using a set of embeddings [2 3 To ensure effective transformation and avoid introducing noisy information, these methods usually require careful design the encoding strategy to obtain effective scene information embeddings. The second approach explicitly explores the scene information by simultancously keeping track of both targets and distractors 19 41. However, these methods are susceptible to environmental interference, and their matching strategies relying appearance information may miss the target when the target and distractor trajectories are close.
In fact, event data can not only provide the edge information to improve the RGB feature represen- tations but also contains abundant motion cues to reflect the motion state of the objects, which meaningful to differentiate between targets and distractors, even if they may look similar. Motivated by these observations, we propose an Appearance-
Modeling RGB-E tracking framework with Cascade Structure, referred to CSAM, that goes beyond leveraging complementary appearance information within RGB-E data. As shown in Fig. (c), the proposed CSAM framework employs an appearance model to initially determine the candidates with similar appearance the targets, and then designs a candidate matching network with encoder-decoder structure to dynamically incorporate motion information contained in the RGB-E E videos to track all the candidates with the Multi-OU Multi-Object Tracking (MOT) philosophy. The candidate that matches the historical target tracklet will be regarded the final tracking result.
Specifically, candidate matching network consists of a Candidate Encoding Module (CEM), Opatial-Temporre Transformer Encoder (STTE), and a Dual-branch Transformer Decoder (DBTD). Recognizing the critical roles of the two types information in candidate association, our proposed CEM is used encode both appearance cues and motion representations for each candidate. Sub- sequently, an STTE block, comprising a Spatial Encoder and a Temporal Encoder, is introduced to model spatial-temporty relationships among candidates by synergistically utilizing such appearance and motion embeddings. Finally, a DBTD block, comprising a Motion Decoder, is presented to match candidates with historical tracklets by using both appearance and motion information.
Decoder and a
Contributions: In summary, our contributions are: (i) We propose novel RGB-E tracking frame- work, i.e., CSAM, which first predicts the candidates by using an appearance model and then keeps track of both targets and distractors with MOT philosophy. To the best of our knowledge,

---

Input
Appearance Model
Candidate Encoding
Infereace
(Swc
App Model
AFE
Spatial Encorder
Arrmgonnnt
RC-
Temporal Encodde
CLES
Matchin
App Modell 100n-310 2111
Encoder
Spatial
Ri- Arroneement
Temporal Enicoder
MFF
Earpet
Agpo Model (Sc-311
Encoderr
Temporal Encorder
MFF
M
Resporse
Appearane
map
region tokens
resolts
region tokens
ambolkingg
ambudklings
Appesrsnoe
Motion
Spurial-Temporal
Dual-branch
AFF
Feuture Encoding
MFF
Feature Encoding
Transfomer Encoder
Transfomer Decoder
Figure 2: Overview of the proposed RGB-E tracking pipeline.
the first to introduce the MOT philosophy for the SOT task using RGB-E data. (ii) We propose three effective modules: a Candidate Encoding Module, a Spatial-Temporal Transformer Encoder and Dual-branch Transformer Decoder. The appearance information well as the motion cues within the RGB-E data can be fully exploited by the proposed modules for accurate candidate association. (iii) We show significantly improved statc-of-the ofce----the-rr art results of our proposed method on multiple RGB-E tracking benchmarks.
Related Work
Visual object tracking: The current prevalent tracking pipelines can be categorized into three groups: CNN-based trackers, CNN-Transformer trackers and Transformer-basee trackers. CNN- based trackers utilize Siamese network [5] or Discriminative Correlation Filter (DCF) B 1 to
address tracking tasks by matching templates and search regions. However, the inherent properties of CNNs limit their ability for global information exploration and interaction, thereby constraining advancement CNN-based trackers. In response, some ENN-Transformer trackers 33 28
employ attention mechanisms to establish global dependencies between template features and search features. But these hybrid CNN-Transformer trackers still independently extract features from templates and search regions using CNN networks, resulting in extracted features being unaware of the tracking target. To address this issue, several pure Transformer based trackers [35] 32 40]
overcome the challenge by unifying feature extraction and feature relation modeling through single Transformer backbone, leading to of-te-of-the-ar tracking performance.
RGB-E object tracking: In recent years, there has been growing interest among researchers in merging RGB frames and event streams for object tracking. Some researchers focus on exploring complementary information within RGB-E data via specially designed cross-modal interaction For instance, Zhang et al. (29 proposed a cross-domain feature integrator to dynamically fuse visual cues from both the frame and event domains. Alternatively, CEU'track [24] proposed one-stream framework based on the Transformer, which simultancously addresses feature extraction, template-see search relation modeling, and cross-modal interaction. Recently, some methods aim to adapt the RGB tracking model to RGB-E tracking in the prompt learning manner. However, existing RGB-E E tracking frameworks cannot fully explore the abundant motion cues within the event stream, consequently limiting tracking performance in the presence of distractors.
strategies 29 24 44
13 42 31
Multi object tracking: Multi-object tracking (MOT) aims to track multiple objects in video paradigm 77,) where an object detector initially employed to locate all proposals, followed by an association network to match all of these objects, gaining popularity for the MOT task. Additionally, some researchers have explored the joint- detection-an and and-tracking pipeline (23) aiming to achieve detection and tracking simultancously in
sequence. Currently, the bocking-bb-detectti

---

single stage. There are some approaches that aim to enhance the tracking performance of Single Object Tracking (SOT) via the use of an MOT philosophy. For instance, DMTrack [4T] designed a lightweight detector and an explicit object association module to track both targets and distractors. KeepTTrack [19] proposed learnable candidate matching network and designed several mechanisms, including partial supervision, self-supervised learning and sample-mining to address the problem of incomplete annotation in SOT training data. However, these methods only use the RGB modality and overlook the importance of Opatial-temporee relationships among candidates for matching candidates and tracklets.
3 Method
As shown in Fig. our framework first employs appearance model to generate the potential proposals. Subsequently, several modules are proposed to identify targets and maintain tracking of all candidates to prevent tracking drift. Specifically, the appearance model predicts the target scores and bounding boxes for M candidates of the th current frame and N candidates for each previous frame (see Sec Secondly, set of features frames, including target classification scores, appearance features, event embeddings and candidate locations. These features then aggregated into appearance embeddings and motion embeddings for each proposal (see Sec(3.2). Thirdly, the STTE is employed to jointly model spatial-tempon relationships for each tracklet (see Sec 3.3). Fourthly, utilizing the DBTD, an N X M assignment matrix is calculated for matching tracklets from previous frames with candidates from the current frame (see Sec 3.4). It should be noted that not every tracklet has all the candidates in the previous T frames due to occ ssclusion, missing detection, etc. For illustrative purposes, we consider the situation that there are no missing candidates for each tracklet. Our method solves the cases with missing tracklets similarly to the typical multi-object tracking method [6] (see supplementary material Sec.B). In the following, we will describe the proposed tracking framework in detail.
extracted for each candidate from T previous
3.1 Appearance Model
Here, we employ CEUTrack 24 as our appearance model. Specifically, the event streams are initially transformed into voxel representations through a voxelization operation [24]. Subsequently, given the initial locations and tracking results, we crop the template patch, the template voxel, the search patch and the search voxel, respectively. After that, the projection layers are adopted to transform the four inputs into token representations, which are then fed into the vanilla ViT [0] for joint feature extraction, cross-modal interaction and search-template matching, Finally, the tracking head, employing the same structure as that OStrack [355, takes the concatenated RGB and event search region features from the backbone input to predict the appearance tracking results. Please refer to the supplementary material (Sec.A.1) for additional details about the appearance model.
3.2 Candidate Embedding Module
The goal of CEM to first select initial candidates similar to the target and filter out the most simple negative backgrounds, and then obtain the appearance embeddings well as the motion embeddings. With the classification scores and regression offsets outputted by the appearance model, generate N candidates similar to the target for each previous frame by using
Suppression (NMS) 20 The appearance features of each candidate can be obtained by performing the PRolAligh 22 on the RGB backbone features based on its corresponding location. We can
represent the appearance features of each candidate in the (t - T) th frame as Ft-T f T RdJA Jo and represent their corresponding classification scores as C {ct E RIJN JR- Both the appearance features and the classification scores convey essential appearance cues for each candidate. To integrate these two types of information, we propose an Appearance Feature Encoding (AFE) layer. Specifically, AFE processes the backbone features fI-T via a single convolution layer to obtain more discriminative features fey T-T E Rdd and employs several MLP layers on N to generate the classification embeddings ce T-T
T- E Rd. These features are then combined as: aN few T-T + cely T- The appearance feature embeddings for the (t - T) - th frame are thus represented by A'-T {ayy 1n=1.
T- T

---

Spatial Encoder
Re-arrangement
Temporal Encoder
Head- Heud-Amentti Amtention
Add & Nomn
Re-
Add & Nerm
Multi Head-Ant Amention
Add Nonn
Add & Norm
PFN
FTV
Norm
al
ASE
Mf-11
m2
mi
ASE
ASE
N
T
Figure 3: Architectures of the proposed Spatial-Temporal Transformer Encoder.
Additionally, we obtain the event features of each candidate in the (t - T) th frame as Ft-T (eN E RdjN 1-T 1-T Jn=1 and represent their locations as pt- {ph T--TT E _ where PN 1-T {"N .VN "N T-T 'N !) denotes the normalized bounding box coordinates. Both the event stream and the location set contain rich motion information about those candidates. To fuse these two types of features, Motion Feature Encoding (MFE) layer, which has similar structure of AFE, is first used to obtain the enhanced event representations ee RR and location embeddings pek t-T Rd. Then these features are fused as: m e + pe The motion feature embeddings of each candidate in the (t T) th frame can be thus represented by M {m TIN Jn=1-
3.3
Transformer Encoder
The proposed STTE aims at learning more discriminative feature representations for each tracklet and establishing effective relationships among objects both spatial and temporal domains. The inputs for STTE include T sets of appearance embeddings {A*-T. ..... A }} and T sets of motion representations { {M*
M }. All of these embeddings will be first processed via a Spatial
Encoder construct the spatial relationships among candidates in each frame. Subsequently, these spatially encoded features are re-arranged to construct tracklets across T frames, and their temporal relationships established via the proposed Temporal Encoder.
Spatial Transformer Encoder: The proposed Spatial Encoder independently processes each frame to construct spatial correlations, and we take the (t (t-11 1) th frame as an example for illustration. It is difficult to establish dis- tinctive spatial relationships by only using the appearance information. Differently, the motion information is more suitable to establish meaningful spatial affinities. Inspired by the graph attention networks (26), each candidate's s ap- pearance embedding is regarded as a node, and the edge weight between each nodes is defined by the motion infor- mation.
eWai
rme E-1
W3
eW1
Twg
Vi
M1-1
mi
42
+1-1
Figure 4: Architectures of proposed Graph Multi-head Attention Block.
Specially, at the (t-1) 1) th frame, consider each candidate's appearance and motion representations in 1 and M as two node sets /t-1 and Vt-1 respectively. As shown in Fig. 3 (a),
complete bipartite graph G (VL-1 Et-1) to model the object-level relations between these candidates. Here, E = {(u, v)|Vu, v Vn Then, the edge weight between node i and node in Vt-1 will be denoted calculated through the inner product operation. More specifically, as that in a typical Transformer block, some normalization layers and línear transformations first applied these motion nodes, followed by an inner product calculation, to achieve Cij- Formally,
Cij, which can be
"
where Wi and Wg are the linear transformations and ey will be further normalized by the softmax fuction, obtaining
exp(ery)
ewiy Etent- exp(e()
(2)
5

---

With the edge weights passed from all nodes in V to the i-th node in Va-1, the aggregated representation of the i-th node in Vt-1 can be transformed by:
me --1
Σ eway W.m.
(3)
JEVI
where W. is a matrix for linear transformation. It should be noted that we adopt the multi-head attention structure to improve the discriminability of graph attention learning.
Finally, we fuse the aggregated features me; me with the appearance features d-1 to obtain a more powerful feature representation:
FFN (mee -aa 1)),
(4)
where cat(-) represents vector concatenation, (-) denotes the feedforward neural network. The final spatial encoded features ASE {s} 1-1 ..... NN t-1 are obtained by further employing the residual connections and two FFN layers as that in a typical Transformer block. These spatial encoded features are re-arranged to N tracklet sets {I' 1...... ..... I'N) J of all candidates through the T frames, where IN { T- s/x ). Meanwhile, the motion embeddings also re-arranged N tracklet sets {o....., ON], where ON {mk m. '.
These N tracklet sets are further encoded by a Temporal Trans- former Encoder. Here, take N- th tracklet as example for illustration. As shown in Fig. first fuse the spatial encoded features I' with the motion feature set ON by the element-wise addition operation, and then employ the multi-head attention calculate the attention weights AN E T'x thus generating the attention attention-weiihheed features. After that, these weighted features are processed through two FFN layers and residual connections to obtain the final output I EN, which consists of the N th tracklet's feature representations {thT (--1) The outputs of the Temporal Transformer Encoder , } are re-arranged =en R NTxd
Temporal Transformer Encoder:
3.4 Dual-branch Transformer Decoder
Motion Decoder
The proposed DBTD generates the assignment matrix A by using the output of the Transformer encoder g and the features of M candidates in the current frame. Initially, we generate the spatial-encoded feature set (s......s } and re-arrange it to 3ea" E RMxdd as that in the Sec 3.3 Then, Ecan is duplicated N times, re- sulting in =ean en
moo A mppp
Va
A-dd
ssi
m5
See
& Doplicati
Eean E NMxd en
Xd1
moog 30
Sorth
After that, the proposed Spatail- temporal Decoder follows the stan- dard Transformer framework, which takes the spatial-encoded features of the current frame Êcan as the query, and uses the encoded features of the previous frame =eu as the key and value. The Multi-Head Attention mechanism
gea
Opatial-Tempora Decoder
Figure 5: Architectures of the proposed Dual-branch Trans- former Decoder.
24 is calculated for Scan and Eeh to generate attention weights. The output passes through two FFN layers and residual connections, generating the output tensor R. x The output of the appearance decoder can be processed through an FFN and a softmax layer to generate the assignment matrix A N
Moreover, to better match each tracklet with the candidates in the current frame, the motion infor- mation of the (t - 1) th frame is utilized as the motion information for the tracklet. The Motion Decoder employs the same structure as that in the appearance decoder to obtain the assignment matrix A E N The final assignment matrix A can be obtained by A = A + m

---

3.5 Object Association
During inference, we first employ the appearance model to generate candidates of each frame. If only one target with high score is present in both the previous and current frames, this candidate is selected as the target, and the candidate matching model omitted reduce computational costs and accelerate inference. In contrast, when multiple candidates exist during tracking, the assignment matrix A calculated using the proposed candidate matching model. After that, threshold 7th is adopted on A to remove the ambiguous correspondence, Finally, we match the predicted boxes and the candidate boxes using the Hungarian algorithm [1]. The candidates that do not match any tracklet will be assigned new ID, and the tracklets that do not match any of the detections in the past consecutive T frames will be terminated. T experimentally set to 15. Please refer to the supplementary material (Sec.B.3) for more details.
4 Experiment
Table I: Comparison with state-of-the-art trackers on COESOT 24
FE108 38
and VisEvent [2
The numbers with red and blue colors indicate the best and the second best results, respectively.
FE108
VisEvent NPR
COESOT
Method
Source
Backhone
Type
RSR
RPR
PR
NPR
STNet 36
CVPR22
Event
58.5
89.6
49.2
35.2
MonTrack
NeurlPS22
Event
63.3
90.7
[1
TIP23
Res18
56.7
89.2
54.5
39.8
HDETrack 30
CVPR24
VIT-B
Event
59.8
92.2
59.0
59.0
52.3
DiMP
ICCV19
Res50
RGB-E
57.1
85.1
67.0
58.1
47.8
67.1
65.9
58.9
PrDiMP 9
CVPR20
Res50
RGB-E
55.2
86.8
65.3
57.7
47.6
65.0
64.0
57.9
SiamRCNN
CVPR20
Res Res 101
RGB-E
68.0
62.6
52.7
67.5
66.3
60.9
TrDiMP
CVPR21
Res50
RGB-E
60.3
91.2
66.9
65.8
60.1
TransT
CVPR21
RGB-E
63.9
93.0
67.9
66.6
ToMP 18
CVPR22
Res101
RGB-E
61.8
91.1
67.2
66.0
59.9
FENet(38
ICCV21
Res
RGB-E
63.1
91.8
CEUTrack [4]
ArXiv22
VIT-B
RGB-E
55.6
84.5
71.8
66.4
53.5
70.5
69.0
62.0
HRCEUTTrac-B 45
ICCV23
VIT-B
RGB-E
71.9
70.2
63.2
HRCEUTTrak-LL
ICCV23
VIT-L
RGB-E
73.8
71.9
65.0
HRMonTTrckkTTi 45
ICCV23
VIT-B
RGB-E
66.3
95.3
HRMonTrack-B
ICCV23
VIT-L
RGB-E
68.5
96.2
AFNet 37
CVPR23
Res 18
RGB-E
67.8
59.2
ViPTi
CVPR23
VIT-B
RGB-E
76.6
73.0
60.8
73.9
72.2
65.7
SDSTrack1
CVPR24
VIT-B
RGB-E
79.3
75.5
OncTrack
CVPR24
VIT-B
RGB-E
78.1
75.6
SeqTrrackv-BB2211 4
ArXiv24
VIT-B
RGB-E
79.9
76.5
63.7
SeqTrackv2-L25611 4
ArXiv24
VIT-L
RGB-E
80.6
77.8
65.2
CSAM-T+
2024
VIT-T
RGB-E
66.7
95.5
76.1
72.4
61.5
73.3
70.5
63.6
CSAM-BB
2024
VIT-B
RGB-E
70.5
97.1
81.6
78.6
65.9
76.7
74.8
68.1
4.1 Implementation details
Our proposed CSAM is implemented in Python 3.8 using PyTorch 1.7.1. The CSAM training is conducted two Nvidia RTX 3090 GPUs. For inference, test our tracker a single Nvidia RTX 3090 GPU. The search region 4a times the target object area and resized to a resolution of 256x256 pixels, whilst the template is 22 times the target object area and resized to 128 128 pixels.
Architectures: We instantiate two models of CSAM: CSAM-T and CSAM-B, by varying the backbone network in the appearance model, i.c., ViT-Tiny and ViT-Base. We initialize ViT-Tiny using the weights from DeiT--inyl (25), and the backbone weights ViT-B are initialized with corresponding MAE encoders [2]. In the candidate matching network, both the proposed STTE and DBTD apply one individual layer. Please refer to the supplementary material (Sec.A.1 and Sec.B.1) for more details.
Training:
The training our CSAM comprises three parts. In the first part, the following three loss functions are adopted: the focal loss for classification, and LI loss and GIOU loss for bounding box regression [35]. We employ the same training setting as that in OSTrack 35 train an RGB tracker. Secondly, employ the same training setting that in HRCEUTrack to train the
7

---

appearance model. In the third part, the parameters of the appearance model are fixed and other parameters in our proposed framework are set be trainable. Since only the targets' locations are provided in the existing RGB-E tracking dataset, the partial supervision loss and self-supervised loss in KeepTran [19] are employed to supervise the assignment matrix generated by our proposed model. Please refer to the supplementary material (Sec.A.2 and Sec.B.2) for additional details about the implementation details.
4.2 Evaluation datasets and metrics
We evaluate the performance of our proposed CSAM on three large-scale RGB-E single object tracking datasets: VisEvent [29] FE108 [38] and COESOT (24). These three datasets were captured using DAVIS346, with a spatial resolution of 346 x 260, a dynamic range of 120 dB and the minimum latency of 20 jss. The COESOT [24] dataset comprises 578K RGB-E pairs, divided into 827 and 527 sequences for training and testing, respectively. These sequences were collected from both indoor and outdoor scenarios, covering range of 90 classes and 17 challenging attributes. The FE108 [38] dataset contains 108 RGB-E sequences, which capture 21 different types of objects. is divided into 76 and 32 sequences for training and testing, respectively. VisEvent [29] dataset collects 820 RGB-E video pairs, divided into 500 and 320 sequences for training and testing, respectively. Following (37, after removing sequences that miss event data or have misaligned timestamps, the VisEvent dataset includes 377 sequences for training and 172 for testing.
Dataset:
Metries: In FE108 [38). we use representative success rate (RSR) and representative precision rate (RPR) to evaluate all trackers. In COESOT [24) and VisEvent [29], we use success rate (SR), precision rate (PR) and normalized precision rate (NPR) for evaluation.
4.3 Comparisons with State-of-the-art Methods
To show the effectiveness of the proposed method, we evaluate and compare our CSAM with several state-of-the-art trackers, including 4 Event trackers and 17 RGB-E trackers. As shown in Table. * denotes that the RGB trackers are extended to RGB-E trackers via the early fusion approach. t and denotes that the model is pre-trained on RGB tracking datasets.
Results on FE108: As shown in Table. our proposed CSAM-B outperforms other top-performing trackers, such as HRMonTrrcc-B [44], HDETrack [30] and TransT (5), with clear margin, and achieves the best performance with an RSR score of 70.5%. and an RPR score of 97.1%. Even when compared to HRMonTrackk which has already obtained impressive tracking performance, our approach demonstrates notable improvements, with a 2.0% increase in RSR and a 0.9% increase in RPR. These comparisons fully demonstrate the effectiveness of tracking multiple candidates for robust tracking.
From Table. I we find that our method sets new state-of-the-art score on VisEvent. First, our proposed framework outperforms the Event trackers, e.g., STNet (36] and DANet M. by clear margin. Secondly, compared with appearance trackers SDSTrack 31 and ViPT 42), our model further improves the PR score by 4. % and 1.6% in NPR scores, respectively. This enhancement is attributed our model's comprehensive utilization of both appearance and motion information, enabling effective tracking of targets and distractors. Thirdly, our tracker surpasses the previous best tracker SeqTrackv22LL [4. which demonstrates that our method has stronger capability in handling various challenges.
Results VisEvent:
Results on COESOT: As shown in Table. [ our proposed CSAM-B achieves a PR score of 76.7% and a SR score of 68.1%, surpassing recent statc-of-the-aar trackers. Compared with the most competitive RGB-E tracker ViPT [42] 2.. our CSAM-B achieves performance gains of 2.4% in NPR score. These results meet our expectation that the exploration of spatial-temporal relationships from the appearance cues as well as the motion cues can effectively match the candidates and tracklets for SOT task.
Speed Analysis: As shown in Table 2 despite increased computational costs and parameters for simultaneous target and distractor tracking, CSAM maintains real-time performance on the RTX 3090 GPU. It strikes good balance between resource consumption and efficacy compared to competitors. Compared with the appearance tracker, our CSAM introduces limited computation costs, while significantly improving the tracking performance. Consequently, CSAM-B achieves an
8

---

average rumning speed of 53 frames per second (FPS). We also notice CSAM's superior performance compared with the second best RGB-E tracker SeqTTrack2-LL25.
Table CSAM-B's efficiency analysis on VisEvent with a fixed candidate count of 4 for FLOPS calculation.
ViPT
Trackv222-8
SoyTrack22-L256
Appearance Tracker
CSAM-B
PR/SR
76.6/60.8
79.9/63.7
80.666..2
75.3/60.6
81.6/65.9
FPS
75
40
15
75
53
Model size (M)
93.3
89
309
92.5
106.9
FLOPS (G)
52.1
232
62.7
83.2
4.4
Ablation Study
To verify the effectiveness our designed framework, we perform ablation analysis to evaluate differ- ent components in method by using the COESOT test [24] Please refer to the supplementary material (Sec.C) for more ablation experiments.
Table 3: Experiment results of different variants for Candidate Encoding Module (CEM), Spatial- Temporal Transformer Encoder (STTE) and Dual-Branch Transformer Decoder (DBTD), Here, 'AppModel', 'SE', 'TE', 'STD' and 'MD' denote the appearance model, Spatial encoder, Temporal encoder, Spatial-temoral temoral decoder and Motion decoder, respectively.
AppModel AFE MFE
OTE
SuperGlue
TE
OTD
STD
MD
SR
PR
65.5
74.8
CEM
67.1
75.9
67.3
76.4
67.5
76.3
STTR
V
66.2
75.3
66.8
75.8
67.1
75.9
V
67.4
76.2
66.6
75.5
DBTD
66.7
75.3
67.7
76.3
66.4
75.5
CSAM
68.1
76.7
Effectiveness of the proposed CEM: To investigate the impact of our proposed CEM, several versions of our proposed method are provided, including @: Removing the AFE and MFE sub- modules in CEM block. @:: Removing the AFE sub-module in CEM block. @: Removing the MFE sub-module in CEM block. As can seen in Table 3 the tracking performance degrades after removing AFE or MFE sub-modules, which demonstrates the necessity of embedding appearance and motion information from the classification scores and bounding box coordinates.
Effectiveness of the proposed STTE: several variants are proposed, including @: Removing the spatial encoder (SE) and temporal encoder (TE) @ Replacing the spatial encoder by the original Transformer Encoder (OTE) block [10] 3:
To further verify the effectiveness of the proposed STTE,
Replacing the spatial encoder by SuperGlue [9]. @: Removing the temporal encoder in STTE block.
6: Removing the spatial encoder in STTE block. As can be seen in Table 3 the tracking performance experiences significant decline upon the removal of temporal encoder or spatial encoder, which confirms the necessity of spatial-temporal relationships in enhancing feature representations of the candidates. Furthermore, compared with several existing methods, c.g., OTE and SuperGlue, the proposed spatial encoder takes full advantage of the appearance information as well motion information constructing robust spatial correlations, thus achieving better results.
Effectiveness of the proposed DBTD: To further verify the effectiveness of the proposed DBTD, several variants are proposed, including ①: Replacing the apatial-tempore decoder (STD) by the original Transformer Decoder (OTD) block [0] @: Removing the motion decoder (MD) in DBTD block. @: Removing the spatial-temporal decoder in DBTD block. As can seen in Table 3 not using
decoder or motion decoder substantially deteriorates the performance. This demonstrates that the exploration of both appearance and motion information crucial for ensuring robust tracking.

---

Visualizations of the MOT philosophy Based on the proposed candidate matching network, CSAM- can track both the targets and distractors. As shown in Fig. candidates in each frame. An object disappears from the scene if none of the current candidates are associated with it. The MOT philosophy in CSAM-B can effectively suppress the negative influence of distractors for tracking.
*) denote the target and the
130
155
#1
980
#120
CSAMBB
ToTrckk
HRCEUTTakk
KeaTTrack
CELTTec
Figure 6: Visualizations of the MOT philosophy. (+. denote the target and the candidates in each frame. Red bounding boxes refer to the tracking results. Event images are used for visualizations only.
Conclusion
In this paper, novel RGB-E tracking framework with MOT philosophy has been proposed in order to keep track of both targets and distractors to robustly track a single object. Specifically, a Spatial- Temporal Transformer Encoder is proposed to establish rich temporal-spatial relationships by using appearance information in combination with motion information. Furthermore, by formulating the tracklets and candidates with the appearance features and motion embeddings, the affinities of the tracklets and candidates are explicitly modeled and leveraged. We conduct comprehensive experimental validation and analysis of our approach on three RGB-E object tracking benchmarks and produce new state-of-the-an results.
The current method is dedicated to constructing effective framework for RGB-E tracking with MOT philosophy, but it pays less effort to improve the efficiency of the supervision signals, which will consider in future work.
Limitation:
6 Acknowledgments
This work was supported in part by the State Key Laboratory of Reliability and Intelligence of Electrical Equipment under Grant EERI KF2022005, part by the Hebei University of Technology, in part by the National Natural Science Foundation of China under Grant 61803290 and Grant 61773301, in part by China Postdoctoral Science Foundation under Grant 2023M742745. and in part by the Natural Science Foundation of Shaanxi Province under Grant 2019IQ-312.
References
[1] Bhat, G., Danelljan, M.. Gool, L.V., Timofte, R.: Learning discriminative model prediction for tracking, In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6182-6191 (2019)
[2] Bhat, G., Danelljan, M., Van Gool, L., Timofte, R.: Know your surroundings: Exploiting scene information for object tracking. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16. pp. 205-221. Springer (2020)
10

---

[3] Cai, W., Liu, Q. Wang, Y.: Hiptrack: Visual Tracking with Historical Prompts. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
[4] Chen, X., Peng, H., Wang, D., Lu, H., Hu, H.: Unified Sequence-to-Sequence Learning for Single- and Multi-Modal Visual Object Tracking. arXiv (2023)
[5] Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., Lu, H.: Transformer tracking. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8126 8135 (2021)
[6] Chu, P., Wang. J., You, Q., Ling, H., Liu, Z.: Transmot: Spatial-tempont graph transformer for
multiple object tracking. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 4870-4880 (2023)
[7] Dai, P., Feng, Y., Weng, R., Zhang, C.: Joint spatial-temprrl and appearance modeling with transformer for multiple object tracking. arXiv preprint arXiv:2205.1544 (2022)
[8] Danelljan, M., Bhat, G., Khan, F.S., Felsberg, M.: Atom: Accurate tracking by overlap maximization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4660-4669 (2019)
[9] Danelljan, M., Gool, L.V., Timofte, R.: Probabilistic regression for visual tracking. In: Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition. pp. 7183-7192 (2020)
[10] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al.: An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11922 (2020)
[11] Fu, Y., Li, M., Liu, W., Wang. Y., Zhang, Yin, B., Wei, X., Yang, X.: Distractor-Aware Event-Based Tracking. IEEE Transactions on Image Processing 32, 6129-6141 (2023)
[12] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders scalable vision learners. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16000-1600) (2022)
[13] Hong, L., Yan, S., Zhang, R., Li, W., Zhou, X., Guo, P., Jiang, K., Cheng, Y., Li, J., Chen, Z., Zhang, W.: Onetracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
[14] Kuhn, H.W.: The hungarian method for the assignment problem. Naval research logistics quarterly 2(1-2), 83-97 (1955)
[15] Li, B., Wu, W., Wang, Q.. Zhang, F., Xing, J., Yan, J.: Siamrpn++: Evolution of siamese visual tracking with very deep networks. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 4282-4291 (2019)
[16] Lin. T.Y., Goyal, P.. Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980-2988 (2017)
[17) Lu, J., Li, S., Guo, W., Zhao, M., Yang, J., Liu, Y., Zhou, Z.: Siamese graph attention networks for robust visual object tracking. Computer Vision and Image Understanding p. 103634 (2023)
[18) Mayer, C., Danelljan, M., Bhat, G., Paul, M., Paudel, D.P., Yu, F,, Van Gool, L.: Transforming model prediction for tracking. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8731-8740 (2022)
[19] Mayer, C., Danelljan, M., Paudel, D.P.. Van Gool, L.: Learning target candidate association to keep track of what not to track. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14444-13444 (2021)
[20] Neubeck, A., Van Gool, L..: Efficient non-maximum suppression. 18th international confer- ence pattern recognition (ICPR '6)). vol. 3, pp. 850-855. IEEE (2006)

---

[21] Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized inter- section over union: A metric and loss for bounding box regression. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 658 6666 (2019)
[22] Su, H., Wei, S., Yan, M., Wang, C., Shi, J., Zhang, X.: Object detection and instance segmen- tation in remote sensing imagery based on precise mask r-cnn. In: IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium. pp. 1454-1457. IEEE (2019)
[23] Sun, P., Cao, J., Jiang. Y., Zhang, R., Xie, E., Yuan, Z., Wang. C., Luo, P.: Transtrack: Multiple object tracking with transformer. arXiv preprint arXiv: (2020)
[24] Tang, C., Wang, X., Huang. J., Jiang, B., Zhu, L.., Zhang, J., Wang, Y., Tian, Y.: Revisiting color- event based tracking: A unified network, dataset, and metric. arXiv preprint arXiv:2211.11010 (2022)
[25] Touvron, H., Cord, M., Jégou, H.: Deit ii:: Revenge of the vit. In: European conference on computer vision. pp. 516-533. Springer (2022)
[26] Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv: 10903 (2017)
[27] Voigtlaender, P., Luiten, J., Torr, P.H., Leibe, B.: Siam r-cmn: Visual tracking by re-detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6578-6588 (2020)
[28] Wang, N., Zhou, W., Wang. J., Li, H.: Transformer meets tracker: Exploiting temporal context for robust visual tracking. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1571-1580 (2021)
[29] Wang, X., Li, J., Zhu, L.,, Zhang, Z., Chen, Z., Li, X., Wang, Y., Tian, Y., Wu, F:: Visevent: Reliable object tracking via collaboration of frame and event flows. IEEE Transactions on Cybernetics (2023)
[30] Wang, X., Wang. S., Tang, C., Zhu, L., Jiang, B., Tian, Y., Tang. J.: Event Stream-based Visual Object Tracking: A Hegg--Resluttio Benchmark Dataset and A Novel Baseline. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
[31] Wu, Z., Zheng, J., Ren, X., Vasluianu, F.A., Ma, C., Paudel, D., Gool, L.V., Timofte, R.: Single-Model and Any-Modality for Video Object Tracking. In: IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) (2024)
[32] Xie, F, Chu, L., Li, J., Lu, Y., Ma. C.: Videotrack: Learning to track objects via video transformer. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22826-22222 2203522225222 (2023)
[33] Yan, B., Peng, H., Fu, J., Wang. D., Lu, H.: Learning spatio-temporal transformer for visual tracking. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10448-10457 (2021)
[34] Yan, Y. Mao, Y., Li, B.: Second: Sparsely embedded convolutional detection. Sensors 18(10), 3337 (2018)
[35] Ye, B., Chang, H., Ma. Shan, S., Chen, X.: Joint feature learning and relation modeling for tracking: A one-stream framework. In: European Conference on Computer Vision. pp. 341-357, Springer (2022)
[36] Zhang. J., Dong, B., Zhang, H., Ding, J., Heide, F., Yin, B., Yang, X.: Spiking Transformers for Event-based Single Object Tracking. In: Computer Vision and Pattern Recognition (CVPR). pp. 8791-8800 (2022)
[37] Zhang, J., Wang, Y., Liu, W., Li, M., Bai, J., Yin, B., Yang, X.: Frame-event alignment and fusion network for high frame rate tracking. In: Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition. pp. 9781-9790 (2023)
12

---

[38] Zhang, J., Yang. X., Fu, Y., Wei, X., Yin, B., Dong, B.: Object tracking by jointly exploit- ing frame and event domain. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13043-13052 (2021)
[39] Zhang, J., Zhao, K., Dong, B., Fu, Y., Wang, Y. Yang, X., Yin, B.: Multi-domain collaborative feature representation for robust visual object tracking. The Visual Computer 37(9-11), 2671- 2683 (2021)
[40] Zhang, T., Jin, Z., Debattista, K., Zhang, Q., Han, J.: Enhancing visual tracking with unified temporal Transformer framework. IEEE Transactions on Intelligent Vehicles pp. 1-15 (2024)
[41] Zhang, Z., Zhong, B., Zhang, S., Tang, Z., Liu, X., Zhang, Z.: Distractor-aware fast tracking via dynamic convolutions and mot philosophy. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1024-1025 1033 (2021)
[42] Zhu, J., Lai, S., Chen, X., Wang, D., Lu, H.: Visual prompt multi-modal tracking. In: Proceed- ings of the IERE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9516-9526 (2023)
[43] Zhu, Z., Hou, J., Lyu, X.: Learning graph-embedded key-event back-tracing for object tracking in event clouds. Advances in Neural Information Processing Systems 35, 7462-7476 (2022)
[44] Zhu, Z., Hou, J., Wu, D.O.: Cross-modal orthogonal high-rank augmentation for rgb-event transformer-tracken, In: Proceedings of the IEER/CVF International Conference Computer Vision. pp. 22045-22055 (2023)
[45] Zhu, Z., Hou, J., Wu, D.O.: Cross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackees In: IEEE International Conference on Computer Vision (ICCV). pp. 21988-21999 (2023)
13

---

Appendix
In this supplementary material, first provide details of the appearance model in Sec. A Subse- quently, we introduce the training and inference details of the candidate matching network in Sec. [ More experimental results shown in Sec.
A Appearance Model
In this section, we first describe the architectural details of the appearance model, which consists of the input representation, the projection layer, the backbone network and the tracking head. Then, we introduce the training details of the Appearance model.
A.1 Architectural details of the appearance model
In this paper, we employ an existing RGB-E tracker, i.e., CEUTrack[244 as our appearance model, which can be divided into four main parts: input representations, projection layer, backbone network and head network, as shown in Fig.
Transformer block
Trans Transforattion
Add Norm
Projection Layer
FFN
Head network
x
Add & Norm
N
Multi
Result
Head-A tlention
Layer Norm
www.raaeeeeeeeeeeeo....... www.
Backhone network
Figure 7: Architectures of the appearance model.
Input Representation: We employ the method used CEUtrack (4) to process the input RGB images and event data. Specifically, the RGB and event data can be represented as follows:
I (Ii,
& [ea) (1r,, Ms. La, p.I}
(5)
where I, denotes the t-th RGB image, (0t, 91) represents the spatial coordinates with {1...., W),j E (1..... "I....,H,, H}. Here, W, H are the width and height of the RGB images. E, is composed of multiple event points cs, where points. (x,, ys) are spatial coordinates of the event point, t, is the timestamp of each event point, and P. represents the polarity of the corresponding event point, i.e. an ON or OFF event.
f1....,S}; S}. represents the total number event
For the RGB frames, we first crop the template and search region, each containing 2 times and 4 times more extensive regions than the provided annotations, respectively. Then, the template and search patches of the RGB modality are resized into zī E xHHx*W,x3 and x E IR H respectively. In this paper, the template patch is uniformly resized into 128 128 while the search patch resized to 256 x 256 x 3.
For the event data, to transform the event points into voxels, first split the whole event stream into & based on the time interval of the RGB frames, obtaining {Ei,.., &.). is the total number of RGB frames. Then, transform each event set E, to volumetric representations V (To, Bp. tq: fop via the voxelization process SECOND 34 where Q is the total voxel number the current voxel set. (Fn, 3/p 1q) E Rxx3 and Sapq E I Q denote the 3D coordinates and corresponding feature representations, respectively. Subsequently, the voxel grids of the event modality, whose coordinates are not the template or search region, are removed. We further select the top 4096 and 1024 voxel grids in the cropped template and search voxels based the density of
14

---

event points, respectively. thereby obtaining the template voxels zj E R 1024x 16 and search voxels 10
x
We employ a projection layer, consisting of four parallel Comw_BN BN__eelu blocks, transform the input RGB and event data into token sequences. Specifically, for the RGB frame, similar to VIT[10], the template z and search patches x are projected to feature embeddings P7 E XC and 1x E RN,xC by using two non-shared 16 16 convolution layers, where N₂ W. and Nx W. H. For the event modality, the search voxel x and template voxel zî are projected to feature embeddings Fx E RNNxC and F E RNx by using two non-shared x convolution layers. Furthermore, the position embeddings in VIT are further added to those feature embeddings.
Projection layer:
16 16
16 16
The template and search tokens of the RGB modality and the event modality will be first concatenated Ur and then fed into the vanilla ViT, which consists of 12 stacked Transformer layers, to extract RGB features. Each Transformer layer consists of two layernorms (LN), the multi-head self-attention (MHSA) block, multi -alli-layyerree perceptron (MLP), and some residual connections. The detailed computation process can be formulated as:
Backbone network:
MHSA(Q, K, V) Softmax U, Ur MHSA (LN1 (Up, UI, Ut)),
QK Vdg V,
(6)
Up Ü. MLP (LNz (Ú:))
In our experiments, employ ViT-Base 10 and ViT-Tiny 10] our backbone network, resepctively,
To obtain the locations the candidates, we employ a standard tracking head in OSTrack 35 to directly estimate the center positions and scales of the bounding boxes. Specifically, the tracking head contains three branches that predict the classification score, center offset, and the size of bounding boxes, respectively. Each branch consists of four 3 x 3 Conv BN___N_RRE REEL and Con layer, respectively.
Tracking Head:
1 x
A.2 Training details of the appearance model
To train our proposed appearance model, we employ three loss functions: focal loss [16] for classifi- cation, L1 loss and GIOU loss 21 for bounding box regression. The overall loss function be written as:
L Å Tsicaal + AgTa + Xa Tggionn
where the hyper-parameters X1, A2, and A are set as 1, 1, and 14, respectively.
In the training phase, as illustrated in Sec. 4.1, we first employ the same training setting as that in OSTrack(35 (35] to train RGB tracker. Secondly, we use the RGB-E tracking datasets to train our appearance model. For COESOT (24), we train our proposed appearance model by using the training subset of COESOT. For VisEvent 29) and FE108 [38], train our proposed appearance model by using the training subset of VisEvent. Horizontal flip and brightness jittering are used for data augmentation during model training. Our appearance model is optimized by the AdamW optimizer with weight decay of x 10 for 50 epochs. The initial learning rate for the backbone and other parameters were set to 4 10-5 and x 10-4 respectively.
B Candidate matching model
In this section, we provide detailed information of the architectural details as well as the training and inference details of the candidate matching model.
B.1 Architectural details of the candidate matching network
Adjust layer: Based on the classification and regression results of the appearance model, we can extract the features of the candidates within each frame. For CSAM-B and CSAM-T, a projection layer employed to adjust the dimensions of the RGB and event features from the backbone network to 192 and 256, respectively.
Candidate Embedding Module:
15

---

STTE and DBTD: In the candidate matching model, both the proposed STTE and DBTD apply one individual layer. For CSAM-B, each attention block in the proposed STTE and DBTD employs multi-head attention layers with 6 heads. For CSAM-T, each attention block in the proposed STTE and DBTD employs multi-head attention layers with heads.
B.2 Training details of the candidate matching model
In the existing RGB-E datasets, only the target objects and the corresponding locations are provided for training. To effectively supervise the learning of the candidate matching model, the partial supervision loss and the self-supervision loss in Keep'rrack [99] are employed.
Partially Supervised I Loss: Here, we formulate the problem of target candidate association across two subsequent frames as, obtaining the affinity matrix A between the two candidate sets. the target candidate of corresponds to o Aij 1, otherwise Ar O. For each consecutive frame in a video sequence, we retrieve the single candidate corresponding to the annotated target. For the candidates ...," the assignment matrix A can be obtained by the proposed candidate matching network, which reflects the association between eg and of- The supervised loss is then given by the negative log-likelihood of the assignment probability,
Lsp = log A.,--
(8)
Self-Supervised Loss:
To improve the robustness of the candidate matching network, the appearance
model is first employed to predict the candidate sets V and its corresponding ground-trute truth association set C {ii, i) } from any given frame. Then, series of candidate sets (V"-T ..... V--13 are generated from V by feature augmentation. The feature augmentation involves randomly translating the locations of the candidates, randomly adjusting the classification scores of the candidates, and transforming the given image before extracting the multi-modal features. The self-supervised loss is given by,
Lietft
Σ log A.j- (N,))CC
Finally, we combine both self-supervised loss Lustt and partially supervised Loss Luup as Lootal = Luelf + Lsap.
To train the candidate matching network. we use the same training data for the appearance model. The appearance model is first employed to generate the score maps, the search regions and the regression results of each sequence, thereby locating the candidates. During the training of the candidate matching network, the weights of the appearance model will be frozen. Our candidate matching network is optimized by the Adam optimizer with weight decay of 0.2 for 15 epochs. The initial learning rate is set to 4 x 10-5
B.3 Inference details of the candidate matching netwrok
In this sub-section, we provide the detailed algorithm that describes the candidate matching associ- ation. During the inference, we first generate several candidates via the appearance model. Then, based on the affinity matrix A produced by the candidate matching network, we check whether a target candidate matches any of those previously detected objects. Only the tracklet-candidate pairs with affinity larger than 7sh 0.75 can be associated. After that, straightforward Hungarian algorithm [4] is utilized to generate the tracking output. If unmatched candidates exist, they will be connected to newly initialized trajectories. Finally, we check whether the object previously selected as the target is still visible in the current frame. If the previous tracklet of the target is visible in the current frame, the new target in the current frame. If the previous tracklet of the target is invisible in the current frame, we check the classification scores of other candidates. When a candidate with the highest classification score is greater than the threshold Ç = 0.25, we select this candidate as the new target. When the classification scores of all candidates are lower than the threshold that the target is not visible in the current frame and search for the target again in the next frame.
select the candidate which is matched to the previous tracklet of the target as
0.25, we determine
C Experiments
We provide more details to complement the state-of-the-an comparisons presented in the paper.
16

---

ALL
ROI
ALL
ROT
BOM
DFF
BOM
70
DEP
65
75
70
IV
11
IV
LI
NMO
OV
NMO
OV
FM
POc
POC
MB
SV
MD
SV
BC
(a) The PR scores of different attributes.
(b) The SR scores of differeent attributes.
Soy Trecs2-2255
OneTrack
VIPT
Ours
Figure 8: The SR and PR scores of the proposed CSAM and other RGB-E trackers under different attributes VisEvent.
C.1 Attributes Performance
To further verify the effectiveness of our proposed CSAM, we further analyze the serribute-based performance on VisEvent (29]. VisEvent annotates each testing video sequence with 17 attributes facilitate researchers in assessing the performance their trackers under specific challenging scenes. These challenges encompass full occlusion (FOC), deformation (DEF), rotation (ROT), fast motion (FM), partial occlusion (POC), low illumination (LI), scale variation (SV), background object motion (BOM), motion blur (MB), overexposure (OE), camera motion (CM), out of the view (OV). Viewpoint change (VC), Background clutter (BC), illumination variation (IV), no motion (NMO) and aspect ratio change (ARC). As shown in Fig. we analyze the attribute-basee based performance on VisEvent (29]. For clarity, we only illustrate 4 RGB-E trackers, including ViPT [42]. SDSTrack B1 One'Track 13 and SeqTrackv22BB2
Details of VisEvent:
44. All of these compared trackers employ the same training subset and backbone network. From the results, we can see that our proposed method performs the best in all annotated attributions.
Compared with existing RGB-E trackers, our approach exhibits significant improvements, particularly in cases LI, CM, NMO and OE. For the above-mentioned challenges, the data of a certain modality cannot contain valid target information demonstrates the ability of our CSAM to fully leverage the complementary information within RGB-E data. Furthermore, in the challenges information, our CSAM leverages motion information within the scenes accurately determine target positions. Besides, considering the common occurrence of distractors in tracking scenarios, desirable performance various challenges demonstrates the effectiveness of employing MOT philosophy.
tracking. Our proposed method still performs well, which
FOC and OV, where scenes lack discernible appearance
C.2 Visualization of Candidate Features.
In the proposed CSAM framework, differentiating various candidates the key improving tracking robustness. In Figh, candidate features are visualized via t-SNE. With the help of the proposed spatial encoder, we can better distinguish candidates the tracking scenes. demonstrates the effectiveness of motion information provided by the event stream on enhancing tracking performance.
C.3 Qualitative performance
The visual comparisons between our proposed method and the other four state-of-the-arr tracking algorithms, including CEUTrack (24). HRCEU'rrack [44), OSTrack (35) and KeepTrack [19], are illustrated in Fig. TO We can observe that our method performs better than other trackers these complex scenes, such cases with background cluster, motion blur and low illumination.
17

---

Tegeet
Detsole
Deget
Ditnohel
Dimated
Dissen
Toget
Danater:
Beaved
Diorrate Distooted
Nashal
Mash
Nasd
Evern imague
Nakhal
Inoonddd Summer STTE
Ehooied Foueros STTE
Maach reado with SITT.
Figure 9: Candidate feature clustered by I-SNE.
#70
#1
#30
#55
#60
#120
CSAM-B
OSTack
HRCEUTock
KeepTrock
CEUTack
GT
Figure 10: Visualization of tracking results on COBSOT dataset. Event images used for visual comparison only.
C.4
The impact of CSAM on distractors:
Based on our appearance tracker's response maps, FE108 averages 1.26 distractors per frame, while COESOT has 1.91 distractors per frame. Tracking accuracy declines significantly in the presence of distractors. We categorize sequences into 1) $1: 291 distractor-fre sequences and 2) S2: 237 sequences with distractors, as shown Table 4 Our framework steadily enhances tracking performance in sequences with distractors. What's 's more, compared with ViPT[42), our tracker achieves notable performance gains 4.7% and 4.6% in PR and SR, respectively.
C.5 Experiments of Different MOT Philosophies.
Table presents the performance of using various multi-object tracking methodologies for candidate matching, including KeepTrrack [19 and TransSTAM (23] For fair comparisons, all these variants employ the same appearance model for candidate generation and utilize the same training data and loss functions for model training. The results in T that our proposed CSAM performs the best. Existing MOT methods generally concentrate on temporal appearance information from RGB images and motion information from historical candidate locations. In contrast, our proposed CSAM leverages the additional motion information from the event stream, thereby enhancing the robustness of candidate matching.
18

---

Table 4: The influence of distractors COESOT.
Sequences
ViPT
Appearance Tracker
CSAM
A
S1 (PR/SR)
77.6/68.4
78.2/68.6
78.2/68.6
S2 (PR/SR)
70.8/63.4
72.0/62.9
75.5/68.0
3.5/5.1
Table 5: Ablation study of different MOT philosophies.
Method
KeepTrack [19
TranSTAM 23
Ours
PR/SR
75.1/65.7
75.4/66.3
76.7/68.1
Table 6: Ablation study T.
T=1
T=5
T=10
T=15
T=20
PR
75.7
75.9
76.1
76.7
76.6
Table 7: Ablation study of Tth-
==55
T0,50.65
Tea 0175
T+A=0.85
PR
75.1
75.7
76.7
76.2
Table 8: Ablation study of ç.
(=0.0
<=0.25
(=0.5
Ç=0.75
PR
75.1
76.7
76.1
75.2
Table 9: Ablation study of the number of layers in STTE and DBTD.
Layer Number
1
2
3
4
PR
76.7
76.9
75.7
75.3
C.6 Experiments of the hyper-parameters
In the proposed CSAM framework, the involved hyper-paun parameters are the time interval T, relevance threshold Teh and classification threshold Ç.
The ablation study of T: The ablation study about T is shown in Table 6 When T=1, there is temporal information included in the proposed framework. T> 1 can improve the association accuracy. In addition, since increasing T also adds more tracklets for the association, increases the complexity of the association task. Finally, T 15 is used in all experiments.
The ablation study TzA: We show the ablation studies Teh in Table 7 we achieve the best performance when =075, When is too small, the proposed tracker may cause false tracklet- matching. When Tsh is too big, the proposed tracker may not be able to complete the track-matching.
The ablation study of Ç: We show the ablation studies of in Table g we achieve the best performance when (=0.25 When threshold Ç is too big, is hard to relocate the target.
The number of layers the proposed STTE and DBTD: In addition, we provide the experiments about using more layers in STTE and DBTD in Table 8 We found that layer number of 2 can achieve better performance but at the expense of operational efficiency. Furthermore, more layers will introduce large number of parameters, which may cause over fitting problems and lead to performance degradation.
D Societal impacts
Object tracking has diverse applications extending from visual surveillance systems, autonomous vehicles and intelligent transportation systems. In-depth research on RGB-E tracking has wide
19

---

range of positive impacts on practical applications in these fields. However, the misuse of Object tracking technology can have negative impact on personal privacy.
20

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
We claim our contributions in the abstract (Lines 5-18) and introduction (Lines 74-82).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our proposed work conclusion (Lines 362-364).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:[Yes)
21

---

Justification: Our proposed method aims to design an effective structure (Lines: 120-254) to improve the tracking performance. And we verify the effectiveness of our method through experiments. (Lines: 255-352, 605-659).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our contribution is specific model. We describe the proposed methods in Section (Lines: 120-254), and provide the implementation details in Section 4.1 (Lines 256-276) and Appendices and (Lines 485-604).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
22

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [No]
Justification: We illustrate the exact command and environment needed to run to reproduce the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwtpcc/pcccpcpcccceooooooop / ceabbbmssssonni seiniisssiipyccyy Policy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the training and test details in Sec 4 (Lines 256-291) and Appendix (Lines 539-551, 564-604).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? [No]:
Question:
Justification: We conducted experiments multiple times on the same equipment and found that the experimental results were fixed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the needed computer resources in Sec 4.3 (Lines 318-324).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpps:/neeurpp:: http:://wwurrpp.cppppoiieee
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the both potential positive societal impacts and negative societal impacts of the work on Appendix (Lines 659-663).
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "CJ3ujpX2pr", "reviewer": "Reviewer_fGZm", "summary": "This paper focuses on RGB-Event tracking. The authors propose to leverage MOT philosophy to distinguish the distractors to enhance the robustness of the tracker. Following a tracking-by-detection framework, the authors first generate a series of candidates and then match them with historical known priors with a CEM, STTE and DBTD module. The authors conduct experiments on several datasets to demonstrate the proposed methods.", "questions": "Spatio-temporal architecture is widely used in SOT and MOT, what is the novelty of the article?", "limitations": "Please refer to the above weaknesses.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The proposed method achieves state-of-the-art results on several datasets.", "weaknesses": "- The motivation of the article is unclear. The article aims to enhance the robustness of the tracker by distinguishing between interfering objects. However, instead of analyzing in detail the causes of the model's inability to distinguish between interfering objects, the authors introduce a number of complex modules to perform the matching. In addition, the authors only analyze the effectiveness of the proposed modules through some experimental results, without detailed and deep analysis such as visually demonstrating the proposed modules through feature visualization, trajectory visualization, and so on. Therefore, I am concerned about the innovativeness of this paper.\n- The proposed method seems to be a combination of existing techniques. Spatio-temporal encoder-decoder is widely explored in SOT and MOT community. The authors seem to have only modified some blocks, yet what does that have to do with what the article is claiming to prove.\n- The article writing needs to be improved. Line 5-6 is a misrepresentation, beacuse the RGB-Event tracker, which tracks a single target, does not need to track interfering objects. I can understand that the author utilizes the MOT idea, but this representation may misleading the readers.\n- The model is complex, and I'm concerned about whether the model is far more complex and computationally intensive than other algorithms.\n- No open source code", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your constructive and insightful comments. Thanks for your recognition of our work. \n\n**Q1. The motivation of the paper is unclear. Lack of analysis.**\n\n**Motivation:**\nAs shown in Fig. 1 (a) in the manuscript, the co-occurrence of distractor objects similar in appearance to the target is a common problem in the SOT task. Most SOT trackers based on appearance information struggle to identify the target in such cases, often leading to tracking failure. \n\nExisting RGB-E trackers primarily concentrate on exploring complementary appearance information within RGB and event data to enhance tracking performance. \nDespite achieving commendable improvements, mainstream RGB-E tracking algorithms still cannot solve the association problem of the targets and distractor objects in the temporal domain.\n\nSome RGB trackers (e.g., KYS, KeepTrack, TrDiMP and DMTrack) propagate valuable scene information to improve the discriminative ability.\nHowever, these methods are susceptible to environmental interference, and their matching strategies relying on appearance information may miss the target when the target and distractor trajectories are close. \n\nIn fact, event data can not only provide the edge information to improve the RGB feature representations but also contains abundant motion cues to reflect the motion state of the objects, which is meaningful to differentiate between targets and distractors, even if they may look similar. The event camera offers a new perspective to reflect the motion state of the objects, which is crucial to accurately match the tracklet in the tracking sequences.\n\nTherefore, we propose an Appearance-Motion Modeling RGB-E tracking framework to dynamically incorporate motion information as well as appearance information contained in the RGB-E videos to track all the candidates with a Multi-Object Tracking (MOT) philosophy, thus avoiding the tracking drift.\n\n**Experimental results:** \nWe provide the qualitative analysis (trajectory visualization) in lines 634-644 (Fig 9， Fig 10).\nIn addition, we provide the visualization of candidate features via t-SNE in lines 629-633 (Fig 8).\nFurthermore, more ablation studies are covered in Appendix C to verify the effectiveness of the proposed framework. We will add more visualization results in the revision.\n\n\n**Q2. Spatio-temporal encoder-decoder is explored in SOT and MOT community**\n\n**Different Motivations with existing SOT methods:**\nSome SOT works (e.g., TrDiMP, TCTrack, HIPTrack) employ the encoder-decoder structure to explore historical information. \nMost of these methods do not explicitly distinguish the motion trajectories of distractors in the scene.\nThe encoder and decoder are usually used to combine historical features and enhance current frame features, respectively.\nOur work is different, in the proposed CSAM framework, the encoder establishes the association between different candidates in each frame and the temporal relationships of each candidate. And the decoder is used to match candidates with historical tracklets.\n\n**Different Design with existing  MOT methods:**\nNumerous MOT methods use the encoder-decoder structure to construct the temporal relationships and generate the assignment matrix for matching. However, most of them rely on the appearance information from the input RGB data.\nThese methods are susceptible to environmental interference, and their matching strategies relying on appearance information may miss the target when the target and distractor trajectories are close.\nDifferently, in the proposed CSAM framework, taking into account the fact that event data can provide reliable motion status, we fully exploit the appearance and motion information in the tracking scene during the encoding and decoding stages to enhance tracking performance.\n\n\n**Q3. Improving Writing.**\n\nSorry for the unclear description. We will modify the abstract in the revision as follows: RGB-Event single object tracking (SOT) aims to leverage the merits of RGB and event data to achieve higher performance. \nHowever, existing frameworks focus on exploring complementary appearance information within multi-modal data, and struggle to address the association problem of targets and distractors in the temporal domain using motion information from the event stream. \nIn this paper, we introduce the Multi-Object Tracking (MOT) philosophy into RGB-E SOT to keep track of targets as well as distractors by using both RGB and event data.\n\n**Q4. The model is complex.**\n\nIn lines 318-324, Table 2 shows that our CSAM introduces limited computation costs (5.6 ms latency and 14.4M parameters), while significantly improving the tracking performance.\nIn addition, compared with recently advanced trackers, i.e., ViPT and SeqTrackv2-L256, our model size has not increased significantly and we can still run at real-time speeds.\n\n**Q5: No open source code.**\n\nWe describe the proposed methods in Section 3 and provide the implementation details in Section 4.1, Appendices A and B.\nWe illustrate the exact command and environment needed to run to reproduce\nthe results. We will release the source code when the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DkDStq1hrZ", "reviewer": "Reviewer_KN3J", "summary": "•\tThis paper proposes a novel RGB-E tracking framework, i.e., CSAM, which first predicts the candidates by using an appearance model and then keeps track of both targets and distractors with an MOT philosophy. The model show significantly improved state-of-the-art results.", "questions": "•\tPlease refer to weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to understand.\n2. Comparisons over several datasets demonstrate the effectiveness of tracking multiple candidates for robust tracking.", "weaknesses": "Because I am not a researcher on RGB-E tracking, and the authors have given me enough information in the main text and appendix, it is difficult for me to point out the shortcomings from a technical perspective. One small suggestion is that the authors gave some examples of qualitative analysis in the supplementary materials, but it is recommended to put some of the content in the main text, so that readers can understand the advantages and disadvantages of the method by combining quantitative numerical results and qualitative examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your constructive and insightful comments. Thanks for your recognition of our work. \n\n**Q1. Putting some qualitative analysis in the main text.**\n\nDue to the limited page space, our previous manuscript only provides the qualitative analysis in the supplementary materials. We will put some of them in the main text in the revision space permitting. \n\nIn addition, please allow me to restate the novelty of our work briefly:\n\n1) To the best of our knowledge, we are the first to introduce the MOT philosophy for the SOT task using RGB-E data. The proposed CSAM framework significantly improves the tracker's ability to cope with tracking drift caused by distractors.\n\n2) We propose three effective modules: a Candidate Encoding Module, a Spatial-Temporal Transformer Encoder and a Dual-branch Transformer Decoder. \nThe Candidate Encoding Module initially aggregates the RGB features and corresponding classification scores to generate the appearance embedding. Meanwhile, the event features and corresponding bounding box coordinates are fused to obtain motion information.\nIn the Spatial-Temporal Transformer Encoder, the spatial encoder construct\nspatial correlations of each frame by using motion information, and the temporal encoder establishes the temporal relationships of each tracklet.\nIn the proposed Dual-branch Transformer Decoder, we also generate the assignment matrix by using both the appearance information and motion information, simultaneously.\n\n3) We show significantly improved state-of-the-art results of our proposed method on multiple RGB-E tracking benchmarks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "maKx9ZSVot", "reviewer": "Reviewer_SevM", "summary": "A novel RGB-E tracking framework with MOT philosophy has been proposed in order to keep track of both targets and distractors to robustly track a single object. It includes a Candidate Encoding Module, a Spatial-Temporal Transformer Encoder and a Dual-branch Transformer Decoder. Within these modules, the authors exploited the appearance information in combination with motion cues to improve the candidate matching and affinities. Experimental validation is comprehensive and results on three RGB-E object tracking benchmarks are state-of-the-art.", "questions": "See the Weakness and give the corresponding explaination.", "limitations": "NA", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Using the MOT philosophy with spatial-temporal information fusion to solve the similar distractors problem in RGB-E tracking domain is new attempt.\n\n2. The proposed method achieves the new sota RGB-E SOT tracking results.", "weaknesses": "1.The ablation study about N/M/T is missing and the selection of hyperparameters(e.g., τth and ζ) lacks discussion.\n\n2.Does the delay caused by the Hungarian algorithm constitute the majority of the additional time compared with the pure appearance-based tracker? \n\n3.Is the training process carried out in a sequential iterative manner? That is to say, every frame will be output a prediction result or just the last frame in a batch. It is best to provide details in the main paper.\n\n4.Have authors tried more layers in STTE and DBTD? Will they lead to continuous improvement in performance?\n\n5.More thorough checks are needed for the paper writing and illustrations, including but not limited to the following parts:\na. The description about ΓEN in Line216 is not consistent with that in Figure 3\nb. In Figure 2, “TIR” should be “Event”. Colors of Appearance and Motion feature embeddings are swapped.\nc. In Figure 5,  A^{sa} should be  A^s. M^{t-1}should only have N blocks not M.\nd. In Line336 and Line338, “encoder” should be “decoder”.\ne. “COEST” should be “COESOT”. It has multiple mis-usages across whole paper.\nf. In Figure8, “SGTE” should be “STTE”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt gratitude for your perceptive insights and your recognition of the unique and meaningful contributions made by our research. Your support is highly valued, and we would be honored if you could serve as an advocate for our work.\n\n**Q1. The ablation study about N/M/T/$\\tau_{th}$/$\\zeta$:**\n\n**N and M:** N and M denote the number of candidates of the previous and current frame respectively. Therefore, N and M are actually determined by the tracking sequences and are not one of our settings. \nIn supplementary C.5, we provide the influence of candidates to illustrate the effectiveness of the proposed framework. \n\n**The ablation study of T:**\nThe ablation study about T is shown below. When T=1, there is no temporal information included in the proposed framework. T > 1 can improve the association accuracy. In addition, since increasing T also adds more tracklets for the association, it increases the complexity of the association task. Finally, T = 15 is used in all experiments.\n\n|   | T=1| T=5| T=10| T=15| T=20|\n|---------|------|------|------|------|------|\n| PR score| 75.7 | 75.9 | 76.1  | 76.7| 76.6|\n\n**The ablation study of $\\tau_{th}$:**\nWe show the ablation studies of $\\tau_{th}$ below. As shown in the following table, we achieve the best performance when $\\tau_{th}$=0.75.\nWhen $\\tau_{th}$ is too small, the proposed tracker may cause false tracklet-matching. When $\\tau_{th}$ is too big, the proposed tracker may not be able to complete the tracklet-matching. \n\n|   | $\\tau_{th}$=0.55| $\\tau_{th}$=0.65| $\\tau_{th}$=0.75| $\\tau_{th}$=0.85|\n|---------|------|------|------|------|\n| PR score| 75.1 | 75.7 | 76.7  | 76.2|\n\n**The ablation study of $\\zeta$:**\nWe show the ablation studies of $\\zeta$ below. As shown in the following table, we achieve the best performance when $\\zeta$=0.25\nWhen threshold $\\zeta$ is too big, it is hard to relocate the target. \n\n|   | $\\zeta$=0.0| $\\zeta$=0.25| $\\zeta$=0.5| $\\zeta$=0.75|\n|---------|------|------|------|------|\n| PR score| 75.1 | 76.7 | 76.1  | 75.5|\n\n**Q2. The delay of the Hungarian algorithm.**\n\n**Hungarian algorithm:** As shown in the table below, the Hungarian algorithm takes 0.6 ms for object association. Because when we tested on the SOT dataset, most tracking scenes only contained a limited number of candidates (Please refer to Appendix C.5).  \n\n**Other parts:** To model the temporal associations in historical frames, the temporal encoder takes a longer reasoning time in the CSAM framework. In the future, we will study how to use a more compact representation of historical information to efficiently establish temporal associations.\n\n| Module | Appearance model | NMS | Candidate Encoding Module| Spatial Encoder|\n|---------|------|------|------|------|\n| Latency (ms) | 13.3 | 0.8| 0.5  | 0.8|\n\n| Module | Temporal encoder | Dual-branch Transformer Decoder| Hungarian algorithm|  |\n|---------|------|------|------|------|\n| Latency (ms) | 2.1| 0.7| 0.6  |  |\n\n**Q3. Training process.**\n\nWe do not use the sequential iterative manner during training.\nWe employ a similar training strategy as that in KeepTrack, which has been carefully introduced in Appendix A and B.\nFor partial supervision, we use two consecutive frames as the historical frames and select one frame to output the prediction results and calculate the training loss.\nFor self-supervision training, we only require a single frame and its candidates for training loss calculation. \nWe will add relevant content in the main text in the revision.\n\n**Q4. More layers in STTE and DBTD**\n\nWe provide the experiments about using more layers in STTE and DBTD in the following Table.\nWe found that a layer number of 2 can achieve better performance but at the expense of operational efficiency.\nFurthermore, more layers will introduce a large number of parameters, which may cause overfitting problems and lead to performance degradation.\n\n| Layer Number | 1| 2| 3| 4|\n|---------|------|------|------|------|\n| PR score| 76.7 | 76.9 | 75.7  | 75.3|\n\n**Q5. More thorough checks are needed.**\n\nWe are very sorry for the above mistakes in the previous manuscript. We will modify all of these typos in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3xieLF0LBh", "reviewer": "Reviewer_fErD", "summary": "This work proposes a novel RGB-E tracking framework, CSAM. CSAM first predicts candidates and then tracks both targets and distractors using MOT philosophy. Comprehensive experiments are conducted on three RGB-E tracking benchmarks, showing that CSAM achieves state-of-the-art performance.", "questions": "1. Where are the bottlenecks in the inference latency of the CSAM method, and how much time does each module consume? Additionally, the Candidate Embedding Module in CSAM uses NMS, which is likely one of the most time-consuming aspects. If so, can anchor-free methods address this?\n2. Why does CSAM-B achieve a high FPS in Table 2 despite having a larger model size and higher FLOPs compared to SeqTrackv2-B256?\n3. In MoT, the frequent appearance and disappearance of targets is a major challenge. The applicability of the CSAM method in this context is particularly interesting, as discussed on pages 130-134 of the paper. The authors could include more examples of tracking in situations where objects frequently disappear due to occlusion.\n4. The total training time on two RTX 3090 GPUs?\n5. Why are RSR and RPR metrics used in FE108, while SR and PR metrics are used in COESOT and VisEvent? What are the distinctions between them?\n6. The DVS sensor introduces significant event noise during nighttime or in varying lighting conditions. It appears that the CSAM method implicitly filters out this noise using Transformer. However, the authors voxelized the DVS sensor input without applying denoising, could this potentially degrade performance? Besides, what unique approach does CSAM employ for handling DVS sensor data? CSAM appears to be applicable to other modality combinations.", "limitations": "Please see the weaknesses part.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clear and the problem studied is important\n2. This is the first work to introduce MOT philosophy to the SOT task using RGB-E data.\n3. The authors proposed a novel RGB-E tracking framework, CSAM.\n4. Extensive experimental results back up the effectiveness of CSAM.", "weaknesses": "1. Adding more experimental results under varying illumination conditions could be helpful to highlight the necessity of DVS.\n2. The paper uses too many abbreviations, especially in CSAM, where each module has its own set of abbreviations, making it hard to remember them all.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to express our gratitude for your support and valuable insights. It's reassuring to know that you recognize the originality and efficacy of our research. We would greatly appreciate your continued support in championing our work.\n\n**W1 and Q6. More experimental results under varying illumination conditions**\n\n**Advantages and disadvantages of event stream:** \nAs discussed with FE108, VisEvent and COESOT, event cameras (Dynamic Vision Sensors (DVS)) excel at capturing motion information at very low latency,  and are almost free from the trouble of motion blur.  In addition, DVS sensors also outperform the visible (RGB) cameras on dynamic range (140 vs 60dB), which enables them to work effectively even under poor illumination conditions. Simultaneously, we noticed that the event camera cannot provide effective information for tracking of slow-moving or stationary objects. \n\n**Input representation:**\nPrevious event trackers usually transform the asynchronous event flows into synchronous event images by stacking the events in a fixed time interval. CEUTrack transforms the event points into\nvoxels which preserves the temporal information well, and the voxels top-selection mechanism also fully exploits the sparse representation of event data which reduces the computational complexity. In this paper, we also adopt the same transformation to get the event images but focus on designing a novel tracking framework to explore both the effective appearance information and motion information. Besides, the proposed CSAM framework employs both the appearance information and the motion information to suppress the noisy information within the event data.\n\n**Cannot be directly applied to other multimodal tracking tasks:**\nConsidering the motion information provided by the event data, in the proposed STTE, we design a spatial transformer encoder to establish distinctive spatial relationships by using an event stream. In addition, in the proposed DBTD, a motion decoder is introduced to match each tracklet by using the event tokens. Differently, for other multi-modal tracking tasks (e.g., RGB-Thermal, RGB-Depth), it is still difficult for us to construct accurate spatial correlations or match tracklets by using other modality data like thermal images or depth images.\n\n**Attributes Performance:**\nHere, in order to prove the necessity of DVS, we provide the results of with and without event stream under the challenges of low illumination (LI) and Illumination variation (IV). As shown below,  our tracker with event stream shows substantially better results, which demonstrates the effectiveness of DVS.\n\n| Challenges     | LI | IV | \n|---------|------|------|\n| CSAM with event data (PR score)| 74.1 | 82.3| \n| CSAM without event data (PR score)| 50.8 | 71.8| \n\n\n**W2. Too many abbreviations.**\n\nwe are very sorry for the confusion. In the revision, we will avoid unnecessary abbreviations, and the major mathematical symbols used in this paper will be summarized in a Table for ease of explanation in the revision. \n\n**Q1. Latency of the CSAM method.**\n\n**Latency of NMS:**\nIn this Table, we provide the time each module takes. We found that NMS takes very little time in the PyTorch framework. This is because the tracking scenes usually contain only a limited number of candidates. \nBesides, we can use the cuda version of NMS to speed up its operation.\n\n**Latency of Hungarian algorithm:**\nSimilarly, due to the limited number of candidates, the Hungarian algorithm used in multi-target tracking will not take too long. To model the temporal associations in historical frames, the temporal encoder occupies a longer reasoning time in the CSAM framework.\nIn the future, we will study how to use a more compact representation of historical information to efficiently establish temporal associations.\n\n| Module | Appearance model | NMS | Candidate Encoding Module| Spatial Encoder|\n|---------|------|------|------|------|\n| Latency (ms) | 13.3 | 0.8| 0.5  | 0.8|\n\n| Module | Temporal encoder | Dual-branch Transformer Decoder| Hungarian algorithm|  |\n|---------|------|------|------|------|\n| Latency (ms) | 2.1| 0.7| 0.6  |  |\n\n**Anchor-free:** Our appearance model follows the anchor-free pipeline to generate a series of candidates. The candidate matching network also does not use anchor boxes. We argue that the design of a one-stage framework in the future can improve the tracking efficiency; this will be our follow-on work.\n\n**Q2. The FPS and FLOPs of SeqTrackv2-B256.**\n\nSeqTrack casts visual tracking as a sequence generation task, forecasting object bounding boxes in an autoregressive manner. Such a design requires the gradual generation of tracking results and frequent access to intermediate calculation results, resulting in lower FLOPs and slower inference speed. \n\n**Q3. More examples of appearance and disappearance of targets.**\n\nWe will add more examples in the revision. We also provide the tracking performance under the challenges of FOC (target is fully occluded) and OV (target completely leaves the video sequence), we noticed that our framework significantly improves the tracking performance when targets frequently appear and disappear.\n\n| Challenges     | FOC| OV| \n|---------|------|------|\n| CSAM-B| 70.7 | 70.3| \n| Apperance model| 62.6 | 61.3| \n\n**Q4. The total training time.**\n\nOur proposed model is trained in two stages. In the first stage, the appearance model’s training took 12 hours. In the second stage, the proposed model took 7 hours for training.\n\n**Q5. Different metrics in FE108, COESOT and VisEvent**\n\nAs in FENet[38], to mitigate small alignment errors, we utilize two widely used metrics, i.e., maximum precision rate (MPR) and maximum success rate (MSR), to evaluate the tracking performance on FE108. Differently, VisEvent and COESOT employ better alignment, they directly use the PR and SR metrics to evaluate different trackers. All evaluations are tested under the official codes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the comprehensive reviews provided by all the reviewers. The valuable feedback has significantly contributed to improving the quality of our manuscript. We extend our gratitude to Reviewer KN3J, Reviewer SevM, and Reviewer fErD for recognizing the novelty of our work. Their positive acknowledgments of our research's innovation are greatly appreciated. Additionally, we kindly request Reviewer fGZm to reconsider our work after reviewing our response. Your reconsideration will be highly valued.

Based on the comments from the reviewers, I have summarized the strengths of our paper as follows:

1. The utilization of event data combined with the MOT philosophy can ensure more robust target identification for SOT tasks.

2. The proposed CSAM framework only introduces limited parameters and computational costs but significantly improves the tracking performance.

3. The proposed method achieves very competitive performance compared with state-of-the-art methods. Also, an ablation study is conducted to validate the effectiveness of different components in the proposed method. 

4. There is currently a lack of research on multi-modal multi-target tracking. This paper can provide a new perspective for future research.

We have summarized our novelty as follows:

1. To the best of our knowledge, we are the first to introduce the MOT philosophy for the SOT task using RGB-E data. The proposed CSAM framework significantly improves the tracker's ability to cope with tracking drift caused by distractors.

2. We propose three effective modules: a Candidate Encoding Module, a Spatial-Temporal Transformer Encoder and a Dual-branch Transformer Decoder. 
The Candidate Encoding Module initially aggregates the RGB features and corresponding classification scores to generate the appearance embedding. The event features and corresponding bounding box coordinates are also fused to obtain motion information.
In the Spatial-Temporal Transformer Encoder, the spatial encoder constructs
spatial correlations of each frame by using motion information, and the temporal encoder establishes the temporal relationships of each tracklet.
In the proposed Dual-branch Transformer Decoder, we also generate the assignment matrix by using both the appearance information and motion information, simultaneously.

3. We show significantly improved state-of-the-art results of our proposed method on multiple RGB-E tracking benchmarks.

We believe that these innovative contributions enhance the value and significance of our research in the field of object tracking. We kindly request the reviewers to reassess our work in light of these contributions and extend their support to our efforts.

We plan to implement the reviewers' insightful suggestions by incorporating additional essential experiments. Furthermore, we will thoroughly review the manuscript to correct any typographical and grammatical errors.

We have provided detailed responses to each reviewer, carefully addressing all specific points raised. We sincerely thank the reviewers for their valuable feedback and appreciate the dedication of the program chair and area chair. Your support in our endeavors is greatly appreciated. We are fully committed to addressing the concerns raised and refining our manuscript accordingly.

Meta-review: This paper proposes a novel framework that integrates Multi-Object Tracking (MOT) philosophy into RGB-Event (RGB-E) tracking, aiming to enhance robustness by distinguishing between targets and distractors. The approach utilizes a Candidate Encoding Module, a Spatial-Temporal Transformer Encoder, and a Dual-Branch Transformer Decoder to effectively leverage both appearance and motion information. Some reviewers raised concerns about the clarity of the motivation and the complexity of the proposed model. Suggestions were made to include more qualitative analysis and refine the writing for better clarity. Despite these concerns, the paper presents a valuable contribution to the field of RGB-E tracking, especially by addressing the challenges of distractors. The ACs recommend accepting the paper, with the expectation that the authors will address the reviewers' feedback in the final version.
Decision: Accept (poster)