Revisiting motion information for RGB-Event tracking with MOT philosophy

Authors: tianlu  zhang
kurt  debattista
qiang  zhang
guiguang  ding
jungong  han
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 01:46:50

Abstract: RGB-Event single object tracking (SOT) aims to leverage the merits of RGB and event data to achieve higher performance. However, existing frameworks focus on exploring complementary appearance information within multi-modal data, and struggle to address the association problem of targets and distractors in the temporal domain using motion information from the event stream. In this paper, we introduce the Multi-Object Tracking (MOT) philosophy into RGB-E SOT to keep track of targets as well as distractors by using both RGB and event data, thereby improving the robustness of the tracker. Specifically, an appearance model is employed to predict the initial candidates. Subsequently, the initially predicted tracking results, in combination with the RGB-E features, are encoded into appearance and motion embeddings, respectively. Furthermore, a Spatial-Temporal Transformer Encoder is proposed to model the spatial-temporal relationships and learn discriminative features for each candidate through guidance of the appearance-motion embeddings. Simultaneously, a Dual-Branch Transformer Decoder is designed to adopt such motion and appearance information for candidate matching, thus distinguishing between targets and distractors. The proposed method is evaluated on multiple benchmark datasets and achieves state-of-the-art performance on all the datasets tested.

Main paper: 
 introduction: 
 Single object tracking (SOT) aims to predict the position of a target in videos, by being given only the position of the target in the initial frame. While traditional RGB-based trackers [1, 8] can effectively capture comprehensive scene representations, including color and semantic information, they face significant performance degradation in challenging conditions like fast motion, low illumination and distractions from similar objects.To address such challenges associated with frame-based cameras, some researchers have taken advantage of event cameras [39] , which are characterized by high temporal resolution and high dynamic range, to augment the RGB data for reliable object tracking. In the past few years, various methods have been proposed for RGB-Event (RGB-E) object tracking. Existing RGB-E trackers primarily concentrate on exploring complementary appearance information within RGB and event data to enhance tracking performance with three typical approaches i.e., early fusion method [29] , middle fusion method [37, 38] and one-stream method [24, 44] . Despite achieving commendable improvements, mainstream RGB-E tracking algorithms still cannot solve the association problem of the targets and distractor objects in the temporal domain, as shown in Fig. 1 (a) .Alternatively, some RGB trackers [2, 3, 19] propagate valuable scene information through the sequence to improve their discriminative ability, as shown in Fig. 1 (b) . These methods mine the information in two main ways. The first one relies on implicitly transforming the scene information to locate the targets, and the scene information is generally represented using a set of embeddings [2, 3] . To ensure effective transformation and avoid introducing noisy information, these methods usually require careful design of the encoding strategy to obtain effective scene information embeddings. The second approach explicitly explores the scene information by simultaneously keeping track of both targets and distractors [19, 41] . However, these methods are susceptible to environmental interference, and their matching strategies relying on appearance information may miss the target when the target and distractor trajectories are close.In fact, event data can not only provide the edge information to improve the RGB feature representations but also contains abundant motion cues to reflect the motion state of the objects, which is meaningful to differentiate between targets and distractors, even if they may look similar. Motivated by these observations, we propose an Appearance-Motion Modeling RGB-E tracking framework with a Cascade Structure, referred to as CSAM, that goes beyond leveraging complementary appearance information within RGB-E data. As shown in Fig. 1 (c ), the proposed CSAM framework employs an appearance model to initially determine the candidates with similar appearance to the targets, and then designs a candidate matching network with encoder-decoder structure to dynamically incorporate motion information contained in the RGB-E videos to track all the candidates with the Multi-Object Tracking (MOT) philosophy. The candidate that matches the historical target tracklet will be regarded as the final tracking result. Specifically, the candidate matching network consists of a Candidate Encoding Module (CEM), a Spatial-Temporal Transformer Encoder (STTE), and a Dual-branch Transformer Decoder (DBTD). Recognizing the critical roles of the two types of information in candidate association, our proposed CEM is used to encode both appearance cues and motion representations for each candidate. Subsequently, an STTE block, comprising a Spatial Encoder and a Temporal Encoder, is introduced to model spatial-temporal relationships among candidates by synergistically utilizing such appearance and motion embeddings. Finally, a DBTD block, comprising a Spatial-temporal Decoder and a Motion Decoder, is presented to match candidates with historical tracklets by using both appearance and motion information.Contributions: In summary, our contributions are: (i) We propose a novel RGB-E tracking framework, i.e., CSAM, which first predicts the candidates by using an appearance model and then keeps track of both targets and distractors with an MOT philosophy. To the best of our knowledge, we are the first to introduce the MOT philosophy for the SOT task using RGB-E data. (ii) We propose three effective modules: a Candidate Encoding Module, a Spatial-Temporal Transformer Encoder and a Dual-branch Transformer Decoder. The appearance information as well as the motion cues within the RGB-E data can be fully exploited by the proposed modules for accurate candidate association. (iii) We show significantly improved state-of-the-art results of our proposed method on multiple RGB-E tracking benchmarks.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] We claim our contributions in the abstract (Lines 5-18) and introduction (Lines 74-82).Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Visual object tracking: The current prevalent tracking pipelines can be categorized into three groups: CNN-based trackers, CNN-Transformer trackers and Transformer-based trackers. CNNbased trackers utilize a Siamese network [17, 15] or Discriminative Correlation Filter (DCF) [8, 1] to address tracking tasks by matching templates and search regions. However, the inherent properties of CNNs limit their ability for global information exploration and interaction, thereby constraining the advancement of CNN-based trackers. In response, some CNN-Transformer trackers [33, 5, 28] employ attention mechanisms to establish global dependencies between template features and search features. But these hybrid CNN-Transformer trackers still independently extract features from templates and search regions using CNN networks, resulting in extracted features being unaware of the tracking target. To address this issue, several pure Transformer-based trackers [35, 32, 40] overcome the challenge by unifying feature extraction and feature relation modeling through a single Transformer backbone, leading to state-of-the-art tracking performance.RGB-E object tracking: In recent years, there has been a growing interest among researchers in merging RGB frames and event streams for object tracking. Some researchers focus on exploring complementary information within RGB-E data via specially designed cross-modal interaction strategies [29, 24, 44] . For instance, Zhang et al. [29] proposed a cross-domain feature integrator to dynamically fuse visual cues from both the frame and event domains. Alternatively, CEUTtrack [24] proposed a one-stream framework based on the Transformer, which simultaneously addresses feature extraction, template-search relation modeling, and cross-modal interaction. Recently, some methods [13, 42, 31] aim to adapt the RGB tracking model to RGB-E tracking in the prompt learning manner. However, existing RGB-E tracking frameworks cannot fully explore the abundant motion cues within the event stream, consequently limiting tracking performance in the presence of distractors.Multi object tracking: Multi-object tracking (MOT) aims to track multiple objects in a video sequence. Currently, the tracking-by-detection paradigm [7, 6] , where an object detector is initially employed to locate all proposals, followed by an association network to match all of these objects, is gaining popularity for the MOT task. Additionally, some researchers have explored the jointdetection-and-tracking pipeline [23] , aiming to achieve detection and tracking simultaneously in a single stage. There are some approaches that aim to enhance the tracking performance of Single Object Tracking (SOT) via the use of an MOT philosophy. For instance, DMTrack [41] designed a lightweight detector and an explicit object association module to track both targets and distractors. KeepTrack [19] proposed a learnable candidate matching network and designed several mechanisms, including partial supervision, self-supervised learning and sample-mining, to address the problem of incomplete annotation in SOT training data. However, these methods only use the RGB modality and overlook the importance of spatial-temporal relationships among candidates for matching candidates and tracklets.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of our proposed work in conclusion (Lines 362-364).Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 As shown in Fig. 2 , our framework first employs an appearance model to generate the potential proposals. Subsequently, several modules are proposed to identify targets and maintain tracking of all candidates to prevent tracking drift. Specifically, the appearance model predicts the target scores and bounding boxes for M candidates of the t -th current frame and N candidates for each previous frame (see Sec.3.1) . Secondly, a set of features are extracted for each candidate from T previous frames, including target classification scores, appearance features, event embeddings and candidate locations. These features are then aggregated into appearance embeddings and motion embeddings for each proposal (see Sec.3.2). Thirdly, the STTE is employed to jointly model spatial-temporal relationships for each tracklet (see Sec.3.3). Fourthly, utilizing the DBTD, an N × M assignment matrix is calculated for matching tracklets from previous frames with candidates from the current frame (see Sec.3.4). It should be noted that not every tracklet has all the candidates in the previous T frames due to occlusion, missing detection, etc. For illustrative purposes, we consider the situation that there are no missing candidates for each tracklet. Our method solves the cases with missing tracklets similarly to the typical multi-object tracking method [6] (see supplementary material Sec.B). In the following, we will describe the proposed tracking framework in detail.Here, we employ CEUTrack [24] as our appearance model. Specifically, the event streams are initially transformed into voxel representations through a voxelization operation [24] . Subsequently, given the initial locations and tracking results, we crop the template patch, the template voxel, the search patch and the search voxel, respectively. After that, the projection layers are adopted to transform the four inputs into token representations, which are then fed into the vanilla ViT [10] for joint feature extraction, cross-modal interaction and search-template matching. Finally, the tracking head, employing the same structure as that in OStrack [35] , takes the concatenated RGB and event search region features from the backbone as input to predict the appearance tracking results. Please refer to the supplementary material (Sec.A.1) for additional details about the appearance model.The goal of CEM is to first select initial candidates similar to the target and filter out the most simple negative backgrounds, and then obtain the appearance embeddings as well as the motion embeddings. With the classification scores and regression offsets outputted by the appearance model, we generate N candidates similar to the target for each previous frame by using Non-maximum Suppression (NMS) [20] . The appearance features of each candidate can be obtained by performing the PRoIAligh [22] on the RGB backbone features based on its corresponding location. We can represent the appearance features of each candidate in the (t -T) -th frame asF t-T = {f t-T N ∈ R d } Nn=1 and represent their corresponding classification scores asC t-T = {c t-T N ∈ R 1 } N n=1 .Both the appearance features and the classification scores convey essential appearance cues for each candidate. To integrate these two types of information, we propose an Appearance Feature Encoding (AFE) layer. Specifically, AFE processes the backbone features f t-T N via a single convolution layer to obtain more discriminative features f e t-T N ∈ R d and employs several MLP layers on c t-T N to generate the classification embeddings ce t-T N ∈ R d . These features are then combined as:a t-T N = f e t-T N + ce t-TN . The appearance feature embeddings for the (t -T) -th frame are thus represented byA t-T = {a t-T N } N n=1 . Graph Multi Head-Attention FFN 𝐴𝐴 t-1 𝑀𝑀 t-1Temporal Encoder Additionally, we obtain the event features of each candidate in the (t -T) -th frame as… 𝑎𝑎 N t-1 𝑎𝑎 2 t-1 𝑎𝑎 1 t-1 … 𝑚𝑚 N t-1 𝑚𝑚 2 t-1 𝑚𝑚 1 t-1 … 𝑠𝑠 N t-1 𝑠𝑠 2 t-1 𝑠𝑠 1 t-1 … 𝑡𝑡 N t-T 𝑡𝑡 N t-2 𝑡𝑡 N t-1 Add & Norm Add & Norm Multi Head-Attention FFN Add & Norm Add & Norm Add & Norm 𝐴𝐴𝐴𝐴𝐸𝐸 t-1 𝐴𝐴𝐴𝐴𝐸𝐸 t-2 𝐴𝐴𝐴𝐴𝐸𝐸 t-𝑇𝑇 … Re-arrangement … 𝑠𝑠 N t-𝑇𝑇 𝑠𝑠 N t-2 𝑠𝑠 N t-1 Γ N Γ 2 Γ 1 … … 𝑚𝑚 N t-T 𝑚𝑚 N t-2 𝑚𝑚 N t-1 Θ N Γ𝐸𝐸 N Γ𝐸𝐸 2 Γ𝐸𝐸 1 … Γ𝐸𝐸 1 Γ𝐸𝐸 N-1 … 𝜩𝜩 𝐞𝐞𝐞𝐞 Re-arrangement Γ N Γ𝐸𝐸 NE t-T = {e t-T N ∈ R d } Nn=1 and represent their locations asP t-T = {p t-T N ∈ R 4 } N n=1, where p t-T N = {x t-T N , y t-T N , w t-T N , h t-T N } denotes the normalized bounding box coordinates. Both the event stream and the location set contain rich motion information about those candidates. To fuse these two types of features, a Motion Feature Encoding (MFE) layer, which has a similar structure of AFE, is first used to obtain the enhanced event representations ee t-T N ∈ R d and location embeddings pe t-T N ∈ R d . Then these features are fused as: m t-T N = ee t-T N + pe t-T N . The motion feature embeddings of each candidate in the (t -T) -th frame can be thus represented by M t-T = {m t-T N } N n=1 .The proposed STTE aims at learning more discriminative feature representations for each tracklet and establishing effective relationships among objects in both spatial and temporal domains. The inputs for STTE include T sets of appearance embeddings {A t-T , ..., A t-1 } and T sets of motion representations {M t-T , ..., M t-1 }. All of these embeddings will be first processed via a Spatial Encoder to construct the spatial relationships among candidates in each frame. Subsequently, these spatially encoded features are re-arranged to construct N tracklets across T frames, and their temporal relationships are established via the proposed Temporal Encoder. 𝑊𝑊 v 𝑉𝑉 a t-1 𝐴𝐴 t-1 𝑀𝑀 t-1 … 𝑎𝑎 N t-1 𝑎𝑎 2 t-1 𝑎𝑎 1 t-1 … 𝑚𝑚 N t-1 𝑚𝑚 2 t-1 𝑚𝑚 1 t-1 𝑉𝑉 m t-1 𝑊𝑊 q … 𝑒𝑒𝑒𝑒 𝑖𝑖1 𝑒𝑒𝑒𝑒 𝑖𝑖N 𝑚𝑚 𝑖𝑖 𝐭𝐭-𝟏𝟏 𝑊𝑊 k … 𝑚𝑚𝑒𝑒 𝑖𝑖 𝐭𝐭-𝟏𝟏 𝑎𝑎 𝑖𝑖 t-1 FFN … ŝ𝑠N t-1 ŝ𝑠2 t-1 ŝ𝑠1 t-1The proposed Spatial Encoder independently processes each frame to construct spatial correlations, and we take the (t -1) -th frame as an example for illustration. It is difficult to establish distinctive spatial relationships by only using the appearance information. Differently, the motion information is more suitable to establish meaningful spatial affinities. Inspired by the graph attention networks [26] , each candidate's appearance embedding is regarded as a node, and the edge weight between each nodes is defined by the motion information.Specially, at the (t -1) -th frame, we consider each candidate's appearance and motion representations in A t-1 and M t-1 as two node sets V t-1 a and V t-1 m , respectively. As shown in Fig. 3 (a), we use a complete bipartite graph G t-1 m = (V t-1 m , E t-1 m ) to model the object-level relations between these candidates. Here,E t-1 m = {(u, v)|∀u, v ∈ V t-1 m }.Then, the edge weight between node i and node j in V t-1 m will be denoted as e ij , which can be calculated through the inner product operation. More specifically, as that in a typical Transformer block, some normalization layers and linear transformations are first applied on these motion nodes, followed by an inner product calculation, to achieve e ij . Formally,EQUATIONwhere W k and W q are the linear transformations and e ij will be further normalized by the softmax fuction, obtainingEQUATIONWith the edge weights passed from all nodes in V t-1 m to the i-th node in V t-1 m , the aggregated representation of the i-th node in V t-1 m can be transformed by:EQUATIONwhere W v is a matrix for linear transformation. It should be noted that we adopt the multi-head attention structure to improve the discriminability of graph attention learning.Finally, we fuse the aggregated features me t-1 i with the appearance features a t-1 i to obtain a more powerful feature representation:EQUATIONwhere cat(•) represents vector concatenation, F F N (•) denotes the feedforward neural network. The final spatial encoded features ASE t-1 = {s t-1 1 , ..., s t-1 N } are obtained by further employing the residual connections and two FFN layers as that in a typical Transformer block. These spatial encoded features are re-arranged to N tracklet sets {Γ 1 , ..., Γ N } of all candidates through the T frames, where Γ N = {s t-T N , ..., s t-1 N }. Meanwhile, the motion embeddings are also re-arranged to N tracklet sets {Θ 1 , ..., Θ N }, where Θ N = {m t-T N , ..., m t-1 N }. Temporal Transformer Encoder: These N tracklet sets are further encoded by a Temporal Transformer Encoder. Here, we take N -th tracklet as an example for illustration. As shown in Fig. 3 , we first fuse the spatial encoded features Γ N with the motion feature set Θ N by the element-wise addition operation, and then employ the multi-head attention to calculate the attention weights A N ∈ R T×T , thus generating the attention-weighted features. After that, these weighted features are processed through two FFN layers and residual connections to obtain the final output ΓE N , which consists of the N -th tracklet's feature representations {t t-T N , ..., t t-1 N }. The outputs of the Temporal Transformer Encoder {ΓE 1 , ..., ΓE N } are re-arranged to Ξ en ∈ R NT×d . en as the query, and uses the encoded features of the previous frame Ξ en as the key and value. The Multi-Head Attention mechanism [24] is calculated for Ξcan en and Ξ en to generate attention weights. The output passes through two FFN layers and residual connections, generating the output tensor R NM×d . The output of the appearance decoder can be processed through an FFN and a softmax layer to generate the assignment matrix A s ∈ R N×M . Moreover, to better match each tracklet with the candidates in the current frame, the motion information of the (t -1) -th frame is utilized as the motion information for the tracklet. The Motion Decoder employs the same structure as that in the appearance decoder to obtain the assignment matrix A m ∈ R N×M . The final assignment matrix A can be obtained by A = A s + A m .t … 𝑎𝑎 M t 𝑎𝑎 2 t 𝑎𝑎 1 t Motion Decoder Multi Head-Attention FFN Add & Norm Add & Norm Spatial Encoder 𝑀𝑀 t Γ𝐸𝐸 N Γ𝐸𝐸 2 Γ𝐸𝐸 1 … 𝜩𝜩 𝐞𝐞𝐞𝐞 … 𝐴𝐴 M t 𝐴𝐴 2 t 𝐴𝐴 1 t FFN SoftMax Multi Head-Attention FFN Add & Norm Add & Norm FFN SoftMax … 𝑚𝑚 N t-1 𝑚𝑚 2 t-1 𝑚𝑚 1 t-1 … 𝑚𝑚 M t 𝑚𝑚 2 t 𝑚𝑚 1 t Spatial-Temporal Decoder A 𝑨𝑨 𝐦𝐦 𝑨𝑨 𝐬𝐬 Add 𝑀𝑀 t-1 � 𝜩𝜩 𝐜𝐜𝐜𝐜𝐞𝐞 𝐞𝐞𝐞𝐞 Re-arrangement & DuplicationDuring inference, we first employ the appearance model to generate candidates of each frame. If only one target with a high score is present in both the previous and current frames, this candidate is selected as the target, and the candidate matching model is omitted to reduce computational costs and accelerate inference. In contrast, when multiple candidates exist during tracking, the assignment matrix A is calculated using the proposed candidate matching model. After that, a threshold τ th is adopted on A to remove the ambiguous correspondence. Finally, we match the predicted boxes and the candidate boxes using the Hungarian algorithm [14] . The candidates that do not match any tracklet will be assigned a new ID, and the tracklets that do not match any of the detections in the past consecutive T frames will be terminated. T is experimentally set to 15. Please refer to the supplementary material (Sec.B.3) for more details.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer:[Yes] 
experiment: 
 Table 1 : Comparison with state-of-the-art trackers on COESOT [24] , FE108 [38] and VisEvent [29] . The numbers with red and blue colors indicate the best and the second best results, respectively. Architectures: We instantiate two models of CSAM: CSAM-T and CSAM-B, by varying the backbone network in the appearance model, i.e., ViT-Tiny and ViT-Base. We initialize ViT-Tiny using the weights from DeiT-tiny [25] , and the backbone weights ViT-B are initialized with corresponding MAE encoders [12] . In the candidate matching network, both the proposed STTE and DBTD apply one individual layer. Please refer to the supplementary material (Sec.A.1 and Sec.B.1) for more details.Training: The training of our CSAM comprises three parts. In the first part, the following three loss functions are adopted: the focal loss for classification, and L1 loss and GIOU loss for bounding box regression [35] . We employ the same training setting as that in OSTrack [35] to train an RGB tracker. Secondly, we employ the same training setting as that in HRCEUTrack [45] to train the appearance model. In the third part, the parameters of the appearance model are fixed and other parameters in our proposed framework are set to be trainable. Since only the targets' locations are provided in the existing RGB-E tracking dataset, the partial supervision loss and self-supervised loss in KeepTrack [19] are employed to supervise the assignment matrix A generated by our proposed model. Please refer to the supplementary material (Sec.A.2 and Sec.B.2) for additional details about the implementation details.Dataset: We evaluate the performance of our proposed CSAM on three large-scale RGB-E single object tracking datasets: VisEvent [29] FE108 [38] and COESOT [24] . These three datasets were captured using DAVIS346, with a spatial resolution of 346 × 260, a dynamic range of 120 dB and the minimum latency of 20 µs. The COESOT [24] dataset comprises 578K RGB-E pairs, divided into 827 and 527 sequences for training and testing, respectively. These sequences were collected from both indoor and outdoor scenarios, covering a range of 90 classes and 17 challenging attributes. The FE108 [38] dataset contains 108 RGB-E sequences, which capture 21 different types of objects. It is divided into 76 and 32 sequences for training and testing, respectively. VisEvent [29] dataset collects 820 RGB-E video pairs, divided into 500 and 320 sequences for training and testing, respectively. Following [37] , after removing sequences that miss event data or have misaligned timestamps, the VisEvent dataset includes 377 sequences for training and 172 for testing.Metrics: In FE108 [38] , we use representative success rate (RSR) and representative precision rate (RPR) to evaluate all trackers. In COESOT [24] and VisEvent [29] , we use success rate (SR), precision rate (PR) and normalized precision rate (NPR) for evaluation.To show the effectiveness of the proposed method, we evaluate and compare our CSAM with several state-of-the-art trackers, including 4 Event trackers and 17 RGB-E trackers. As shown in Table . 1. * denotes that the RGB trackers are extended to RGB-E trackers via the early fusion approach. † and * denotes that the model is pre-trained on RGB tracking datasets.Results on FE108: As shown in Table . 1, our proposed CSAM-B outperforms other top-performing trackers, such as HRMonTrack-B [44] , HDETrack [30] and TransT [5] , with a clear margin, and achieves the best performance with an RSR score of 70.5%. and an RPR score of 97.1%. Even when compared to HRMonTrack-B, which has already obtained impressive tracking performance, our approach demonstrates notable improvements, with a 2.0% increase in RSR and a 0.9% increase in RPR. These comparisons fully demonstrate the effectiveness of tracking multiple candidates for robust tracking.Results on VisEvent: From Table . 1, we find that our method sets a new state-of-the-art score on VisEvent. First, our proposed framework outperforms the Event trackers, e.g., STNet [36] and DANet [11] , by a clear margin. Secondly, compared with appearance trackers SDSTrack [31] and ViPT [42] , our model further improves the PR score by 4.1% and 1.6% in NPR scores, respectively. This enhancement is attributed to our model's comprehensive utilization of both appearance and motion information, enabling effective tracking of targets and distractors. Thirdly, our tracker surpasses the previous best tracker SeqTrackv2-L [4] , which demonstrates that our method has a stronger capability in handling various challenges. To verify the effectiveness of our designed framework, we perform ablation analysis to evaluate different components in our method by using the COESOT test set [24] . Please refer to the supplementary material (Sec.C) for more ablation experiments. Effectiveness of the proposed CEM: To investigate the impact of our proposed CEM, several versions of our proposed method are provided, including ①: Removing the AFE and MFE submodules in CEM block. ②: Removing the AFE sub-module in CEM block. ③: Removing the MFE sub-module in CEM block. As can be seen in Table 3 , the tracking performance degrades after removing AFE or MFE sub-modules, which demonstrates the necessity of embedding appearance and motion information from the classification scores and bounding box coordinates.To further verify the effectiveness of the proposed STTE, several variants are proposed, including ①: Removing the spatial encoder (SE) and temporal encoder (TE) . ②: Replacing the spatial encoder by the original Transformer Encoder (OTE) block [10] . ③: Replacing the spatial encoder by SuperGlue [19] . ④: Removing the temporal encoder in STTE block. ⑤: Removing the spatial encoder in STTE block. As can be seen in Table 3 , the tracking performance experiences a significant decline upon the removal of temporal encoder or spatial encoder, which confirms the necessity of spatial-temporal relationships in enhancing feature representations of the candidates. Furthermore, compared with several existing methods, e.g., OTE and SuperGlue, the proposed spatial encoder takes full advantage of the appearance information as well as motion information in constructing robust spatial correlations, thus achieving better results.To further verify the effectiveness of the proposed DBTD, several variants are proposed, including ①: Replacing the spatial-temporal decoder (STD) by the original Transformer Decoder (OTD) block [10] . ②: Removing the motion decoder (MD) in DBTD block. ③: Removing the spatial-temporal decoder in DBTD block. As can be seen in Table 3 , not using spatial-temporal decoder or motion decoder substantially deteriorates the performance. This demonstrates that the exploration of both appearance and motion information is crucial for ensuring robust tracking.  
conclusion: 
 In this paper, a novel RGB-E tracking framework with MOT philosophy has been proposed in order to keep track of both targets and distractors to robustly track a single object. Specifically, a Spatial-Temporal Transformer Encoder is proposed to establish a rich temporal-spatial relationships by using appearance information in combination with motion information. Furthermore, by formulating the tracklets and candidates with the appearance features and motion embeddings, the affinities of the tracklets and candidates are explicitly modeled and leveraged. We conduct comprehensive experimental validation and analysis of our approach on three RGB-E object tracking benchmarks and produce new state-of-the-art results.Limitation: The current method is dedicated to constructing an effective framework for RGB-E tracking with MOT philosophy, but it pays less effort to improve the efficiency of the supervision signals, which we will consider in future work.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We illustrate the exact command and environment needed to run to reproduce the results.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
acknowledgments: 
 This Input Representation: We employ the method used in CEUtrack [24] to process the input RGB images and event data. Specifically, the RGB and event data can be represented as follows:EQUATIONwhere I t denotes the t-th RGB image, (x i , y j ) represents the spatial coordinates with i ∈ {1, . . . , W}, j ∈ {1, . . . , H}. Here, W, H are the width and height of the RGB images. E t is composed of multiple event points e s , where s ∈ {1, . . . , S}. S represents the total number of event points. (x s , y s ) are spatial coordinates of the event point, t s is the timestamp of each event point, and p s represents the polarity of the corresponding event point, i.e., an ON or OFF event.For the RGB frames, we first crop the template and search region, each containing 2 times and 4 times more extensive regions than the provided annotations, respectively. Then, the template and search patches of the RGB modality are resized into z r 1 ∈ R Hz×Wz×3 and x r t ∈ R Hx×Wx×3 , respectively. In this paper, the template patch is uniformly resized into 128 × 128 × 3, while the search patch is resized to 256 × 256 × 3.For the event data, to transform the event points into voxels, we first split the whole event stream E into E t based on the time interval of the RGB frames, obtaining {E 1 , ..., E t }. T is the total number of RGB frames. Then, we transform each event set E t to volumetric representations V t = {[x o , y p , t q , f opq ]} Q l=1 via the voxelization process in SECOND [34] , where Q is the total voxel number of the current voxel set. (x o , y p , t q ) ∈ R Q×3 and f opq ∈ R Q×16 denote the 3D coordinates and corresponding feature representations, respectively. Subsequently, the voxel grids of the event modality, whose coordinates are not in the template or search region, are removed. We further select the top 4096 and 1024 voxel grids in the cropped template and search voxels based on the density of event points, respectively, thereby obtaining the template voxels z e 1 ∈ R 1024×16 and search voxels x e t ∈ R 4096×16 . Projection layer: We employ a projection layer, consisting of four parallel Conv_BN _Relu blocks, to transform the input RGB and event data into token sequences. Specifically, for the RGB frame, similar to ViT [10] , the template z r 1 and search patches x r t are projected to feature embeddings F r z ∈ R Nz×C and F r x ∈ R Nx×C by using two non-shared 16 × 16 convolution layers, where Backbone network: The template and search tokens of the RGB modality and the event modality will be first concatenated as U f and then fed into the vanilla ViT, which consists of 12 stacked Transformer layers, to extract RGB features. Each Transformer layer consists of two layernorms (LN), the multi-head self-attention (MHSA) block, multi-layer perceptron (MLP), and some residual connections. The detailed computation process can be formulated as:N z = WzEQUATIONIn our experiments, we employ ViT-Base [10] and ViT-Tiny [10] as our backbone network, resepctively.Tracking Head: To obtain the locations of the candidates, we employ a standard tracking head in OSTrack [35] to directly estimate the center positions and scales of the bounding boxes. Specifically, the tracking head contains three branches that predict the classification score, center offset, and the size of bounding boxes, respectively. Each branch consists of four 3 × 3 Conv_BN _RELU and 1 × 1 Conv layer, respectively. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?[No] :Justification: We conducted experiments multiple times on the same equipment and found that the experimental results were fixed.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To train our proposed appearance model, we employ three loss functions: focal loss [16] for classification, L1 loss and GIOU loss [21] for bounding box regression. The overall loss function can be written as:EQUATIONwhere the hyper-parameters λ 1 , λ 2 , and λ 3 are set as 1, 1, and 14, respectively.In the training phase, as illustrated in Sec. 4.1, we first employ the same training setting as that in OSTrack [35] to train an RGB tracker. Secondly, we use the RGB-E tracking datasets to train our appearance model. For COESOT [24] , we train our proposed appearance model by using the training subset of COESOT. For VisEvent [29] and FE108 [38] , we train our proposed appearance model by using the training subset of VisEvent. Horizontal flip and brightness jittering are used for data augmentation during model training. Our appearance model is optimized by the AdamW optimizer with a weight decay of 1 × 10 -4 for 50 epochs. The initial learning rate for the backbone and other parameters were set to 4 × 10 -5 and 4 × 10 -4 , respectively.In this section, we provide detailed information of the architectural details as well as the training and inference details of the candidate matching model.Candidate Embedding Module: Adjust layer: Based on the classification and regression results of the appearance model, we can extract the features of the candidates within each frame. For CSAM-B and CSAM-T, a projection layer is employed to adjust the dimensions of the RGB and event features from the backbone network to 192 and 256, respectively.In the candidate matching model, both the proposed STTE and DBTD apply one individual layer. For CSAM-B, each attention block in the proposed STTE and DBTD employs multi-head attention layers with 6 heads. For CSAM-T, each attention block in the proposed STTE and DBTD employs multi-head attention layers with 3 heads.In the existing RGB-E datasets, only the target objects and the corresponding locations are provided for training. To effectively supervise the learning of the candidate matching model, the partial supervision loss and the self-supervision loss in KeepTrack [19] are employed.Partially Supervised Loss: Here, we formulate the problem of target candidate association across two subsequent frames as, obtaining the affinity matrix A between the two candidate sets. If the target candidatev t i corresponds to v t-1 j , A ij = 1, otherwise A ij = 0.For each consecutive frame in a video sequence, we retrieve the single candidate corresponding to the annotated target. For the candidates {v t-T i , ..., v t i }, the assignment matrix A can be obtained by the proposed candidate matching network, which reflects the association between v t i and v t-1 i . The supervised loss is then given by the negative log-likelihood of the assignment probability,EQUATIONSelf-Supervised Loss: To improve the robustness of the candidate matching network, the appearance model is first employed to predict the candidate sets V t and its corresponding ground-truth association set C = {(i, i)} N i=1 from any given frame. Then, a series of candidate sets {V t-T , ..., V t-1 } are generated from V t by feature augmentation. The feature augmentation involves randomly translating the locations of the candidates, randomly adjusting the classification scores of the candidates, and transforming the given image before extracting the multi-modal features. The self-supervised loss is given by,EQUATIONFinally, we combine both self-supervised loss L self and partially supervised Loss L sup as L total = L self + L sup .To train the candidate matching network, we use the same training data for the appearance model. The appearance model is first employed to generate the score maps, the search regions and the regression results of each sequence, thereby locating the candidates. During the training of the candidate matching network, the weights of the appearance model will be frozen. Our candidate matching network is optimized by the Adam optimizer with a weight decay of 0.2 for 15 epochs. The initial learning rate is set to 4 × 10 -5 .In this sub-section, we provide the detailed algorithm that describes the candidate matching association. During the inference, we first generate several candidates via the appearance model. Then, based on the affinity matrix A produced by the candidate matching network, we check whether a target candidate matches any of those previously detected objects. Only the tracklet-candidate pairs with affinity larger than τ th = 0.75 can be associated. After that, a straightforward Hungarian algorithm [14] is utilized to generate the tracking output. If unmatched candidates exist, they will be connected to newly initialized trajectories. Finally, we check whether the object previously selected as the target is still visible in the current frame. If the previous tracklet of the target is visible in the current frame, we select the candidate which is matched to the previous tracklet of the target as the new target in the current frame. If the previous tracklet of the target is invisible in the current frame, we check the classification scores of other candidates. When a candidate with the highest classification score is greater than the threshold ζ = 0.25, we select this candidate as the new target.When the classification scores of all candidates are lower than the threshold ζ = 0.25, we determine that the target is not visible in the current frame and search for the target again in the next frame.We provide more details to complement the state-of-the-art comparisons presented in the paper. Details of VisEvent: To further verify the effectiveness of our proposed CSAM, we further analyze the attribute-based performance on VisEvent [29] . VisEvent annotates each testing video sequence with 17 attributes to facilitate researchers in assessing the performance of their trackers under specific challenging scenes. These challenges encompass full occlusion (FOC), deformation (DEF), rotation (ROT), fast motion (FM), partial occlusion (POC), low illumination (LI), scale variation (SV), background object motion (BOM), motion blur (MB), overexposure (OE), camera motion (CM), out of the view (OV), Viewpoint change (VC), Background clutter (BC), illumination variation (IV), no motion (NMO) and aspect ratio change (ARC). As shown in Fig. 8 , we analyze the attribute-based performance on VisEvent [29] . For clarity, we only illustrate 4 RGB-E trackers, including ViPT [42] , SDSTrack [31] OneTrack [13] and SeqTrackv2-B256 [4] . All of these compared trackers employ the same training subset and backbone network. From the results, we can see that our proposed method performs the best in all annotated attributions.Compared with existing RGB-E trackers, our approach exhibits significant improvements, particularly in cases LI, CM, NMO and OE. For the above-mentioned challenges, the data of a certain modality cannot contain valid target information for tracking. Our proposed method still performs well, which demonstrates the ability of our CSAM to fully leverage the complementary information within RGB-E data. Furthermore, in the challenges of FOC and OV, where scenes lack discernible appearance information, our CSAM leverages motion information within the scenes to accurately determine target positions. Besides, considering the common occurrence of distractors in tracking scenarios, our desirable performance on various challenges demonstrates the effectiveness of employing MOT philosophy.In the proposed CSAM framework, differentiating various candidates is the key to improving tracking robustness. In Fig 9 , candidate features are visualized via t-SNE. With the help of the proposed spatial encoder, we can better distinguish candidates in the tracking scenes. It demonstrates the effectiveness of motion information provided by the event stream on enhancing tracking performance.The visual comparisons between our proposed method and the other four state-of-the-art tracking algorithms, including CEUTrack [24] , HRCEUTrack [44] , OSTrack [35] and KeepTrack [19] , are illustrated in Fig. 10 . We can observe that our method performs better than other trackers in these complex scenes, such as cases with background cluster, motion blur and low illumination. Based on our appearance tracker's response maps, FE108 averages 1.26 distractors per frame, while COESOT has 1.91 distractors per frame. Tracking accuracy declines significantly in the presence of distractors. We categorize sequences into 1) S1: 291 distractor-free sequences and 2) S2: 237 sequences with distractors, as shown in Table 4 . Our framework steadily enhances tracking performance in sequences with distractors. What's more, compared with ViPT [42] , our tracker achieves notable performance gains of 4.7% and 4.6% in PR and SR, respectively.Table 5 presents the performance of using various multi-object tracking methodologies for candidate matching, including KeepTrack [19] and TransSTAM [23] . For fair comparisons, all these variants employ the same appearance model for candidate generation and utilize the same training data and loss functions for model training. The results in Table 5 reveal that our proposed CSAM performs the best. Existing MOT methods generally concentrate on temporal appearance information from the RGB images and motion information from historical candidate locations. In contrast, our proposed CSAM leverages the additional motion information from the event stream, thereby enhancing the robustness of candidate matching. range of positive impacts on practical applications in these fields. However, the misuse of Object tracking technology can have a negative impact on personal privacy. 


Decision: This paper proposes a novel framework that integrates Multi-Object Tracking (MOT) philosophy into RGB-Event (RGB-E) tracking, aiming to enhance robustness by distinguishing between targets and distractors. The approach utilizes a Candidate Encoding Module, a Spatial-Temporal Transformer Encoder, and a Dual-Branch Transformer Decoder to effectively leverage both appearance and motion information. Some reviewers raised concerns about the clarity of the motivation and the complexity of the proposed model. Suggestions were made to include more qualitative analysis and refine the writing for better clarity. Despite these concerns, the paper presents a valuable contribution to the field of RGB-E tracking, especially by addressing the challenges of distractors. The ACs recommend accepting the paper, with the expectation that the authors will address the reviewers' feedback in the final version.

Reviews: review: paper_summary: paper_summary: This paper focuses on RGB-Event tracking. The authors propose to leverage MOT philosophy to distinguish the distractors to enhance the robustness of the tracker. Following a tracking-by-detection framework, the authors first generate a series of candidates and then match them with historical known priors with a CEM, STTE and DBTD module. The authors conduct experiments on several datasets to demonstrate the proposed methods.
strength_weakness: strength_weakness: - The motivation of the article is unclear. The article aims to enhance the robustness of the tracker by distinguishing between interfering objects. However, instead of analyzing in detail the causes of the model's inability to distinguish between interfering objects, the authors introduce a number of complex modules to perform the matching. In addition, the authors only analyze the effectiveness of the proposed modules through some experimental results, without detailed and deep analysis such as visually demonstrating the proposed modules through feature visualization, trajectory visualization, and so on. Therefore, I am concerned about the innovativeness of this paper.
- The proposed method seems to be a combination of existing techniques. Spatio-temporal encoder-decoder is widely explored in SOT and MOT community. The authors seem to have only modified some blocks, yet what does that have to do with what the article is claiming to prove.
- The article writing needs to be improved. Line 5-6 is a misrepresentation, beacuse the RGB-Event tracker, which tracks a single target, does not need to track interfering objects. I can understand that the author utilizes the MOT idea, but this representation may misleading the readers.
- The model is complex, and I'm concerned about whether the model is far more complex and computationally intensive than other algorithms.
- No open source code
questions: questions: Spatio-temporal architecture is widely used in SOT and MOT, what is the novelty of the article?
limitations: limitations: Please refer to the above weaknesses.
paper_summary: paper_summary: •	This paper proposes a novel RGB-E tracking framework, i.e., CSAM, which first predicts the candidates by using an appearance model and then keeps track of both targets and distractors with an MOT philosophy. The model show significantly improved state-of-the-art results.
strength_weakness: strength_weakness: Because I am not a researcher on RGB-E tracking, and the authors have given me enough information in the main text and appendix, it is difficult for me to point out the shortcomings from a technical perspective. One small suggestion is that the authors gave some examples of qualitative analysis in the supplementary materials, but it is recommended to put some of the content in the main text, so that readers can understand the advantages and disadvantages of the method by combining quantitative numerical results and qualitative examples.
questions: questions: •	Please refer to weaknesses.
limitations: limitations: Yes
paper_summary: paper_summary: A novel RGB-E tracking framework with MOT philosophy has been proposed in order to keep track of both targets and distractors to robustly track a single object. It includes a Candidate Encoding Module, a Spatial-Temporal Transformer Encoder and a Dual-branch Transformer Decoder. Within these modules, the authors exploited the appearance information in combination with motion cues to improve the candidate matching and affinities. Experimental validation is comprehensive and results on three RGB-E object tracking benchmarks are state-of-the-art.
strength_weakness: strength_weakness: 1.The ablation study about N/M/T is missing and the selection of hyperparameters(e.g., τth and ζ) lacks discussion.

2.Does the delay caused by the Hungarian algorithm constitute the majority of the additional time compared with the pure appearance-based tracker? 

3.Is the training process carried out in a sequential iterative manner? That is to say, every frame will be output a prediction result or just the last frame in a batch. It is best to provide details in the main paper.

4.Have authors tried more layers in STTE and DBTD? Will they lead to continuous improvement in performance?

5.More thorough checks are needed for the paper writing and illustrations, including but not limited to the following parts:
a. The description about ΓEN in Line216 is not consistent with that in Figure 3
b. In Figure 2, “TIR” should be “Event”. Colors of Appearance and Motion feature embeddings are swapped.
c. In Figure 5,  A^{sa} should be  A^s. M^{t-1}should only have N blocks not M.
d. In Line336 and Line338, “encoder” should be “decoder”.
e. “COEST” should be “COESOT”. It has multiple mis-usages across whole paper.
f. In Figure8, “SGTE” should be “STTE”
questions: questions: See the Weakness and give the corresponding explaination.
limitations: limitations: NA
paper_summary: paper_summary: This work proposes a novel RGB-E tracking framework, CSAM. CSAM first predicts candidates and then tracks both targets and distractors using MOT philosophy. Comprehensive experiments are conducted on three RGB-E tracking benchmarks, showing that CSAM achieves state-of-the-art performance.
strength_weakness: strength_weakness: 1. Adding more experimental results under varying illumination conditions could be helpful to highlight the necessity of DVS.
2. The paper uses too many abbreviations, especially in CSAM, where each module has its own set of abbreviations, making it hard to remember them all.
questions: questions: 1. Where are the bottlenecks in the inference latency of the CSAM method, and how much time does each module consume? Additionally, the Candidate Embedding Module in CSAM uses NMS, which is likely one of the most time-consuming aspects. If so, can anchor-free methods address this?
2. Why does CSAM-B achieve a high FPS in Table 2 despite having a larger model size and higher FLOPs compared to SeqTrackv2-B256?
3. In MoT, the frequent appearance and disappearance of targets is a major challenge. The applicability of the CSAM method in this context is particularly interesting, as discussed on pages 130-134 of the paper. The authors could include more examples of tracking in situations where objects frequently disappear due to occlusion.
4. The total training time on two RTX 3090 GPUs?
5. Why are RSR and RPR metrics used in FE108, while SR and PR metrics are used in COESOT and VisEvent? What are the distinctions between them?
6. The DVS sensor introduces significant event noise during nighttime or in varying lighting conditions. It appears that the CSAM method implicitly filters out this noise using Transformer. However, the authors voxelized the DVS sensor input without applying denoising, could this potentially degrade performance? Besides, what unique approach does CSAM employ for handling DVS sensor data? CSAM appears to be applicable to other modality combinations.
limitations: limitations: Please see the weaknesses part.

Comments: title: Re
Re:
Re
Official Comment by Reviewer KN3J
comment: We are delighted that our rebuttal has addressed your concerns. We sincerely appreciate your recognition of our work.
I appreciate the authors' response to several of my questions. My rating remains the same.
Once again, we sincerely thank you for your valuable feedback.

Best regards
We greatly appreciate your recognition of our work and the increased score!
Thank the authors for their reply. I have updated my given score.
As I said in my initial comments, I am not particularly familiar with the details of this field. But I am very grateful for the author's response to me. I also read the authors' responses to other reviewers and thanked them for their efforts during the rebuttal. I want to maintain the original score first and refer to other reviewers' opinions on the author's rebuttal. If most reviewers approve of the author's efforts, I think I will consider giving a more positive score.