CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
Authors: Xiaokun Feng
Dailing Zhang
Shiyu Hu
Xuchen Li
Meiqi Wu
Jing Zhang
Xiaotang Chen
Kaiqi Huang
Venue: ICML 2025 poster (2025)
Keywords: RGB-X Tracking
Multimodal Tracking
Compact Spatiotemporal Features
Primary area: deep_learning->algorithms

Abstract: Effectively modeling and utilizing spatiotemporal features from RGB and other modalities (e.g., depth, thermal, and event data, denoted as X) is the core of RGB-X tracker design. 
Existing methods often employ two parallel branches to separately process the RGB and X input streams, requiring the model to simultaneously handle two dispersed feature spaces, which complicates both the model structure and computation process. 
More critically, intra-modality spatial modeling within each dispersed space incurs substantial computational overhead, limiting resources for inter-modality spatial modeling and temporal modeling.
To address this, we propose a novel tracker, CSTrack, which focuses on modeling Compact Spatiotemporal features to achieve simple yet effective tracking.
Specifically, we first introduce an innovative Spatial Compact Module that integrates the RGB-X dual input streams into a compact spatial feature, enabling thorough intra- and inter-modality spatial modeling. 
Additionally, we design an efficient Temporal Compact Module that compactly represents temporal features by constructing the refined target distribution heatmap. 
Extensive experiments validate the effectiveness of our compact spatiotemporal modeling method, with CSTrack achieving new SOTA results on mainstream RGB-X benchmarks.  The code and models will be released at: https://github.com/XiaokunFeng/CSTrack.

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
Xiaokun Feng" 12 Dailing Zhang 12 Shiyu Hu 3 Xuchen Li
Meiqi Wu* 4 Jing Zhang?
Xiaotang Chen? 2 Kaiqi Huang 12
Abstract
1. Introduction
Effectively modeling and utilizing spatiotempo- ral features from RGB and other modalities (e.g.. depth, thermal, and event data, denoted as X) is the core of RGB-X tracker design. Existing methods often employ two parallel branches to separately process the RGB and X input streams, requiring the model to simultaneously handle two dispersed feature spaces, which complicates both the model structure and computation pro- cess. More critically, intra-modality spatial mod- eling within each dispersed space incurs substan- tial computational overhead, limiting resources for motalito spatial modeling and temporal modeling. To address this, we propose a novel tracker, CSTrack, which focuses on modeling Compact Spatiotemporal features to achieve sim- ple yet effective tracking. Specifically, we first introduce an innovative Spatial Compact Mod- ule that integrates the RGB-X dual input streams into a compact spatial feature, enabling thorough intra- and inter-modality spatial modeling. Addi- tionally, we design an efficient Temporal Com- pact Module that compactly represents temporal features by constructing the refined target distri- bution heatmap. Extensive experiments validate the effectiveness of our compact spatiotempo- ral modeling method, with CSTrack achieving new SOTA results on mainstream RGB-X bench- marks. The code and models will be released at: http://github.co Www.aokkneng//STrreaa
As a fundamental visual task, object tracking (Yilmaz et al., 2006) aims to localize a target object within video se- quence based on its initial position. To achieve robust track- ing in complex and corner cases, such occlusions (Stadler & Beyerer, 2021). low visibility (Zhang et al., 2022a), and fast-moving objects (Tang et al., 2022), leveraging the com- plementary advantages of RGB and other modalities (e.g.. depth, thermal, and event data, collectively denoted as X) has emerged as a promising approach. Recently, several specialized benchmarks (Yan et al., 2021b; Li et al., 2021; Wang et al., 2023) have been introduced, along with a grow- ing body of outstanding RGB-X trackers (Zhu et al., 2023a; Hong et al., 2024; Hou et al., 2024; Wu et al., 2024; Chen et al., 2025; Hu et al., 2025).
Auxiliary modality enhances the potential for robust track- ing but requires the tracker handle RGB-X dual input streams simultaneously. Effective modeling and utilization of RGB-X spatiotemporal features are central to tracker de- sign (Zhang et al., 2024b). Existing trackers typically adopt two parallel branches to process the RGB and X modali- ties separately, each retaining its own feature space (Hou et al., 2024). Specifically, they often initialize the RGB branch using the backbone of pre-trained RGB tracker (Ye et al., 2022) and design dedicated architectures for the X branch. For example, ViPT (Zhu et al., 2023a) and One- Tracker (Hong et al., 2024) leverage the paradigm of prompt learning (Lester et al., 2021) to design the branch, where the X modality is processed as prompts for utilization. In contrast, SDSTrack (Hou et al., 2024) and BAT (Cao et al., 2024) employ symmetrical architectures, using the similar RGB-tracker backbone to process the X modality.
Eqqual contribution 'School of Artificial Intelligence, Univer- sity of Chinese Academy of Sciences, Beijing, China 'hhe Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy Sciences, Beijing, China 'School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore "School of Com- puter Science and Technology, University of Chinese Academy of Sciences, Beijing, China. Correspondence to: Kaiqi Huang <kaiqi huang @ @nIpria.ac.cn>>.
Although achieving certain effectiveness, these methods require handling two modality-specinn dispersed feature spaces simultaneously. This necessitates performing intra- modality spatial modeling between search images and tem- plate cues within each modality while also considering fea- ture interactions across modalities, which complicates both the model structure and computation process. More criti- cally, intra-modality modeling within each dispersed space incurs significant computational overhead, leaving insuffi- cient room for other spatiotemporal feature modeling, result-
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
ing in the following limitations. 1) For spatial modeling:
We design the innovative Spatial Compact Module and Temporal Compact Module, which leverage the compact spatiotemporal features for comprehensive intra- and inter-modality spatial modeling and dense temporal modeling.
There exists a severe imbalance between the intra- and inter- modality modeling. For instance, in BAT (Cao et al., 2024), the parameter proportions for intra- and inter-modality com- putations are 92.0% and 0.3%, respectively. Insufficient interaction across modalities limits the potential to fully exploit the complementary advantages of multimodal infor- mation. 2) For temporal modeling: Most existing RGB-X trackers (Hong et al., 2024; Hou et al., 2024; Wu et al., 2024) either lack temporal modeling rely solely on sparse temporal cues, such as dynamic templates (Wang et al., 2024a; Sun et al., 2024), which restricts their robustness when facing challenging scenarios.
Through extensive experiments, we demonstrate the effectiveness of our proposed spatiotemporal compact approach. Furthermore, CSTrack achieves new SOTA performance on mainstream RGB-D/T/E benchmarks.
2. Related Works
2.1. Spatial Modeling in RGB-X Tracking
Given the limitations of handling two dispersed spaces and the highly aligned and overlapping spatial semantic infor- mation between RGB and X images (Yang et al., 2022; Wu et al., 2024), a natural question arises: Is it necessary to always maintain these two dispersed spaces? The Atten- uation Theory (Treisman, 1964) suggests that, when faced with multiple sources of information, the human brain fol- lows a coarse-to-fin processing approach. It integrates the information into a small set of key elements before further processing (Zhao et al., 2024). Motivated by this, we pro- pose novel tracker, CSTrack, which emphasizes modeling and utilizing the Compact Spatiotemporal feature. By em- ploying a single-branch compact spatiotemporal feature, we can overcome the limitations of dual dispersed spaces, en- abling more effective multimodal spatiotemporal modeling.
To leverage the complementary advantages of RGB and X modalities, existing RGB-X trackers typically use two parallel branches for spatial modeling(Zhang et al., 2024b). These trackers include intra-modality modeling between the search images and template cues of each modality, as well as inter- modaliyyy modeling through feature interaction between modalities (Wang et al., 2024b). Specifically, the RGB branch usually adopts the backbone of pretrained RGB tracker (Ye et al., 2022), while the X branch follows different implementation strategies. Inspired by prompt learning (Lester et al., 2021), ViPT (Zhu et al., 2023a) and OneTracker (Hong et al., 2024) design a handcrafted X branch that treats the X modality as prompts, progressively embedding them into the RGB branch to facilitate inter- modality interaction. Additionally, SDSTrack (Hou et al., 2024) and BAT (Cao et al., 2024) adopt a symmetrical ar- chitecture, where the X branch is also derived from the pretrained RGB tracker and introduces adapter modules to enable knowledge transfer and inter-modality modeling. Managing the RGB-X dual dispersed feature spaces simul- taneously introduces certain limitations, prompting us to propose novel approach that integrates the RGB-X dual in- put streams into compact feature space, enabling effective intra- and inter-modality spatial modeling.
Specifically, CSTrack consists of two innovative mod- ules: the Spatial Compact Module (SCM) and the Tem- poral Compact Module (TCM). For the RGB-X dual input streams. SCM facilitates essential interaction between the two modalities to generate a compact spatial feature. Ad- ditionally, SCM introduces a small set of learnable queries to preserve modality-speciff information. This compact visual feature, along with the queries. is then fed into a one-stream backbone network (Ye et al., 2022; Zhang et al., 2022b) to enable comprehensive intra- and inter- modality modeling. Next, TCM constructs the refined target distri- bution heatmap in the search image, selecting key target cues to model compact temporal features at each time step. Dense temporal features are then obtained by aggregating over multiple steps. serving as a supplement to sparse dy- namic templates. Thanks this compact spatiotemporal feature representation, CSTrack achieves new SOTA results on RGB-D/T/E tasks.
2.2. Temporal Modeling in RGB-X Tracking
Due to the dynamic nature of the tracking target, the ini- tially provided static template often fails to offer continuous guidance, especially when the target undergoes significant appearance changes (Feng et al., 2024; Zhang et al., 2024a). However, most existing RGB-X trackers still overly rely on the initial template, with only a few recent works incor- porating temporal modeling to capture the target's evolv- ing appearance cues. Notably, STMT (Sun et al., 2024) and TATrack (Wang et al., 2024a) introduce dynamic mul- timodal templates. However, since these templates encode information from a single time step, such sparse temporal modeling struggles to maintain robust tracking in challeng- ing scenarios (Hu et al., 2024; 2023b). We argue that the
Our contributions are as follows:
To address the limitations of RGB-X dual dispersed fea- ture spaces, we propose a simple yet effective tracker, CSTrack, by modeling and utilizing novel compact multimodal spatiotemporal features.
2

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
Load
Spotial Compact Module
MP
Temporal Guidance Module
RGB
Embedding
FPN
Tracking Resuit
RGB-to-X Cross- Atoss-Attntionn
RGB Cross Cros-AAtention
Consistoncy toloubttion
stream Backbone
Templates
Search
K.
littgraee)
Comper
MAA
Share
A
D/T/E
K. y
peduwa
Embeddingg
Gaussian map generation
Patch
FRN
Save
Templates
Search. U.E.
Temporal Compoct Module
RGB input
input
RGB query
X
Xquery
Compoct spotiof
Compoct temporal
Load
Save
Concatenation
Initialinputtoken
Query token
Compact token
Interact with TemporoliMeory
Figure I: Framework of our proposed CSTrack. Given the RGB and X (e.g., thermal data) input streams at time (t 2 1), the shared Patch Embedding initially transforms them into token sequences. Then, the Spatial Compact Module integrates them into a compact feature space, which subsequently fed into a One-stream Backbone for comprehensive spatial modeling. Next, the Temporal Guidance Module uses the previously stored temporal features (up to t 1) for tracking guidance, after which the Head generates the final tracking results. Subsequently, the Temporal Compact Module constructs compact temporal features for the current time step, which are stored for tracking guidance at the next time step t + 1.
computational overhead of maintaining RGB-X dual dis- persed feature spaces is key factor limiting dense temporal modeling. For instance, in TATrack's ' online branch (Wang et al., 2024a), each additional time step requires spatiotem- poral modeling in both the RGB and X branches, leading to a multiplicative increase in computational costs. What sets us apart is that our compact multimodal features ex- ist in single branch, facilitating efficient dense temporal modeling. Specifically, we devise a parameter-free tempo- ral representation and lightweight spatiotemporal fusion module, substantially enhancing the tracker's robustness in complex scenarios.
3.1. Spatial Compact Modeling
3.1.1. SHARED PATCH EMBEDDING
time step t, the RGB-X dual streams fed into the model consist of the following images: the search im- ages St St 13x xxx,, the initial template images zP, 2 B3xHHxXW and the dynamic template images Z;, Zü E R3xHHxx., In line with the current ViT- tracking paradigm (Ye et al., 2022), we first partition each image into patches and then convert them into token se- quences (Dosovitskiy et al., 2020). We employ the patch em- bedding method from HiViT (Zhang al., 2022b), which, variant of the ViT model, better preserves spatial infor- mation through gradual downsampling, making it widely adopted in recent trackers (Shi et al., 2024; Xie et al., 2024).
3. Methods
The framework of CSTrack shown in Fig. 1. Given the RGB and X modality search images and template cues (ini- tial and dynamic template images) at time (t > 1), the shared Path Embedding first organizes them into token se- quences. The Spatial Compact Module (SCM) then gen- erates a compact spatial feature by essentially interacting the two modalities, incorporating a few queries to preserve ceddality-specifii information. This feature and the queries are fed into a One-stream Backbone for comprehensive intra- and mnter-modality spatial modeling. Next, the Temporal Guidonce Module uses the dense temporal features stored in temporal memory (up to é-1 1 ) to guide the tracking process, which is then passed to the Head to generate the tracking re- sults. Finally, the Temporal Compact Module (TCM) selects key tokens from the search features to compactly represent temporal features, achieved through the target distribution maps derived from the intermediate and final results.
Compared to previous works that design separate patch em- beddings for each modality (Zhu et al., 2023a; Hong et al., 2024), we adopt shared patch embedding module across modalities. The insight behind this is that both RGB and X images are essentially visual signals (Yang et al., 2022), and joint learning of RGB, depth, thermal, and event data can enhance the model's multimodal perception capabil- ity (Zhang et al., 2023). After processing, we obtain the search features s sf E N. the initial template fea- tures 20, = E NRN. and the dynamic template features of z N. We concatenate the visual information of each modality to obtain ft and ft N.. with Nu. 2N, + Ng:
f 1z0 2p s. f 120 2 st.
(I)
where [:] indicates concatenation along the first dimension.
3

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
3.1.2. SPATIAL COMPACT MODULE
further integrate the features of fe.
The ftt and ft represent two modality-specifie and dispersed feature spaces, each containing long token sequences. Exist- ing trackers typically adopt two parallel branches handle these dispersed feature spaces (Hong et al., 2024; Hou et al., 2024), leading to complex model structures and high com- putational costs. To address this limitation, our proposed SCM integrates f and fzz into a compact spatial feature, enabling simplified and effective spatial modeling.
Pt 4 Backhone (Jo).
(8)
Since Sa contains information from both modalities, and the Transformer network excels in token interaction (Vaswani et al., 2017), this process effectively enables comprehensive intra- and inter-modalin spatial modeling.
Specifically, we first introduce two small sets of learnable queries, gr, Oe E IDD, to preserve modality in- formation during the spatial compacting process. These queries are concatenated with the spatial features and partic- ipate in modality interaction via bidirectional cross-attention (Liu et al., 2025). Then, the features of each modality are further integrated through an feed-forward neural network.
Cons
Con
Conc
(A) Search Images &Terpphase
(b) R]
[c) n/
(c) A'
Figure 2: Illustration of different target heatmaps (using an RGB-T sample as an example). (a) Search images and tem- plates, with the green bounding box indicating the target to be tracked. (b-d) Target distribution heatmaps derived from intermediate results, final results, and their combination, i.e., h4, hyy and ht (reshaped into 2D images for visualization).
(oh f7] = Norm([g:; [3] Ccn f.l, (gni f2));;
1d f2] = Norm( f Ca f2]. (9/: f()))
[[") Norm ( FFN (ddi:
(4)
3.2. Temporal Compact Modeling
(d": 12 = (ormm((aa: (d)(()))"
(5)
3.2.1. TEMPORAL. COMPACT MODULE
Here, . represents the cross-attention operation where the first element serves Q and the second element serves to obtain K and V. Norm refers to the normaliza- tion operation, and FFN denotes the feed-forward fully connected network (Vaswani et al., 2017).
In addition to the sparse temporal information provided by the multimodal dynamic template (Sun et al., 2024; Wang et al., 2024a), TCM constructs the temporal feature m for the current time step and stores it in the temporal memory. By aggregating m² from multiple time steps, we obtain the dense temporal feature M up to time step t, which will guide the tracking process for the next time step + 1. The appearance information of the target in the search image reflects its dynamic changes (Ling et al., 2025), providing important guidance cues for the tracker (Zhou et al., 2024), and thus is regarded as the primary modeling object for m'. To this end, TCM designs a parameter-free method for constructing the target distribution heatmap, which precisely reflects the target's 's position distribution in the search image, Then, we extract Nm, key target tokens from the integrated spatial feature ft obtain the compact temporal feature m' E N XD
Although the RGB modality contains rich detailed informa- tion and the X modality focuses more on edge and contour information (Wu et al., 2024), they are well-aligned in spa- tial semantics (Yang et al., 2022). As result, the fr and fllt remain spatially aligned. We obtain the preliminary compact spatial feature foo by simply adding them together:
f
f + p!
(6)
Additionally, the query tokens involved in the entire feature interaction process implicitly preserve the characteristics of each modality. We incorporate them to construct the final compact spatial feature fe E N..
The target distribution heatmap consists of two parts, which are constructed based on the intermediate and final results, respectively. For the former, since f undergoes compre- hensive feature integration through the backbone, the rela- tionships between the search and template features are well- modeled. Thus, the consistency of the search tokens with respect to the template features can be computed to generate a target heatmap h based on the intermediate results. In this process, we first index the search feature s' E IR N, x and template feature a E 2 from fth based on the feature concatenation order in Eq. and Eq. Then, we per-
St = la"s d Sccl-
3.1.3. ONE-STREAM BACKBONE
After the above processing, the length of the visual token sequence handled by the model is reduced from 2. to 2.N, + Nes. where N₂ is much smaller than Nz.. meaning that the size of fo is comparable to that of a single modality. This allows us to use a one-stream Transformer- based back- bone (Ye et al., 2022: Shi et al., 2024; Xie et al., 2024) to
4

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
form correlation-like operations on s to refine the features (Feng et al., 2025):
tracking. Specifically, we first concatenate the individual temporal feature units in M to obtain M
= S (s')
(9)
Then, we employ a vanilla transformer- based decoder (Vaswani et al., 2017) to facilitate the interaction between the search feature and M embedding the temporal vari- ation cues of the target appearance into the search feature:
Next, a dot product operation between s/ and 2c is per- formed to obtain the target heatmap hy E RN,)
b = (6 (ct))) )mean = 1).
(10)
at = Norm( + +calse, s = Norm(s" + +FFN((
(13)
As shown in Fig. 2, hy finely depicts the target's position distribution, which is crucial for selecting target tokens. However, since hg is built through similarity matching with the target appearance in z it may also focus on objects similar to the target, potentially causing interference to the tracker. To address this, we introduce another heatmap, hyy, based on the final result. It is constructed using the predicted bounding box b' = (Iee Ve, w. h), obtained from the subsequent Head module (described in Sec. 3.3). Here, (de)Be), (w, h) represent the target's position and size in the search image, respectively. Based on the target-only position information contained in b', 1t E RW,xH, is constructed by converting it into a 2D Gaussian distribution, as expressed by:
(14)
3.3. Head and Loss
Based on the search feature s s which integrates the tem- plate and temporal cues, we employ a classic CNN-based prediction head (Ye et al., 2022; Xie et al., 2024) to obtain the final bounding box b'. To supervise the prediction of the bounding box, we employ the widely used focal loss Lats (Law & Deng, 2018), L1 loss, and the generalized IoU loss Liou (Rezatofighi et al., 2019). The overall loss function is formulated as follows:
Lail Lads + Niou + XL, In,
(15)
k))( x exp
-
1
Ze
2
y Me h/3
where Aion 2 and Ar, are the specific regularization parameters.
w/3
4. Experiments
where (Jees Me) considered the mean, and (ur, h) represents three times the standard deviation. By downsampling hy to match the 2D scale of s" and flattening it, we obtain ht E M. Compared to hf. as shown in Fig. 2, h2 mainly carries the target's position distribution information, but it is not finely detailed as hţ. To leverage the strengths of both, we compute the weighted sum of the two to obtain the final target distribution map h² E &RN. :N
4.1. Implementation Details
For the implementation of CSTrack, we adopt HiViT- (Zhang et al., 2022b) as the modality-sharce patch embed- ding and backbone. The CSTrack is initialized with Fast- iTPN (Tian al., 2024) pre-trained weights, and the token dimension D is set to 512. which are initialized with the Fast-iTPNN pre-trained parameters and the token dimension D set to 512. In the SCM, the length of modality-specifee queries Ng is set to 4. In the TCM, each temporal feature is represented by 16 tokens (i.e., Nm = 16), with the temporal length L set to by default. The sizes of template patches and search images are 128 x 128 and 256 256, respectively. Our tracker is trained on server equipped with four A5000 GPUs and tested on an RTX-3090 GPU. CSTrack consists of 75M parameters and achieves a tracking speed of 33 FPS.
h = 0.5 x Norm(h)) + 0.5 x Norm(h)).
(12)
The h reflects the likelihood of each token in s belonging to the target. Based on this, we select the top-Nm. tokens from s as the compact temporal feature m² for this time step.
Subsequently, m is stored in the temporal memory to form the multi-step dense temporal feature A1 = {m')) ) M buffer of length L, initialized with m² at the starting time, and updated by sliding window mechanism (Xie et al., 2024; Cai et al., 2024) to store the L most recent m² (See Sec. C.3 for details).
Inspired by recent unified models (Chen et al., 2022a; Yan et al., 2023), we adopt a joint training approach to develop a tracker capable of handling RGB- -D///E tasks simultane- ously. First, our training dataset includes common RGB-X datasets such as DepthTrack (Yan et al., 2021b), LasHeR (Li et al., 2021) and VisEvent (Wang et al., 2023). Furthermore, considering that the scale of these datasets is insufficient to support joint training. we also incorporate widely used RGB tracking datasets, namely LaSOT (Fan et al., 2019), GOT-1OK (Huang et al., 2019), COCO (Lin et al., 2014),
3.2.2. TEMPORAL GUIDANCE MODULE
This module is primarily designed to utilize the temporal feature M stored prior to the current time step to guide
5

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
Table I: Comparison on RGB-Depth datasets. The top two
Table 2: Comparison on
datasets.
results are highlighted in red and blue, respectively.
LasHeR
RGBT234
Method
DepthTrack
VOT-RGBD22
PR
MSR
MPR
Method
F-score
Rc
Rob
SGT (Li et al., 2017)
25.1
36.5
47.2
72.0
DAL (Qian ct al., 2021)
42.9
36.9
51.2
DAHNet (Gao et al., 2019)
54.4
79.6
LTMU-B (Dai al., 2020)
NICAS (Kriszan et 2020)
46.0
41.7
51.2
55.9
73.9
FANet (Zhu et al., 2020)
30,9
44.1
55.3
78.7
(ReilaecKristtnn et al., 2021)
59.2
77.5
76.0
MaCNet (Zhang et al., 2020)
55.4
79.0
KeepTrack (Mayer
2021)
60.5
75.3
79.7
HMFT (Zhang et al., 2022a)
31.3
43.6
DMTackk (Kristan et al., 2022)
65.8
75.8
85.1
CAT (Li al., 2020)
31.4
45.0
56.1
80.4
DeT (Yan et al., 2021b)
$2.9
54.3
56.0
65.7
76.0
84.5
DAPNet (Zhu et al., 2019)
31.4
43.1
SPT (Zhu
2025b)
54.9
52.7
65.1
79.8
85.1
ProTrack (Yang et sl., 2022)
57.8
57.3
58.3
65.1
80.1
82.0
JMMAC (Zhang et al., 202
57.3
79.0
VIPT (Zhu et al., 2023a)
59.6
59.4
59.2
72.1
81.8
86.7
CMPP (Wang et al.. 2020)
57.5
82.3
UnTrack (Wa
2024)
61.0
60.6
72.1
82.0
86.9
APFNet (Xiao et al., 2022)
36.2
50.0
57.9
82.7
One Tracker (Hong et 2024)
60.9
60.4
60.7
72.7
81.9
87.2
OSTrack (Ye ct al., 2022)
41.2
51.5
54.9
72.9
SDSTacc (Hoa et al., 2024)
61.9
60.1
61.4
72.8
81.2
88.3
ProTTrack (Yang et al., 2022)
42.0
53.8
59.9
79.5
CSTrack
66.4
77.4
ViPT (Zhu et al., 2023a)
52.5
65.1
61.7
83.5
TBSI (Hui al., 2023)
55.6
69.2
63.7
BAT (Cao al., 2024)
53.1
66.5
62.5
84.8
UnTrack (Wu et al., 2024)
51.3
64.6
62.5
84.2
SDSTrack (Hou al., 2024)
53.1
66.5
62.5
84.8
OneTraccer (Hong et al., 2024)
53.8
67.2
64.2
85.7
IPL (Lu et al., 2024)
55.3
69.4
65.7
88.3
CSTrack
60.8
75.6
70.9
94.0
4.2. Comparison with State-of-the-arts
4.2.1. QUANTITATIVE COMPARISON
Ground Truth
UNTrack
805Treck
Ours
We evaluate CSTrack on mainstream benchmarks and pro- vide a thorough comparison with SOTA models. Specifi- cally, Tab. 1 presents the evaluation of performance in the RGB-Depth task using Depth Track (Yan et al., 2021b) and VOT-RGBD2022 (Kristan et al., 2022): Tab. 2 evaluates the RGB-Thermal task using LasHeR (Li et al., 2021) and RGBT234 (Li et al., 2019); and Tab. illustrates the model's performance in the RGB-Event task, evaluated with VisEv- ent (Wang et al., 2023). For more information on these benchmarks, refer to Sec. A.
Figure 3: Qualitative comparison results of our tracker with other two trackers (i.e., UNTrack and SDSTrack) on three challenging cases. Better viewed in color with zoom-in.
TrackingNet (Muller et al., 2018), VastTrack (Peng et al., 2024), and TNL2k (Wang et al., 2021b), into our training set. For the RGB datasets, we input RGB images into both the RGB and X interfaces of the model to achieve consistency in input formats with RGB-X datasets. We further ana- lyze datasets from different modalities. Results show that the model is capable of effectively distinguishing between different modalities (see Sec. B for more details).
As shown in Tables 1, 2, and 3, CSTrack achieves new SOTA across all benchmarks, significantly outperforming previ- ous best models. Taking precision rate (PR) as an example, CSTrack surpasses SDSTrack (Hou et al., 2024) by 3.8% in DepthTrack, IPL (Lu et al., 2024) by 6.2% in LasHeR, and OneTraacer (Hong al., 2024) by 5.7% in VisEvent. These outstanding results demonstrate the effectiveness and gener- alization capability of our approach. Compared to existing trackers that separately handle the RGB and X modality feature spaces, CSTrack's distinct advantage lies in model- ing and utilizing a compact spatiotemporal feature. These results strongly validate the effectiveness of our approach.
Our training process consists of two stages. In the first stage, train the model for 150 epochs without incorporating TCM. Each epoch contains 10,000 samples, and each sam- ple consists of single search image. In the second stage, we introduce TCM based on the model trained in the first stage, which only involves spatial compactness modeling. During this stage, the backbone the model frozen, and we train the remaining parameters for 50 epochs. Each epoch includes 3,000 samples, where each sample comprises six search images. We employ the AdamW optimizer to opti- mize the network parameters. During the inference phase, the dynamic template update follows the strategy proposed in TATrack (Wang et al., 2024a). For benchmarks of differ- ent modalities, we adopt different thresholds for dynamic template updating (see Sec. B for more details).
Additionally, the recent PIL (Lu al., 2024) introduces the challenge of modality missing to address more realistic prac- tical scenarios. Specifically, it proposes the LasHeR Miss and RGBT234-Miss datasets for the RGB- -hB-mhermal task. To evaluate CSTrack we adopted simple method by cloning the available modality image onto the missing modality im- age. As shown in Tab. 4, although we did not design a dedi-
6

---

CSTrack: Enhancing CGB-X Tracking via Compact Spatiotemporal Features
Table 3: Comparison on the RGB-Event dataset.
benchmarks. For implementation details of each ablation setting, please refer to Sec. C.
VisEvent
Method
SR
PR
4.3.1. STUDY ON OUR SPATIAL COMPACT METHOD
SiamMask_ E (Wang et al., 2019)
36.9
56,2
SiamBAN_E (Chen et al., 2022b)
40.5
59.1
As described in Sec. 3.1, the primary motivation for our compact spatial modeling is that existing methods handling RGB-X dual feature spaces introduce complex model struc- tures and computation processes, limiting the effectiveness of multimodal spatial modeling. To validate this, con- duct ablation studies on various spatial modeling architec- tures. First, for the RGB-X dual-branch architecture, we implement two representative asymmetric (Zhu et al., 2023a; Hong et al., 2024) and symmetric (Hou et al., 2024; Cao et al., 2024) models (See See. C.I for details). Next, we remove the TCM from CSTrack, treating it as the baseline model for compact spatial modeling. To ensure fairness, we use the same backhone network and training strategy.
ATOM_E (Danelljan et al., 2019)
41.2
60.8
VITAL__ (Song et al., 2018)
41.5
64.9
SiamCar_E (Guo et al., 2020)
42.0
59.9
MDNet E (Nam & Han, 2016)
42.6
66.1
STARK_E E (Yan et al., 2021a)
44.6
61.2
PrDiMP E (Danelljan et al., 2020)
45.3
64.4
LTMU_E (Dai ct al., 2020)
45,9
65.5
TransT E (Chen et al., 2021)
47.4
65.0
SiamRCNN.E (Voigtlacnder ct al., 2020)
49.9
65.9
OSTrack (Ye et al., 2022)
53.4
69.5
ProTrack (Yang ct al., 2022)
47.1
63.2
ViPT (Zhu et al., 2023a)
59.2
75.8
UnTrack (Wu ct al., 2024)
58.9
75.5
Tracker (Hong et al., 2024)
60.8
76.7
SDSTrack (Hou al., 2024)
59.7
76.7
Tab. 5 shows the performance and computational efficiency of different frameworks. Compared to #1 and #2, the sym- metric dual-branch architecture achieves better performance, consistent with the findings in SDSTrack (Hou et al., 2024), but at the cost of higher parameters and computation. In con- trast, our method (#3) achieves superior performance while significantly reducing computational overhead, demonstrat- ing the effectiveness of our compact spatial modeling.
CSTrack
65.2
82.4
Table 4: Comparison on RGB-Thermal modality-misssing
datasets.
LasHeRR-Miss SR
RGBT234-Miss
Method
PR
MSR
MPR
CAT eLi 2020)
20.6
28.5
35.6
$2.1
APENet (Xiao ct al., 2022)
29.0
38.2
44.7
65.3
ViPT (Zhu et al., 2023a)
36.9
44.0
44.8
61.2
Additionally, we perform the ablation studies on different core components of SCM. As shown in Tab. 6, #1 represents the baseline model for our compact spatial modeling. #2 indicates the performance after removing modality-specifft queries (i.e., qr. qa), where performance degradation is observed across all three benchmarks. We attribute this to these queries implicitly capturing key information from each modality, enabling the effective construction of compact spa- tial features. #3 indicates that we replace the inter-modality cross-attention (in Eq. 2 and Eq. 3) with self-attention within each modality. #4 represents the setting where we do not use the shared patch embedding but design separate embed- dings for RGB and X. Compared to RGBT234 and VisEvent, DepthTrack is more challenging, evidenced by its lower absolute precision. We hypothesize that modality interaction before compression and learning a unified representation through shared embeddings contribute to better performance in difficult tracking scenarios.
TBSI (Hui et al., 2023)
47.0
59.0
44,8
61.2
BAT (Cao et al., 2024)
44.0
54.5
52.8
73.4
SDSTrack (Hou al., 2024)
44.4
54.8
52.5
72.4
IPL (Lu al., 2024)
49.4
61.7
59.4
82.0
CSTrack
50.3
62.2
68.0
89.8
cated module for modality missing as PIL did, our method still demonstrates significant performance advantage. For instance, on the RGBT234-Miss, CSTrack surpasses IPL by 8.6% in mean success rate (MSR), showcasing the robust- ness of CSTrack to missing input information (Chen et al., 2024; Feng et al., 2023).
4.2.2. QUALITATIVE COMPARISON
As shown in Fig. 3, we present the tracking results of CSTrack and two existing SOTA trackers (i.e., SDSTrack (Hou et al., 2024), UnTrack (Wu et al., 2024)) on the RGB- D/T/E challenging sequences. More cases can be found in Sec. D. Clearly, CSTrack demonstrates superior robustness and effectiveness.
Finally, Fig. 4 provides a perspective for visual analysis. We analyze two representative cases. RGB(X) advantage indicates that the tracker primarily relies on RGB(X) modal- ity, as target appearance information in the other modality, X(GB), is significantly degraded due to environmental interference (e.g., overlapping similar objects or lighting disturbance). Correspondingly. the model achieves accu- rate tracking with only RGB(X) input but fails with only X(RGB) input. As shown in Fig. 4 (d), the model delivers
4.3. A Ablation Study
To investigate the properties of the compact spatiotemporal modeling approach in CSTrack, we conduct comprehen-
sive ablation studies on the DepthTrack (Yan et al., 2021b),
RGBT234 (Li et al., 2019) and VisEvent (Wang al., 2023)
7

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
Table 5: Comparison of Different RGB-X Spatial Modeling Frameworks.
RGBT234
DepthTrack PR
VisEvent
# Frameworks
Params
FLOPs
Speed
MSR
MPR
Re
SR
PR
F-score
Dual-Branch (Asymmetrical)
65.9
92.0
62.7
63.2
62.2
62.8
79.1
82M
38G
24 FPS
2
Dual-Branch (Symmetrical)
68.2
92.4
63.6
64.1
63.1
62.9
79.6
136M
59G
18 FPS
3
One Compact Branch (ours)
69.6
93.0
65.1
65.3
64.9
64.1
81.0
73M
36G
35 FPS
Table 6: Ablation Study on Spatial Compact Module.
Table 8: Ablation Study on Training Strategies.
RGBT234
Depth Rc
VisEvent
RGBT234
Depth TrachTrack Re
VisEvent
Setting
MSR
MPR
SR
PR
Setting
MSR
MPR
F-scure
PR
Baseline
69.5
93.0
65.1
65.3
64.9
64.1
81.0
Buseline
69.6
93.0
65.1
65.3
64.9
64.1
81.0
queries
67.3
92.0
64.5
63.7
80.3
wo datasets
85.9
60.9
61.2
60.6
79.3
woo cross attention
69.5
92.8
60.6
61.0
60.2
64.1
81.0
3
wob joint train
67.6
90.9
64.2
64.7
63.8
63.0
80.3
unshare embedding
69.2
93.0
61.1
61.3
60.9
64.1
80.8
Table 7: Ablation Study on Temporal Compact Module.
imagen &Tamplates
(b) RGB-only
X-nny
RGBBT234
DepthTrack
VisEvent
Setting
MSR
MPR
F-score
Re
PR
SR
PR
Baseline
69.6
93.0
65,1
65.3
64.9
64.1
81.0
Rol-based
69.0
92.8
64.4
64.9
63.9
64.3
81.1
3
Query-based
69.4
92.2
63.3
63.7
62.8
64.1
80.9
A
70.1
93.6
64.8
65.0
64.6
64.9
82.1
5
70,2
93.8
64.2
64.4
64.0
64.8
82.0
wh (8) & A5)
70,9
94.0
65.8
66.4
65.2
65.2
82.4
Figure 4: Tracking results of the model under different input settings in two categories of cases (using the RGB-T task example). (a) Search images and templates. (b-d) Tracking results with only RGB input, only X input, and both inputs. The heatmap regions are cropped by the tracker. The green and red bounding boxes represent the target to be tracked and the tracking result. Better viewed with zoom-in.
precise tracking results when RGB+X inputs available. This indirectly demonstrates that, even though spatial infor- mation from both modalities is compacted together, critical information from each modality is retained, validating the effectiveness of our spatial compact modeling.
4.3.2. STUDY ON OUR TEMPORAL COMPACT METHOD
Our TCM focuses on selecting a small set of key target to- kens from the search features as compact temporal features. To evaluate this, we conduct the ablation studies on different temporal compacting methods, as shown in Tab. 7. Inspired by existing works, we first explore two widely used methods: #2 involves using predicted bounding boxes for Rol process- ing in the search features (Wang et al., 2021a: la; Zhou et al., 2023), and #3 applies temporal query compression (Zheng al., 2024; Xie et al., 2024). For detailed implementation, please refer to Sec. C.2. Compared to the baseline (#1), both methods lead to performance degradation. We hypothesize that this is due to the introduction of new feature variations for the search tokens, which places the constructed temporal features in different feature space, thus failing to provide effective guidance for the tracker in the subsequent tempo- ral guidance module. In contrast, our temporal features are selected directly from the search tokens without any feature variation. #4 and #5 represent the selection of search to- kens using target heatmaps h and h², both of which lead to performance improvements. As observed in Fig. 2, by combining the two heatmaps, we achieve a more optimal target distribution representation, which further enhances
model performance (#6).
As shown in Fig. 5, we analyze the impact of the temporal memory length L on model performance. We observe that as the increases, the performance initially improves and then stabilizes. To balance performance and computational cost, we select 4 as the optimal temporal length.
4.3.3. OTHER ABLATION ANALYSES
To develop a tracker that simultancously handles RGB- D/T/E tasks, we adopt a strategy of joint training on both RGB and X datasets, which significantly differs from exist- ing approaches. As shown in Tab. 8, we conduct the ablation studies on this strategy. #2 indicates joint training using only the X datasets, resulting in a noticeable performance drop. The precision decreases by 7.1%, 4.3%. and 1.7% across the three benchmarks. highlighting the importance of large- scale RGB tracking datasets in joint training. #3 represents a non-joint training strategy, where a separate tracker is trained for each modality. Although the number of task types is reduced, the loss of cross-modee modal knowledge sharing leads to a decrease in performance.
8

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
Impact Statement
Madel Performence Voviation with Diferent Temporal
R0811254
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
VaDvent
Precison
2.0
0.
0.30
References
0.2
06M
Cai, W., Liu, Q., and Wang, Y. Hiptrack: Visual tracking with historical prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19258-19267, 2024.
Temporal Length
Figure 5: Model performance variation (A precision) with different temporal lengths.
Cao. B., Guo, J.. Zhu, P., and Hu, Q. Bi-direetional adapter for multimodal tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 927 935, 2024.
5. Conclusions
To address the limitations of existing methods handling RGB-X dual dispersed feature spaces, we propose novel tracker named CSTrack, focusing on compact spatiotempo- ral feature modeling. For spatial modeling, we introduce the innovative Spatial Compact Module, which integrates the RGB-X dual input streams into a compact spatial feature, enabling comprehensive intra- and modality spatial modeling. For temporal modeling, we design the efficient Temporal Compact Module to compactly represent temporal features, laying solid foundation for dense temporal mod- eling. Through these combined efforts, CSTrack achieves outstanding performance, significantly outperforming ex- isting methods on mainstream RGB-D/T/E benchmarks. Extensive experiments underscore the effecctiveness of the compact spatiotemporal modeling method and provide a unique perspective for future RGB-X tracker design.
Chen, H., Zhang, Y., Feng, X., Chu, X., and Huang, K. Revealing the dark secrets of extremely large kernel con- vnets on robustness. arXiv preprint arXiv:2407.08972 2024.
Chen, T., Saxena, S., Li, Lin, T.-Y., Fleet, D. J., and Hin- ton, G. E. A unified sequence interface for vision tasks. Advances in Neural Information Processing Systems, 35: 31333-31346, 2022a.
Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., and Lu, H. Transformer tracking, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3126-8135, 2021.
Chen, X., Kang, B., Geng, W., Zhu, J., Liu, Y., Wang, D., and Lu, H. Sutrack: Towards simple and unified single object tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 2239-2247, 2025.
Limitations and future work. Our CSTrack effectively improves RGB-X tracking performance through a com- pact spatio-temporal modeling mechanism. Although this work covers auxiliary modalities the form of visual data, namely depth, thermal, and event modalities, there is cur- rently lack of research on visual-language tracking (i.e., RGB-L) that leverages text-based auxiliary modalities (Hu et al., 2023a; Li et al., 2024a;b;d;c). In the future, we plan to further investigate how to efficiently utilize textual modal- ity cues to develop a tracker capable of accommodating all existing auxiliary modalities.
Chen, Z., Zhong, B., Li, G., Zhang, S., Ji, R., Tang, Z., and Li, X. Siamban: Target-aware tracking with siamese box adaptive network. TPAMI, 45(4):5158-5177) 2022b.
Dai, K., Zhang, Y., Wang. D., Li, J., Lu, H., and Yang, X. long-term tracking with meta-updater. In CVPR, 2020.
Danelljan, M., Bhat, G., Khan, F. S., and Felsherg, M. Atom: Accurate tracking by overlap maximization. In CVPR, 2019.
Danelljan, M., Gool, L. V., and Timofte, R. Probabilistic regression for visual tracking. In CVPR, 2020.
Acknowledgements
This work is supported in part by the National Science and Technology Major Project (Grant 20222001 16403), and the National Natural Science Foundation of China (Grant No. No.621766 62775255).
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn,
Hu, S., Zhao, X., Huang, L., and Huang, K. Global instance tracking: Locating target more like humans. IEFE Trans- actions Pattern Analysis and Machine Intelligence, 45 (1):576-59)) 2023b.
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
Hu, S., Zhao, X., and Huang, K. Sotverse: A user-defined task space of single object tracking. International Journal of Computer Vision, 132:872-955 930, 2024.
Fan, H., Lin, L., Yang, F., Chu, P.. Deng, G., Yu, S., Bai, H., Xu, Y.. Liao, C., and Ling, H. Lasot: A high-quality benchmark for large-scale single object tracking. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 5369-5378, 2019.
Hu, X., Tai, Y., Zhao, X., Zhao, C., Zhang, Z., Li, J., Zhong, B., and Yang, J. Exploiting multimodal temporll patterns for video object tracking. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 3581 1851-356 -3589, 2025.
Feng, X., Hu, S., Chen, X., and Huang. K. A hierarchical theme recognition model for sandplay therapy. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp. 241 252.. Springer, 2023.
Huang, L., Zhao, X., and Huang, K. Got-1Ok: A large high- diversity benchmark for generic object tracking in the wild. IEEE transactions on pattern analysis and machine intelligence, 43(5):1622-1777 2019.
Feng, X., Li, X., Hu, S., Zhang, D., Wu, M., Zhang, J., Chen, X., and Huang, K. Memvlt: Vision-language tracking with adaptive memory-based prompts. In The eiihty- Annual Conference on Neural Information Processing Systems, 2024.
Hui, T., Xun, Z., Peng, F., Huang, J., Wei, X., Wei, X., Dai, J., Han, J., and Liu, S. Bridging search region interaction with template for rgb-t tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13630-13639, 2023.
Feng, X., Zhang, D., Hu, S., Li, Wu, M., Zhang, J., Chen, X., and Huang, K. Enhancing vision-language tracking by effectively converting textual cues into visual cues. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 1-5. IEEE, 2025.
Kristan, M., Leonardis, A., Matas, J., Felsberg, M.. Pflugfelder, R., Kämäräinen, J.-K., Danelljan, M., Zaje, L. C., Lukežič, A., Drbohlav, O., et The eighth visual object tracking vot2020 challenge results. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23- 28, 2020, Proceedings, Part V 16, pp. 547-601. Springer, 2020.
Gao, Y., Li, C., Zhu, Y., Tang. J., He, T., and Wang. F. Deep adaptive fusion network for high performance RGBT tracking. In ICCVW, 2019.
Kristan, M., Matas, J., Leonardis, A., Felsberg, M., Pflugfelder, R., Kämäräinen, J.-K., Chang, H. J., Danell- jan, M., Cehovin, L., Lukežië, A., et al. The ninth visual object tracking vot2021 challenge results. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2012-71122 1738,, 2021.
Guo, D., Wang, J., Cui, Y., Wang, Z., and Chen, S. Siamear: Siamese fully convolutional classification and regression for visual tracking. In CVPR, 2020.
Hong, L., Yan, S., Zhang, R., Li, W., Zhou, X., Guo, P., Jiang, K., Chen, Y., Li, J., Chen, Z. et al. Onetracker: Unifying visual object tracking with foundation models and efficient tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 19079-19091, 2024.
Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Kämäräänen, J.-K., Chang, H. J., Danell- jan, M., Zaje, L. Č., Lukežič, A., et al. The tenth visual object tracking vot2022 challenge results. In European Conference on Computer Vision, pp. 431-460. Springer, 2022.
Hou, X., Xing. J., Qian, Y., Guo, Y., Xin, S., Chen, J., Tang, K., Wang, M., Jiang, Z., Liu, L., et al. Sdstrack: Self- distillation symmetric adapter learning for multi-modal visual object tracking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26551-26561, 2024.
Law, H. and Deng, J. Cornernet: Detecting objects as paired keypoints. In Proceedings of the European conference on computer vision (ECCV), pp. 734-750, 2018.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. arXiv preprint 2rXiv:2104.0880 2021.
Hu, S., Zhang, D., Wu, M., Feng, X., Li, X., Zhao, X., and Huang, K. A multi-modal global instance tracking bench- mark (mgit): Better locating target in complex spatio- temporal and causal relationship. In the 37th Conference on Neural Information Processing Systems, volume 36, pp. 25007-25030 2023a.
Li, C., Zhao, N., Lu, Y., Zhu, C., and Tang, J. Weighted sparse representation regularized graph learning for RGB- T object tracking. In ACMMM, 2017.
10

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
Li, C., Liang, X., Lu, Y., Zhao, N., and Tang, Rgb-t object tracking: Benchmark and baseline. Pattern Recognition, 96: 2019.
of what not track. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 13444- 13454, 2021.
Li, C., Liu, L.,, Lu, A., Ji, Q. and Tang, Challenge-aware RGBT tracking. 2020.
Muller, M., Bibi, A., Giancola, S., Alsubaihi, S., and
Ghanem, B. Trackingnet: A large-scale dataset and bench- mark for object tracking in the wild. In Proceedings of the European conference on computer vision (ECCV), pp. 300-317, 2018.
Li, C., Xue, W., Jia, Y., Qu, Z., Luo, B., Tang, J., and Sun,
D. Lasher: A large-scale high-diversity benchmark for rght tracking. IEEE Transactions on Image Processing, 31:392 404, 2021.
Nam, H. and Han, B. Learning multi-domain convolutional neural networks for visual tracking. 2016.
Li, X., Feng, X., Hu, S., Wu, M., Zhang, D., Zhang, J., and Huang, K. Dit Diverse text generation for visual language tracking based on Ilm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7283-7292, 2024a.
Peng, L.,, Gao, J., Liu, X., Li, W., Dong, S., Zhang, Z., Fan, H., and Zhang, L. Vasttrack: Vast category visual object tracking. arXiv preprint arXiv:2403.0344)) 2024.
Qian, Y., Yan, S., Lukežič, A., Kristan, M., Kämäräinen, J.-K., and Matas, J. Dal: A deep depth- aware long-term tracker. In 2020 25th International conference on pattern recognition (ICPR), pp. 7825-7832. IEEE, 2021.
Li, X., Hu, S., Feng, X., Zhang, D., Wu, M., Zhang, J., and Huang. K. Dtvlt: A mullti-mddl diverse text benchmark for visual language tracking based on Ilm. arXiv preprint 21X.0:2400.02492, 2024b.
Ren, S., He, K., Girshick, R., and Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Li, X., Hu, S., Feng, X., Zhang, D., Wu, M., Zhang, J., and Huang, K. How texts help? a fine-grat greingr evaluation to reveal the role of language in vision-lange language tracking. arXiv preprint arXiv:2411.
2024c.
Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., and Savarese, S. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 658-666, 2019.
Li, X., Hu, S., Feng, X., Zhang. D., Wu, M., Zhang, J., and Huang, K. Visual language tracking with multi- modal interaction: A robust benchmark. arXiv preprint arXiv:2409. 08887, 2024d.
Shi, L., Zhong, B., Liang, Q., Li, N., Zhang, S.,, and Li, X. Explicit visual prompts for visual object tracking. arXiv preprint arXiv:2401.0311
Lin, T.-Y., Maire, M., Belongie. S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. I.. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
2024.
Song, Y., Ma, C., Wu, X., Gong, L., Bao, L., Zuo, W., Shen, C., Lau, R. W., and Yang, M.-H. Vital: Visual tracking via adversarial learning. In CVPR, 2018.
Ling, X., Zhu, C., Wu, M., Li, H., Feng. X., Yang. C., Hao, A., Zhu, J., Wu, J., and Chu, X. Vmbench: A benchmark for ierreptioonaalinnen video motion generation. arXiv preprint ar 250.: 10076, 2025.
Stadler, D. and Beyerer, J. Improving multiple pedestrian tracking by track management and occlusion handling. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10958-10967, 2021.
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang. H., Yang, J., Jiang, Q. Li, C., Yang, J., Su, H., et al. Grounding dino: Marry- ing dino with grounded pre- pre-training for open-set object detection. In European Conference on Computer Vision, pp. 38-55. Springer, 2025.
Sun, D.,, Pan, Y., Lu, A., Li, C., and Luo, B. Transformer rgbt tracking with spatio-temporal multimodal tokens. arXiv preprint 2401:2401.016744 2024.
Tang, C., Wang. X., Huang, J.. Jiang, B., Zhu, L., Zhang, J., Wang, Y., and Tian, Y. Revisiting color-event based tracking: A unified network, dataset, and metric. arXiv preprint arXiv:2211.11010 2022.
Lu, A., Li, C., Zhao, J., Tang, J., and Luo, B. Modality- missing rgbt tracking: Invertible prompt learning and high-quality benchmarks. International Journal of Com- puter Vision, pp. 2024.
Tian, Y., Xie, L., Qiu, J., Jiao, J., Wang, Y., Tian, Q. and Ye, Q. Fast-itpn: Integrally pre-trained transformer pyramid network with token migration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Mayer, C., Danelljan, M., Paudel, D. P., and Van Gool, L. Learning target candidate association to keep track

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
Treisman, A. M. Selective attention in man. British medical bulletin, 1964.
Xie, J., Zhong, B., Mo. Z., Zhang, S., Shi, L.., Song, S., and Ji, R. Autoregressive queries for adaptive track- ing with spatie-lemporlltaasfoomess arXiv preprint 2.d1:22003.157444 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L.,, and Polosukhin, I. Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, 30, 2017.
Yan, B., Peng, H., Fu, J., Wang, D., and Lu, H. Learning spatio-temporal transformer for visual tracking. In ICCV, 2021a.
Voigtlaender, P., Luiten, J., Torr, P. H. S., and Leibe, B. Siam R-CNN: Visual tracking by re-detection. 2020.
Yan, B., Jiang, Y., Wu, J., Wang, D., Luo, P., Yuan, Z.. and Lu, H. Universal instance perception object discovery and retrieval. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 15325-15336, 2023.
Wang, C., Xu, C., Cui, Z., Zhou, L., Zhang, T., Zhang, X., and Yang, J. Cross-modal pattern propagation for rgb-t tracking. In CVPR, 2020.
Wang, H., Liu, X., Li, Y., Sun, M., Yuan, D., and Liu, Temporal adaptive rgbt tracking with modality prompt. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 38, pp. 5436-5444 2024a.
Yan, S., Yang. J., Käpylä, J., Zheng, F., Leonardis, A., and Kämäräänen, J.-K. Depthtrack: Unveiling the power of rgbd tracking. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 10725-10733, 2021b.
Wang, N., Zhou, W., Wang, J. and Li, H. Transformer meets tracker: Exploiting temporal context for robust visual tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1571-1580, 2021a.
Yang, J., Li, Z., Zheng, F., Leonardis, A., and Song, J. Prompting for multi-modal tracking. In Proceedings of the 30th ACM international conference on multimedia, pp. 3492-3500, 2022.
Ye, B., Chang, H., Ma, B., Shan, S., and Chen, X. Joint feature learning and relation modeling for tracking: A one-stream framework. In Proceedings of the European Conference on Computer Vision, pp. 341-357, 2022.
Wang, Q., Zhang, L., Bertinetto, L., Hu, W., and Torr, P. H. S. Fast online object tracking and segmentation: A unifying approach. 2019.
Wang, X., Shu, X., Zhang, Z., Jiang, B., Wang, Y., Tian, Y., and Wu, F. Towards more flexible and accurate object tracking with natural language: Algorithms and bench- mark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13763- 13773, 2021b.
Yilmaz, A., Javed, O., and Shah, M. Object tracking: A survey. Acm computing surveys (CSUR), 38(4):13 2006.
Zhang, D., Hu, S., Feng, X., Li, X., Zhang, J., Huang, K., et al. Beyond accuracy: Tracking more like human via visual search. Advances in Neural Information Processing 2024a.
Wang, X., Li, J., Zhu, L., Zhang. Z. Chen, Z., Li, X., Wang, Y., Tian, Y., and Wu, F. Visevent: Reliable object tracking via collaboration of frame and event flows. IEEE Transactions on Cybernetics, 2023.
Systems,
Zhang, H., Zhang, L., Zhuo, L., and Zhang, J. Object tracking in RGB-T T videos using modal-aware attention network and competitive learning. Sensors, pp. 393, 2020.
Wang, X., Sun, Z., Chehri, A., Jeon, G., and Song, Y. Deep learning and multi-modal fusion for real-time multi- object tracking: Algorithms,, challenges, datasets, and comparative study. Information Fusion, 105:10 2024b.
Zhang, P., Zhao, J., Bo, C., Wang, D., Lu, H., and Yang, X. Jointly modeling motion and appearance cues for robust RGB-T tracking. TIP, pp. 3335-3341, 2021.
Zhang, P., Zhao, J., Wang, D., Lu, H., and Ruan, X. Visible- thermal uav tracking: A large-scale benchmark and new baseline, In CVPR, 2022a.
Wu, Zz,, Zheng, J., Ren, X., Vasluianu, F.A.., Ma, C., Paudel, D. P., Van Gool, L., and Timofte, R. Single-model and any- mod-mttl for video object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19156-19166, 2024.
Zhang, P., Wang, D., and Lu, H. Multi-modal visual track- ing: Review and experimental comparison. Computa- tional Visual Media, 10(2):193-214, 2024b.
Xiao, Y., Yang, M., Li, C., Liu, L., and Tang, J. Attribute- based progressive fusion network for rght tracking. In AAAI volume 36, 2022.
Zhang, X., Tian, Y., Xie, L., Huang, W., Dai, Q., Ye, Q. and Tian, Q. Hivit: A simpler and more efficient de- sign of hierarchical vision transformer. In The Eleventh
12

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
International Conference Learning Representations, 2022b.
Zhang, Y., Gong, K., Zhang, K. Li, H., Qiao, Y., Ouyang, W., and Yue, X. tratfrransformer A uni- fied framework for multimodal learning, arXiv preprint arXiv:2307. 10802 2023.
Zhao, T. Yuan, M., Jiang, F., Wang, N., and Wei, X. Removal and selection: Improving rgb-infrared ob- ject detection via coarse-to-fine fusion. arXiv preprint arXiv:2401. 107331 2024.
Zheng, Y., Zhong, B., Liang, Q., Mo, Z., Zhang, S., and Li, X. Odtrack: Online dense temporal token learning for visual tracking. arXiv preprint arXiv:2401.01633 2024.
Zhou, L., Zhou, Z., Mao, K., and He, Z. Joint visual ground- ing and tracking with natural language specification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23151-23166 2023.
Zhou, X.. Guo, P., Hong, L., Li, J., Zhang, W., Ge, W., and Zhang, W. Reading relevant feature from global repre- sentation memory for visual object tracking. Advances in Neural Information Processing Systems, 36, 2024.
Zhu, J., Lai, S., Chen, X., Wang, D., and Lu, H. Visual prompt multi-modal tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9516 9526, 2023a.
Zhu, X.-F. Xu, T., Tang, Z., Wu, Z., Liu, H., Yang. X., Wu, X.-J., and Kittler, J. Rgbdlk: A large-scale dataset and benchmark for rgb-d object tracking. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 3870-3878, 2023b.
Zhu, Y., Li, C., Luo, B., Tang, J., and Wang, X. Dense feature aggregation and pruning for RGBT tracking. In ACMMM, 2019.
7hu, Y., Li, C., Tang, J., and Luo, B. Quality-aware feature aggregation network for robust RGBT tracking. TIV, 121-130, 2020.
13

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
A. More Details on the RGB-X Benchmarks
As discussed in Sec. 1.2.1, we perform a comprehensive evaluation of CSTrack on mainstream RGB-X benchmarks. Specifically, the model's performance on the RGB-Depth task is evaluated using DepthTrack (Yan et al., 2021b) and VOT-RGBD2022 (Kristan al., 2022). For the RGB-Thermal task, we adopt LasHeR (Li al., 2021) and RGBT234 (Li al., 2019), while the RGB-Event task is assessed based VisEvent (Wang et al., 2023). In this section, we provide an overview of these benchmarks and their respective evaluation metrics.
DepthTrackk DepthTrack (Yan et al., 202 is a comprchensive and long-term RGB-D tracking benchmark. It consists of 150 training sequences and 50 testing sequences, with 15 per-frame attributes. The evaluation metrics include precision rate (PR), recall (Re), and F-score.
VOT-RGBD2022. VOT-RGBD2022 (Kristan et al., 2022) is the latest benchmark in RGB-D tracking, consisting of 127 short-term RGB-D sequences designed explore the role of depth in RGB-D tracking. This dataset employs an anchor-based short-term evaluation protocol (Kristan et al., 2020), which requires trackers restart multiple times from different initialization points. The main performance metrics include Accuracy (Acc), Robustness (Rob), and Expected Average Overlap (EAO).
LasHeR. LasHeR (Li et al., 2021) is a large-scale, high-diversity benchmark for short-term RGB-T tracking, comprising 979 video pairs for training and 245 pairs for testing. The dataset evaluates tracker performance using precision rate (PR) and success rate (SR).
RGBT234, RGBT234 (Li al., 2019) is large-scale RGB-T tracking benchmark consisting of 234 pairs of visible and thermal infrared video sequences. The dataset employs mean success rate (MSR) and mean precision rate (MPR) for performance evaluation.
VisEvent. VisEvent (Wang et al., 2023) is the largest dataset for RGB-E tracking, consisting of 500 pairs of training videos and 320 pairs of testing videos. This dataset employs success rate (SR) and precision rate (PR) for performance evaluation.
B. More Details on Model Implementation
As discussed in Sec. 4.1, perform joint training using datasets from both RGB-ont only and RGB-X X modalities. Here, we conduct statistical analyses on representative RGB and RGB-D/T/E datasets to obtain the mean values used for input normalization. These values are computed over the training videos of each dataset. For comparison, we also include the ImageNet dataset (Deng et al., 2009) as a reference representing natural image distributions. As shown in Tab. 9, for the RGB modality, LaSOT and DepthTrack both collected in typical natural environments exhibit mean values that are similar those of IeageNet. In contrast, Lasher, which often focuses on dark scenes, and VisEvent, which predominantly contains yellow-tinted RGB images, show noticeable deviations from the natural image statistics. For the X modality, as illustrated in Fig. 3 of the paper, the image styles vary significantly across datasets, leading to substantial differences in the corresponding mean values.
Table 9: Statistical analysis of RGB and X modalities across different datasets.
Dataset
RGB Mean
X Mean
ImageNet
0.485, 0.456, 0.406
LaSOT (RGB-only)
0,456, 0.459, 0.426
DepthTrack (RGB-D)
0.417, 0.414, 0.393
0.574, 0.456, 0.240
Lasher (RGB-T)
0.500, 0.499, 0.471
0.372, 0.372, 0.368
VisEvent (RGB-E)
0.418, 0.375, 0.317
0.935, 0.904, 0.949
In addition, we further analyze the model's discriminative ability across modality datasets. Specifically, based on the features from the one-stream backbone (see Eq. in the paper), we extract the tokens associated with learnable queries and construct a flur-class classifier using two fully connected layers to identify the input modality type: RGB-only, RGB-D, RGB-T, and RGB-E. The existing model is frozen, and the classification head is trained for 2 epochs. The accuracy results shown
14

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
in Tab. 10 demonstrate that the model can effectively distinguish between different modality datasets, owing the clear differences in their feature distributions.
Table 10: Classification accuracy (%) of modality type prediction across different datasets.
Dataset
LaSOT (RGB-only)
DepthTrack Track (RGB-D)
Lasher (RGB-T)
VisEvent (RGB-E)
Accuracy
100%
98%
100%
100%
During the inference phase, our dynamic template update follows the strategy proposed in TATrack (Wang et al., 2024a). For benchmarks of different modalities, we adopt modality-specifee thresholds for dynamic template updating. Specifically, for RGB-T benchmarks, namely Lasher and RGBT-234, the update threshold 0.45; for other datasets, the threshold set to 0.7.
C. Experimental Details of Ablation Studies
C.1. Implementation of Different RGB-X Spatial Modeling Frameworks
As shown in Tab. of Sec. 4.3.1, we com- pare our proposed spatial compact model- ing method with existing muul-branch--aar based methods. For the latter, we adopt the exist- ing approach and implement both symmetric (Hou et al., 2024; Cao et al., 2024) and asym- metric (Zhu et al., 2023a; Hong et al., 2024) models using the same backbone network and training strategy as CSTrack. Our compact spatial modeling method, through the pro- posed Spatial Compact Module, integrates the features of both RGB and X modalities into compact feature space. This allows sub- sequent intra- and inter-modality spatial mod- eling to be performed using a dingle-branch network (i.e., Our One-stream Backbone in Sec. 3.1.3). In contrast, the dual-branch meth- ods require separate handling of the RGB and X modality features, using two parallel branches for ciora-mddlltyy spatial modeling and designing inter-modality feature interaction operations.
fex
fato
Fast)
FFN
Interact
Add Norm
FFN
Adf &
Norm
Add Norm
Add & Norm
Add & Norm
Attention
FFN
fri
fask
fra
F12-11
(4) Symmetrical Dual- Branch
(b) Asymmetrical Dual-Branch
Figure 6: Processing workflows of symmetric and asymmetric dual-branch architectures, illustrated using the computational process of their respective k-th network layer as an example.
Specifically, the spatial features S and f of the two modalities (as described in Eq. 1, with the superscript omitted for simplicity) are obtained through shared patch embedding and then input into the RGB and X branches, respectively. Below, we describe the feature processing methods at each layer in the dual-branch- branch- based approaches.
Dual-Branch (Symmetrical). This architecture employs two identical backbone networks as branches to process the two modalities. Consistent with CSTrack, we implement it using HiViT-base (Zhang et al., 2022b). Additionally, feature interaction between the two modalities is achieved by incorporating relevant modules at each layer, utilizing the widely adopted implementation from BAT (Cao et al., 2024). Taking the modeling process at the k-th layer of both branches as an
15

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
example (see Fig. 6 (a)), the specific computation is follows:
1Q--1 & RGB (A CA(A)
(16)
(1 Norm(fr(k-1) + + (CAH)((((-1)))
(17)
fa(K-1) Φx (( fa(
(18)
f2(k-1 Norm(fa(t-1) + l2(k-1) (CACH((r(R-)))
(19)
f1(x-1 1) (FN (F
(20)
[+(0)) Norm(Sr(k-1) (*(K-1)*
VEFNOO((EN-I)) +
(21)
("(0-1 1) FFN (R) (fa(())))
(22)
fa(K) Norm(fa(R-1) + [2(K-11 VPPNN)(RA-I)))
(23)
In the feature modeling process at the k-th layer, the input consists of the output from the previous layer (k - 1), namely fr(k-1) and fz(k-1)- For the RGB branch, the k-th layer includes a transformer-base attention network &b RGB and a CA((k)
feed-forward network FFNNGE Similarly, the symmetric X branch comprises &C and FFN To enable cross- (k) modal interaction, introduce two interaction modules, ())(() and V FFN(()- following the attention and feed-forward operations. Both V CA())) and V FFN(() are constructed using 3-layer fully connected networks. Finally, we obtain the outputs fr(k) and fz(K) from the k-th layer, which are then passed to the k + 1-th layer for a similar modeling process.
Dual-Branch (Asymmetrical). Inspired by the prompt learning paradigm (Lester et al., 2021), this architecture models features auxiliary prompts (Zhu et al., 2023a; Hong al., 2024). Specifically, we utilize the HiViT-base backbone (Zhang al., 2022b) for the RGB branch and adopt the latest modeling approach from OneTracker (Hong al., 2024) to construct the X branch and enable feature interaction between modalities. The detailed operations in the k-th layer of both branches
(see Fig. 6 (b)) are as follows:
(a(k11 = (LD FFN
(24)
(2(k11 = Norm( + FFN (6) (L(tt)))
(25)
Sa(K) FFN (k) (f"((()()))
(26)
B((-1) Norm(ff(t-n) + + CA(A) (Fr(k- (k- 1)) fr(k-1))),
RGB
(27)
((--1) = Norm( + FFN (f ((*(--1))),
(28)
fr(e) Norm(fHm- + fa(K)).
(29)
In the feature interaction process the k-th layer, the input consists of the output from the previous layer (k- i.e., fr(k-11
and f1( The four fully connected networks (FFN)) (R): FFN2
(R)) are used to facilitate modalttyy interaction and update the X modality features, resulting in fz(e)- The k-th layer of the RGB branch, consisting of an
(ke) FFN3
(k): FFNA,
attention network & C(k)) and feed-forward network FFNEGB RGB is used to process RGB features. Subsequently. fx(K)
further influences the RGB modality features (see Eq. 29). Finally, fr(e) and fa(K) are passed to the k + 1-th layer for similar modeling.
C.2. Implementation of Spatial Compact Module Variants
As shown in Tab. 6 of Sec. 1, we conduct ablation studies on the different components of the Spatial Compact Module. Below, we describe the implementation details of each ablation setting.
w/o queries. This setting removes the modality-specifi queries, q, and qr. Specifically, the initial spatial feature interaction (i.e., Eq. 2, 3, 4, 5) is simplified as:
16

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
f [f PCA(Fr:
(30)
J Norm(f² FCAlJa (t)),
(31)
J Norm(S, FFN((.),
(32)
& Norm([=-FFN(EE FFFN("))
(33)
Correspondingly, the composition of the compact spatial feature St also excludes these queries, simplifying Eq. 6 as:
f = Sco-
(34)
w/o cross-attention. This setting replaces the bidirectional cross-attention between the two modalities before compact spatial feature generation with self-attention for each modality. Specifically, Eq. 2 and Eq. 3 are modified to:
La/-5 f Norm([gs f*1 + &c f.l, (.rr):)))) ( fx1 Norm(l f Cc ftl, [9 f2l)).
(35)
(36)
unshare embedding. This setting removes the shared patch embedding and introduces separate patch embeddings for each modality. These patch embeddings share the same network structure.
C.3. Implementation of Temporal Compact Module Variants
As shown in Tab. 7 of Sec. 4.3.2, perform ablation studies different temporal feature representation methods. In this section, we provide the implementation details of each ablation setting.
Rol-based. This setting uses the Region of Interest (Rol) method (Ren et al., 2015) to represent compact temporal features. This approach has been widely adopted in the tracking field, such as in TrDiMP (Wang et al., 2021a) and JointNLT (Zhou et al., 2023). Specifically, we apply Rol processing to the search features sf. using the predicted bounding box scaled by 1.5, resulting in the temporal feature m E RR. xD For fairness, the length of the temporal feature token N%, obtained by Rol processing is set to be equal to the length Nm, of the temporal feature token constructed by our proposed TCM, i.e., No Nn 16.
w Query-based.
Recently, several trackers in the visual-only tracking domain have proposed query-based temporal modeling methods, such as ODTrack (Zheng et al., 2024) and AQATrack (Xie et al., 2024). These methods compress the temporal interaction between search and cue features into a small set of temporal queries, providing temporal guidance for compact way to construct temporal features, we include them in
the tracker. Since these temporal queries can be our comparison. Specifically, we define the temporal queries my E R N xD with N N 16. my is transformed into m' through the following operation:
m Norm(m, + (c Mz-1)),
(37)
q Norm(m' + (ca fc)), m
(38)
m' Norm(m", + (FN("")))
(39)
Here, 1 represents the aggregated dense temporal features from ml (i t 1) different time steps up to 1. The storage and update methods will be described later.
ht. This setting selects the top-N.. search tokens from the search features based solely on hū, which are then used to represent the compact temporal feature m².
hý. This setting selects the top-Nm. search tokens from the search features based solely on h/y which are then used to represent the compact temporal feature m²
17

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
he (hg & kt).. This setting corresponds to the method adopted by the proposed TCM, which combines hy and hig construct a refined target distribution heatmap h', subsequently used to generate the compact temporal feature m².
The above experimental variants demonstrate different methods of constructing temporal features m' at single time step. For storing and updating multi-step temporal features M = {m²) i=11" we employ the intuitive and widely used sliding window approach (Xie et al., 2024; Cai et al., 2024).
For video sequence with frames I t S T), the construction of M" varies at different time steps. At the initial step 1), .M does not yet store temporal features. After obtaining hi from the intermediate tracking results, first generate the temporal feature m based on h4, and initialize the I. temporal units in M This ensures that the tracker can proceed with forward inference and produce the initial tracking results. Using the final tracking results, we obtain hy. Then, combine it with h to derive m', which is used to reinitialize the temporal units, resulting in M
During the time interval t T), temporal features. Specifically, we remove the temporal feature with the smallest index and append the newly generated temporal feature unit m² at the end.
frst-in-first-our sliding window storage method to store the most recent
D. More Qualitative Results
Due to space limitations, Fig. 3 in Sec. 4.2.2 only presents three cases for the qualitative comparison between our model and the latest SOTA models. In this section, provide additional qualitative comparison results, illustrated in Fig. 7.
18

---

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
RGB+Depth
#120
8230
N220
A114
61140
RGB Thermal
EvBnt
8248
NOTO
P110
R160
84400
R5400
0999
PR30
4854
RE00
Ground Truth
UNTrack
SDSTrack
Ours
Figure 7: Qualitative comparison results of our tracker with other two trackers (i.e., UNTrack and SDSTrack) on RGB-D/T/E challenging cases. Better viewed in color with zoom-in.
19


Reviews:
[{"review_id": "InjGXeZqm4", "reviewer": "Reviewer_waVm", "summary": "This paper introduces CSTrack, a novel RGB-X tracker designed to enhance tracking performance by leveraging compact spatiotemporal features. Traditional RGB-X trackers typically process RGB and auxiliary modality (X) inputs separately using dual-branch architectures, which increases computational complexity and limits effective feature fusion. To address this limitation, CSTrack proposes a single-branch compact feature representation that integrates spatial and temporal information more efficiently. This approach incorporates two key modules: the Spatial Compact Module (SCM) and the Temporal Compact Module (TCM). The method is evaluated across several benchmarks, including RGB-D, RGB-T, and RGB-Event tracking datasets, such as DepthTrack, VOT-RGBD2022, LasHeR, RGBT234, and VisEvent.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer waVm,**\n\nThank you for your time and effort in reviewing our work. We appreciate your recognition of our novel, innovative, and efficient method, along with the comprehensive, well-structured, and convincing experiments, as well as easy-to-follow writing and the extensive supplementary material. We note your concerns about the performance and computational efficiency of our model. In response, we've conducted targeted comparative analyses and additional experiments, aimed at addressing your concerns.\n___\n### **Q1: Performance comparison with STTrack [1]**\n\nWe provide a detailed comparison between CSTrack and STTrack across various metrics:\n| Model | DepthTrack (F; Re; PR) | VOT-RGBD22 (EAO; Acc; Rob) | Lasher (SR; PR) | RGBT234 (MSR; MPR) | VisEvent (SR; PR) | Params | Flops | \n|-|-|-|-|-|-|-|-|\n| STTrack | 63.3; 63.4; 63.2 | 77.6; 82.5; 93.7 | 60.3; 76.0 | 66.7; 89.8 | 61.9; 78.6 | 128M | 91G | \n| CSTrack-B | 65.8; 66.4; 65.2 | 77.4; 83.3; 92.9 | 60.8; 75.6 | 70.9; 94.0 | 65.2; 82.4 | 75M | 36G |  \n| CSTrack-L | 67.1; 67.5; 67.3 | 78.2;83.8;93.2 | 61.8; 77.1 | 71.6; 96.0 | 66.1; 82.9 | 254M | 110G |  \n\n\n1. **CSTrack-B, the model presented in our paper, exhibits superior average performance.** As noted, CSTrack-B exhibits performance comparable to STTrack in VOT-RGBD22 (EAO ↓ 0.2%; Acc ↑ 0.8%; Rob ↓ 0.8%) and Lasher (SR ↑ 0.5%; PR ↓ 0.4%). However, CSTrack-B significantly outperforms STTrack in other benchmarks, achieving improvements such as PR ↑ 2.0% in DepthTrack, MPR ↑ 4.2% in RGBT234, and PR ↑ 3.8% in VisEvent, contributing to a superior average performance.\n\n2. **CSTrack targets a more challenging optimization objective.** \nUnlike our CSTrack, which uses a single model weight to simultaneously handle RGB-D/T/E tasks, STTrack optimizes separate weights for each task. Additionally, the STTrack repository suggests that it even optimizes individual checkpoints for each benchmark. While UnTrack [2] indicates that this approach may yield better performance, it is a less practical strategy for real-world applications.\n\n3. **CSTrack-B offers superior computational efficiency.** \nCSTrack-B offers a substantial advantage in Params (↓ 53M) and Flops (↓ 55G), achieving a better performance-computation balance. Focusing on maximizing performance, we develop CSTrack-L by adopting a larger backbone, surpassing STTrack across all benchmarks.\n\n___\n### **Q2: FLOPs reduction compared to dual-branch methods**\n\nAs shown in Table 5 of the paper, our single-branch method reduces FLOPs by only 2G compared to asymmetrical dual-branch methods. Here's the explanation: \n\n1. Asymmetrical dual-branch methods often incorporate parameter-intensive prompter modules into each layer of the RGB branch to integrate RGB-X features, as seen in the fully connected networks in OneTracker [4]. This results in some FLOPs benefits but leads to increased parameters (↑ 9M) and speed limitations (↓ 11 FPS).\n\n2. Table 5 indicates a notable disadvantage in tracking performance with this dual-branch method, e.g., MSR ↓ 3.7% in RGBT234. It suggests that the scale of FLOPs in this method makes it difficult to effectively integrate RGB-X features.\n___\n### **Q3:  Computational efficiency comparison with existing methods**\n\nThanks for your constructive suggestion.  We analyze the computational efficiency of recent open-source RGB-X methods, yielding the following results:\n\n|Model|DepthTrack (F; Re; PR) | RGBT234 (MSR; MPR) | VisEvent (SR; PR) | Params | Flops | Speed |\n|-|-|-|-|-|-|-|\n|UnTrack [2] | 61.0, 61.0, 61.0 | 62.5, 84.2 | 58.9, 75.5 | 99M | 24G | 24 FPS|\n|SDSTrack [3] | 61.4, 60.9, 61.9 | 62.5, 84.8 | 59.7, 76.7 | 102M | 108G | 22 FPS|\n|STTrack [1] | 63.3, 63.4, 63.2 | 66.7, 89.8 | 61.9, 78.6 | 128M | 91G | 27 FPS|\n|CSTrack-B | 65.8, 66.4, 65.2 | 70.9, 94.0 | 65.2, 82.4 | 75M | 36G | 33 FPS|\n\nCSTrack-B excels in Params and Speed, with only Flops lagging behind UnTrack. Similar to the analysis in **Q2**, UnTrack's core modules primarily consist of fully connected networks, offering Flops advantages but limits Params, Speed, and tracking performance. Overall, our method demonstrates superior tracking performance and computational efficiency.\n___\nWe hope these responses address your concerns and would be grateful if you could reconsider your rating. Should you have any additional feedback or require further clarification, please do not hesitate to let us know.\n___\n[1] Exploiting Multimodal Spatial-Temporal Patterns for Video Object Tracking, Hu et al., in AAAI 2025.\n\n[2] Single-Model and Any-Modality for Video Object Tracking, Wu et al., in CVPR 2024.\n\n[3] SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking, Hou et al., in CVPR 2024.\n\n[4] OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning, Hong et al., in CVPR 2024.", "claims_and_evidence": "Overall, the paper presents well-supported claims with convincing evidence through extensive experiments and ablation studies.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The paper primarily presents a methodological and empirical contribution rather than a theoretical one.", "experimental_designs_or_analyses": "The experimental design in the paper is well-structured and comprehensive, evaluating CSTrack across multiple datasets and conducting ablation studies to validate its core components.", "relation_to_broader_scientific_literature": "RGB-X Tracking and Multimodal Fusion: Traditional RGB-X tracking methods typically use dual-branch architectures,\nTemporal Feature Modeling in Tracking:  Many existing RGB-X trackers lack effective temporal modeling.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-structured and easy to follow. Extensive supplementary material provides additional experimental insights, dataset details, and qualitative comparisons.\n2. The Spatial Compact Module (SCM) and Temporal Compact Module (TCM) offer a novel approach to multimodal feature integration while reducing computational overhead.\n3. The use of compact spatiotemporal representations is an innovative departure from the traditional dual-branch RGB-X tracking architectures.\n\n\nWeaknesses: \n1. The proposed method does not achieve state-of-the-art performance on some of the metrics compared to those in the paper 'Exploiting Multimodal Spatial-Temporal Patterns for Video Object Tracking' (AAAI 2025).\n2. The reduction in FLOPs compared to dual-branch methods is not significant (only a minor drop from 38G to 36G) in Table 5.\n3. It is recommended to compare the efficiency of the proposed method, including parameters and FLOPs, with other methods to better highlight its computational performance.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PfvrmGcm3O", "reviewer": "Reviewer_W7oh", "summary": "The paper introduces CSTrack, an RGB‐X tracker that leverages a compact spatiotemporal feature representation to improve tracking performance while reducing computational complexity. Unlike existing methods that typically employ dual-branch architectures to process RGB and X modalities separately, CSTrack integrates both modalities into a single compact feature space using bidirectional cross-attention and learnable modality-specific queries. Moreover, it constructs a refined target distribution heatmap by combining intermediate and final tracking results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer W7oh,**\n\nWe sincerely appreciate your thorough review of our work and are grateful for your recognition of our sound motivation, effective method, and extensive experimental analysis. In response to your concerns, we have provided detailed explanations below:\n___\n### **Q1: Noise Interference in modality-missing tracking**\n\nThanks for your constructive suggestion on analyzing noise impact in modality-missing tracking task. IPL [1] recently proposes the modality-missing tracking benchmarks, along with a baseline method that duplicates available modality data, which we adopt. As shown in Tables 2 and 4 of the paper, modality-missing settings significantly degrade model performance. We attribute this mainly to the influence of two types of noise: \n\n1. **Miss of advantage modality.** Figure 4 in the paper introduces the concept of the advantage modality, which primarily conveys object appearance information in challenging scenarios. In modality-missing task, this advantage modality may be unavailable, leaving the tracker susceptible to noise interference from the non-advantage modality due to its lack of appearance features.\n\n2. **Training and inference bias.** CSTrack is trained with complete RGB-X modalities but directly inferred in modality-missing settings. The difference in input data types introduces noise that significantly affects performance. Despite this, CSTrack exceeds IPL, a tracker tailored for such scenarios. We think this is primarily due to our shared patch embedding, which transforms initial RGB and X data into a unified feature space, improving compatibility with varied inputs. Below, we test the impact of shared embedding to further confirm our analysis:\n|Setting | LasHeR-Miss (SR; PR) | RGBT234-Miss (MSR; MPR) |\n|-|-|-|\n|Unshared Embedding | 48.7, 60.6 | 65.3, 86.5 |\n|Shared Embedding | 50.3, 62.2 | 68.0, 89.8 |\n___\n### **Q2: Distribution analysis of RGB and RGB-X datasets**\n\nWe appreciate your valuable suggestions on the distribution analysis of various modality datasets. The analysis is as follows:\n\n1. **Distribution differences across modality datasets.** We perform statistical analyses on representative RGB and RGB-D/T/E datasets to obtain normalization mean values of training videos, including the ImageNet for comparison as a reference for natural image distribution:\n|Datasets| ImageNet | LaSOT (RGB-only) |DepthTrack (RGB-D)|Lasher (RGB-T)|VisEvent (RGB-E)|\n|:-|:-|:-|:-|:-|:-|\n| RGB Mean| 0.485; 0.456; 0.406 | 0.456; 0.459; 0.426 | 0.417; 0.414; 0.393 | 0.500; 0.499; 0.471 | 0.418; 0.375; 0.317 |\n| X Mean| - | -| 0.574; 0.456; 0.240 | 0.372; 0.372; 0.368 | 0.935; 0.904; 0.949 |\n  \n    - RGB modality: LaSOT and DepthTrack, collected in typical natural environments, have mean values similar to ImageNet. However, Lasher, often focusing on dark environments, and VisEvent, which collects predominantly yellow RGB images, deviate from these natural image means.\n\n    - X modality: as shown in Figure 3 of the paper, X modalities vary greatly in image style, resulting in significant differences in mean values. \n\n2. **Model's discriminative ability across modality datasets.** Using features from the one-stream backbone (see Equation 8 in the paper), we extract tokens associated with learnable queries and build a four-class classifier with two fully connected layers to identify the input modality type: RGB-only, RGB-D, RGB-T, and RGB-E. We freeze the existing model and train this classification head for 2 epochs. The accuracy results show the model can effectively distinguish different modality datasets, thanks to the clear distribution differences.\n|Datasets|LaSOT (RGB-only)|DepthTrack (RGB-D)|Lasher (RGB-T)|VisEvent (RGB-E)|\n|:-|:-|:-|:-|:-|\n|Accuracy|100%|98%|100%|100%|\n3. **Advantages of joint training.** Our ablation study, detailed in Table 8 (#3) of the paper, shows that the benefits from joint training, including larger datasets and knowledge sharing across modalities, outweigh the drawbacks of domain shift, resulting in enhanced model performance.\n___\n### **Q3: Function of bidirectional cross-attention and modality-specific queries.**\n\nOur spatial compact module aims to integrate RGB and X input streams into a unified feature space, facilitating simplified and effective spatial modeling. When a modality has low-quality or noisy data, termed \"non-advantage modality\" (see Figure 4), bidirectional cross-attention first emphasizes the advantage modality's representations while minimizing those of the non-advantage modality. Next, modality-specific queries preserve global semantic information of each modality, offering additional reference information for subsequent feature integration.\n___\nWe hope these explanations can resolve your concerns. If you have any more questions or need further clarification, feel free to reach out to us.\n___\n[1] Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarks,  Lu et al., in IJCV 2O24.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The paper does not include formal proofs for its theoretical claims. Instead, its contributions are presented through the design of SCM and TCM modules, which are supported by extensive empirical evaluations and ablation studies.", "experimental_designs_or_analyses": "Yes, the experimental design leverages foundational benchmarks in the relevant domain.", "relation_to_broader_scientific_literature": "Traditionally, many methods—such as ViPT, OneTracker, and SDSTrack—rely on dual-branch architectures to process RGB and auxiliary modalities separately. CSTrack builds on these approaches by unifying the two modalities into a single compact feature space, a strategy that mirrors recent trends in transformer-based models where learnable queries and cross-attention mechanisms have proven effective for capturing complex relationships in visual data.", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "1. Simplifying the dual-stream network in RGB‐X tracking by integrating both modalities into a single compact feature space is a sound idea, as it helps reduce computational complexity and simplifies the model architecture.\n2. Constructing a refined target distribution heatmap by combining intermediate and final tracking results is an effective practical method for selecting key target features, thereby enhancing tracking robustness.", "other_comments_or_suggestions": "1. The method for modality missing simply duplicates the available modality data (e.g., copying RGB to the X channel) without accounting for potential noise interference in the X modality. This approach may inadvertently introduce erroneous signals when the auxiliary modality is noisy.\n2. The paper uses a joint training strategy with both RGB‐X and RGB datasets, but lacks a detailed analysis of their distribution differences and potential domain shift issues.", "questions_for_authors": "1. For the Spatial Compact Module, could you elaborate on how the bidirectional cross-attention and modality-specific queries behave when one modality (e.g., thermal or depth) provides low-quality or noisy data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ugcyDpjldw", "reviewer": "Reviewer_f47k", "summary": "The article proposes using compact spatiotemporal features for RGB-X tracking. Unlike the commonly used two-stream frameworks, It employs a one-stream structure to reduce computational overhead.  Experiments on multiple downstream tasks demonstrate that this paper achieves the SOTA performance.\n\n## update after rebuttal\nThe authors' responses have resolved most of my concerns, and I maintain my 'weak accept' rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer f47k,**\n\nThanks for your time and effort in reviewing our work. Your recognition of our novel method, comprehensive experiments, SOTA performance, and writing quality greatly encourages us. We hope the following responses can address your concerns.\n___\n### **Q1: Comparison with recent works**\n\nThank you for mentioning some recent trackers. Here’s how our CSTrack differs:\n\n1. TATrack and MCTrack use two symmetric backbones for separate RGB and X processing. ViPT employs an asymmetric X branch, while USTrack and Yang [1] concatenate RGB and X feature tokens for multimodal feature integration with a one-stream backbone. However, they still need to process RGB-X dual inputs within the backbone. M3PT uses middle fusion but relies heavily on the early dual-branch architecture for performance.\n2. In contrast, our CSTrack integrates RGB-X dual inputs into a compact feature space, reducing model complexity and computational costs. Additionally, the table below compares various trackers, highlighting the effectiveness of our method.\n|Tracker|Lasher (SR, PR)|RGBT234 (MSR, MPR)|\n|-|-|-|\n|TATrack|56.1; 70.2|64.4; 87.2|\n|MCTrack|57.1; 71.6|65.6; 87.5|\n|ViPT|52.5; 65.1|61.7; 83.5|\n|USTrack|- |65.8; 87.4|\n|Yang [1]|56.7; 71.4|65.1; 87.3|\n|M3PT|56.1; 70.0|63.9; 86.5|\n|CSTrack-B|60.8; 75.6|70.9; 94.0|\n___\n### **Q2: Further explanation of model details**\n\n1. **Basic model setup.** CSTrack is a unified tracker trained jointly across different datasets, enabling testing without modal differentiation. This is mentioned in Section 4.1 and will be included in the method section of the revised version.\n\n2. **SCM description.** We apologize for the error in Equation 3 of our paper. The RGB related features should be processed as $[q'_r; f^{'t}_r]$, not $[q_r; f^{t}_r]$. Below is the corrected equation, confirming our use of the serial computation method in Figure 1.\n\n\n    $[q' _x; f^{'t} _x] = Norm( [q _x; f^{t} _x] + \\Phi _{CA} ([q _x; f^{t} _x], [q' _r; f^{'t} _r]))$\n\n___\n### **Q3: Further explanation of experimental results**\n\n1. **Ablation study of core modules.** The table below presents ablation results for key modules: Spatial Compact Module (SCM) and Temporal Compact Module (TCM). Adding SCM and TCM improves performance, validating our design. We also offer Lasher benchmark results, consistent with performance rankings observed in RGBT234.\n|Setting|DepthTrack (F; Re; PR)|RGBT234 (MSR; MPR)|Lasher (SR; PR)|VisEvent (SR; PR)|\n|-|-|-|-|-|\n|Baseline|63.6; 64.1; 63.1|68.2; 92.4|57.6; 72.9|62.9; 79.6|\n|+ SCM|65.1; 65.3; 64.9|69.6; 93.0|59.8; 74.7|64.1; 81.0|\n|+ TCM|65.8; 66.4; 65.2|70.9; 94.0|60.8; 75.6|65.2; 82.4|\n\n2. **SCM analysis.**  Table 6 shows cross-attention and shared embedding in SCM boosts DepthTrack but impacts RGBT234 and VisEvent less. We attribute this to higher noise and fluctuations in depth data compared to thermal and event data, as detailed in Figure 3 and our supplementary videos. This suggests these designs are most effective in challenging scenarios. For RGBT234 and VisEvent,  other designs within SCM are sufficient for strong performance, as shown by SCM's gains across all benchmarks (see the table above).\n\n3. **Tracking speed.** In Table 5,  our method achieves 35 FPS, surpassing dual-branch trackers at 24 and 18 FPS. This meets expectations and satisfies real-time tracking requirements (exceeding the VOT challenge threshold of 20 FPS).\n___\n### **Q4: Additional experimental analysis**\n\n1. **Performance with ViT backbone.** We retrain CSTrack using the ViT backbone. The results below demonstrate our compact modeling exceeds the performance gains of the HiViT backbone, and our tracker achieves SOTA performance even with the ViT backbone, validating our approach's effectiveness.\n|Backbone|Compact Modeling|DepthTrack (F; Re; PR)|Lasher (SR; PR)|VisEvent (SR; PR)|\n|-|-|-|-|-|\n|HiViT| ✔ |65.8; 66.4; 65.2|60.8; 75.6|65.2; 82.4|\n|HiViT| ✘ |63.6; 64.1; 63.1|57.6; 72.9|62.9; 79.6|\n|ViT| ✔ |63.9; 64.8; 64.3|58.6; 74.3|64.6; 81.1|\n\n2. **RGB-only tracking test.** Our core compact modeling targets RGB-X dual-stream inputs, making RGB-only tracking not aligned with our objectives. Therefore, we did not conduct this test.\n\n3. **Impact of bidirectional cross-attention order.** We fine-tune our model by swapping the cross-attention order, and the results show minor performance fluctuations. We speculate that the shared patch embedding and the subsequent one-stream backbone can diminish the influence of interaction order.\n|Setting|DepthTrack (F; Re; PR)|Lasher (SR; PR)|VisEvent (SR; PR)|\n|-|-|-|-|\n|CSTrack-B|65.8; 66.4; 65.2|60.8; 75.6|65.2; 82.4|\n|+ swap order|65.8; 66.0; 65.4|60.7; 75.8|65.2; 82.4|  \n___\nWe hope these responses address your concerns and kindly invite you to reassess your rating. Feel free to reach out to us if you have any further questions.\n___\n[1] From Two Stream to One Stream: Efficient RGB-T Tracking via Mutual Prompt Learning and Knowledge Distillation, Yang et al., in arxiv 2024.", "claims_and_evidence": "The article is well-written with clear descriptions and motivation, and comprehensive experiments.", "methods_and_evaluation_criteria": "The evaluation metrics and RGB-X tracking datasets are consistent with this paper. However, in the ablation study, it is suggested to modify the dataset for RGB-T experiments to LasHeR, as this dataset is more challenging and authoritative.", "theoretical_claims": "The article focuses on empirical research and validates the effectiveness of CSTrack through experimental results.", "experimental_designs_or_analyses": "The authors conducted sufficient experiments, including experiments on different modalities and ablation studies on the modules, which basically meet the requirements. However, there are issues with unclear descriptions in the experimental design. For example, it is not clear whether a unified tracker is trained for different modalities or if separate trackers are trained for each modality. Additionally, most of the trackers used for comparison are relatively old, and there is a lack of comparison with the latest trackers.", "relation_to_broader_scientific_literature": "Previous RGB-X tracking studies typically employed a dual-branch structure and focused on fusion methods. In contrast, this paper uses a one-stream architecture, which is somewhat innovative.", "essential_references_not_discussed": "Some of the latest studies have not been included. For example, TATrack (Temporal adaptive RGBT tracking with modality prompt) and MCTrack (Towards modalities correlation for RGB-T tracking). Additionally, M3PT (Middle fusion and multi-stage, multi-form prompts for robust RGB-T tracking) also attempts to use a single branch to accomplish multimodal tracking tasks, but it differs from the approach presented in this paper. Please compare and discuss these methods.\nIn addition, it is key to hightlight the difference with other single-stream multi-modal tracking methods, such as:\n[1] Unified Single-Stage Transformer Network for Efficient RGB-T Tracking. IJCAI 2024\n[2] From Two Stream to One Stream: Efficient RGB-T Tracking via Mutual Prompt Learning and Knowledge Distillation", "other_strengths_and_weaknesses": "Strengths:\n1. This paper proposes a novel approach to integrate fuse spatiotemporal features for multimodal tracking while reducing computational overhead.\n2. The experimental results demonstrate the effectiveness of the module proposed in this paper\n\nWeaknesses:\n1. some sections could benefit from clearer explanations visualizations.\n2. In Table 6, RGBT234 and VisEvent seem to be insensitive to the ablated components, but the paper does not provide an explanation for this observation.\n3. The description of SCM is unclear.  In Equations 2, 3, the cross-attention is computed in parallel, while Figure 1 appears to show a serial computation.  Does the order of cross-attention affect the experimental results, or is there an error in Figure 1?", "other_comments_or_suggestions": "1. The one-stream structure, as a major characteristic of this tracker, is not prominently highlighted, and there is a lack of discussion on other one-stream multimodal trackers, such as ViPT, USTrack, M3PT.  As a distinguishing feature, it is recommended to add content in this regard.\n2.  Although the method presented in this paper has a smaller FLOPs, the speed appears to be abnormal, with only 35 fps, which is comparable to that of dual-stream trackers. Is there anything wrong with the experiments?", "questions_for_authors": "1. How does your model differentiate between the inputs of the three modalities?\n2. The authors use the HiViT backbone, while most trackers employ the ViT backbone. What are the results if the SCM and TCM are moved to the ViT backbone?\n3. The SCM and TCM appear to be transferable to RGB tracking tasks, requiring only modifications to the cross-attention.  Have you conducted such experiments? If so, what is the results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Based on the reviewers’ feedback and the authors’ rebuttals, I recommend weakly accepting this paper. The submission presents a novel approach to RGB-X tracking by integrating multimodal data into a compact feature space, which simplifies the model architecture and reduces computational complexity. The extensive experiments and ablation studies provide strong empirical support for the proposed method. While there are some limitations, such as the handling of noisy data and the need for further analysis on dataset distributions, the authors have addressed these concerns adequately in their rebuttal. The innovations and performance improvements demonstrated by CSTrack make it a valuable contribution to the field of multimodal tracking.
Decision: Accept (poster)