Divert More Attention to Vision-Language Tracking
Authors: Mingzhe Guo
Zhipeng Zhang
Heng Fan
Liping Jing
Venue: NeurIPS 2022 Accept (2022)
Keywords: Visual Object Tracking
Multimodal Learning
Vision-Language Representation
Asymmetrical Searching Strategy

Abstract: Relying on Transformer for complex visual feature learning, object tracking has witnessed the new standard for state-of-the-arts (SOTAs). However, this advancement accompanies by larger training data and longer training period, making tracking increasingly expensive. In this paper, we demonstrate that the Transformer-reliance is not necessary and the pure ConvNets are still competitive and even better yet more economical and friendly in achieving SOTA tracking. Our solution is to unleash the power of multimodal vision-language (VL) tracking, simply using ConvNets. The essence lies in learning novel unified-adaptive VL representations with our modality mixer (ModaMixer) and asymmetrical ConvNet search. We show that our unified-adaptive VL representation, learned purely with the ConvNets, is a simple yet strong alternative to Transformer visual features, by unbelievably improving a CNN-based Siamese tracker by 14.5% in SUC on challenging LaSOT (50.7%$\rightarrow$65.2%), even outperforming several Transformer-based SOTA trackers. Besides empirical results, we theoretically analyze our approach to evidence its effectiveness. By revealing the potential of VL representation, we expect the community to divert more attention to VL tracking and hope to open more possibilities for future tracking beyond Transformer. Code and models are released at https://github.com/JudasDie/SOTS.

Divert More Attention to Vision-Language Tracking
Mingzhe Guo!* Zhipeng Zhang?* Heng Fan
Liping Jing't
Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University
DiiD Chuxing, Beijing, China
DDpartment of Computer Science and Engineering, University of North Texas [mingzheguo, lpjing J @ @bjtu.edu.cn zhipeng.zhang www.goloo.oooooooooooooo heng.fan fan @unt.edu
Abstract
Relying on Transformer for complex visual feature learning, object tracking has wit- nessed the standard for tute-of-theeart (SOTAs). However, this advancement accompanies by larger training data and longer training period, making tracking increasingly expensive. In this paper, demonstrate that the Transformer-rclaance is not necessary and the pure ConvNets are still competitive and even better yet more economical and friendly in achieving SOTA tracking. Our solution is unleash the power of multimodal vision-language (VL) tracking, simply using ConvNets. The essence lies in learning novel unified-adaptive VL representations with our modality mixer (ModaMixer) and asymmetrical ConvNet search. We show that our unified-adaptive VL representation, learned purely with the ConvNets, is simple yet strong alternative to Transformer visual features, by unbelievably improving CNN-based Siamese tracker by 14.5% in SUC on challenging LaSOT (50.7% 70.7%%665.2%% ()). ). even outperforming several Transformer based SOTA trackers. Besides empirical results, we theoretically analyze our approach to evidence its effectiveness. By revealing the potential of VL representation, we expect the community to divert more attention to VL tracking and hope open more pos- sibilities for future tracking beyond Transformer. Code and models are released at https::/ / wtt.ttt...oooo.eeeeeeeem JudasDie/SOTS..
Introduction
Transformer tracking recently receives surge of research interests and becomes almost a necessity to achieve state- of-the-art (SOTA) performance [8, 52, 10]. The success of Transformer trackers mainly attributes to attention that enables complex feature interactions. But, is this complex attention the only way realizing SOTA tracking? Or in
Comparison of Trackers on LaSOT
CNN Tracking
Trandorme fracking
S0C(9)
poos
other words, is Transformer the only path to SOTA?
We answer no, and display Transformer-free path us- ing pure convolutional neural network (CNN). Different than complex interactions in visual feature by attention requiring more training data and longer training time, our alternative to explore simple interactions of multimodal, i.e, vision and language, through CNN. In fact, language, an equally important cue as vision, has been largely ex- plored in vision-related tasks, and is not new to tracking.
borth Troe
4
Trmss
meet
sranke
155
Tracker
Figure 1: Comparison between CNN-
based and Transformer-bass trackers LaSOT [17].
*Corresponding author. This work is co-supervised by Prof. Liping Jing and Dr. Zhipeng Zhang.
Equul Contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Prior works [20, 34, 19] have exploited vision- (VL) multimodal learning for improving tracking. However, the performance falls far behind current SOTAs. For instance on LaSOT [17]. the gap between current best VI. tracker [20] and recent Transformer tracker [8] is absolute 10.9% in SUC (see Fig. 1). So, what is the bottleneck of VI. tracking in achieving SOTA?
The devil is in VL representation. Feature representation has been shown to be crucial in improving tracking [51, 61, 32, 24, 10]. Given two modalities of vision and language, the VL feature is desired to be unified and adaptive [40, 58J. The former property requires deep interaction of vision and language, while the latter needs VL feature to accommodate different scenarios of visual and linguistic information. However, in existing VL trackers, vision and language are treated independently and processed distantly until the final result fusion. Although this fusion may easily get two modalities connected, it does not accords with human learning procedure that integrates multisensory by various neurons before causal inference [3], resulting in a lower upper-bound for VI. tracking. Besides, current VI. trackers treat template and search branches as homoplasmic inputs, and adopt symmetrical feature learning structures for these two branches, inherited from typical vision-only Siamese tracking [20]. We argue the mixed modality may have different intrinsic nature than the pure vision modality, and thus requires a more flexible and general design for different signals.
Our solution. Having observed the above, we introduce a novel unified-adaptiv vision-language representation, aiming for SOTA VL tracking without using Transformer'. Specifically, we first present modality mixer, or ModaMixer, conceptually simple but effective module for VL interaction. Language a high-level representation and its class embedding can help distinguish targets of different categories (e-g., cat and dog) and meanwhile the attribute embedding (e.g., color, shape) provides strong prior to separate targets of same class (e.g., cars with different colors). The intuition is, channel features in vision representation also reveal semantics of objects [22, 56]. Inspired by this, ModaMixer regards language representation as a selector to reweight different channels of visual features, enhancing target-specific channels as well as suppressing irrelevant both intra- and inter-class channels. The selected feature is then fused with the original feature, using a special asymmetrical design (analyzed later in experiments), to generate the final unified VL representation. A set of ModaMixers are installed in a typical CNN from shallow to deep, boosting robustness and discriminability of the unified VL representation at different semantic levels. Despite simplicity, ModaMixer brings 6.9% gains over a pure CNN baseline [23] (i.e., 50.7%-+57.6%).
Despite huge improvement, the gap to SOTA Transformer tracker [8] remains (57.6% v.s. 64.9%). To mitigate the gap, we propose an asymmetrical searching strategy (ASS) to adapt the unified VL representation for improvements. Different from current VL tracking [20] adopting symmetrical and fixed template and search branches as in vision-only Siamese tracking [32], we argue that the learning framework of mixed modality should be adaptive and not fixed. To this end, ASS borrows the idea from neural architecture search (NAS) [63, 42] to separately learn distinctive and asymmetrical networks for mixed modality in different branches and ModaMixers. The asymmetrical architecture, to our best knowledge, is the first of its kind in matching based tracking. Note, although NAS has been adopted in matching-hee -ased tracking [55], this method finds symmetrical networks for single modality. Differently, ASS is applied on mixed modality and the resulted architecture is symmetrical. Moreover, the network searched in ASS avoids burdensome re-training on ImageNet [14], enabling quick reproducibility of our work (only 0.625 GPU days with a single RTX RTX-2080TT). Our ASS is general and flexible, and together with ModaMixer, it surprisingly shows additional 7.6% gains (i.e., 17.6%))).2%) evidencing our argument and effectiveness of ASS.
Eventually, with the unified-adaptive representation, we implement the first pure CNN-based VL tracker that shows SOTA results comparable and even better than Transformer based solutions, without bells and whistles. Specifically, we apply our method to a CNN baseline SiamCAR [23]. and the resulted VL tracker VLTSCAR shows 65.2% SUC on LaSOT [17] while running at 43FPS, unbelievably improving the baseline by 14.5% and outperforming SOTA Transformer trackers [8, 52] (see again Fig. 1). We observe similar improvements by our approach on other four benchmarks. Besides empirical results,
provide theoretical analysis to evidence the effectiveness of our method. Note that, our approach is general in improving vision-only trackers including Transformer-ban based ones. We show this by applying it to TransT [8] and the resulted tracker VLTTT shows 2.4% SUC gains (i.e., 64.9% -6.9%%.7..7) 67.3%), evidencing its effectiveness and generality.
'eere we stress that we do not use Transformer for visual feature learning as in current Transformer trackers or for multimodal learning. We only use it in language embedding extraction (i.c., BERT (15))

---

Vision Template
Stage1
Stage2
Stage3
Stage4
Cls. Head
Language black blcycle man the mountaãn road
M Ldod Miteer
Moda Miterr
Moda Mitter
Modi Mixer
Language Model
Match
Modia Mixer
Moda Mixer
Moda Mixer
Moda Mixer
Reg.
Head
Vision Search
V'siin-Lannnn anguage Representatinn earning
Localization
Figure 2: The proposed vision-language tracking framework. The semantic information of language description is injected to vision from shallow to deep layers of the asymmetrical modeling architecture learn unified-adaptive vision-language representation.
We are aware that one can certainly leverage the Transformer [39] to learn a good (maybe better) VL representation for tracking, with larger data and longer training period. Different than this, goal is to explore a cheaper way with simple architectures such as pure CNN for SOTA tracking performance and open possibilities for future tracking beyond Transformer. In summary, contributions are four-fold: (i) we introduce a novel andatie- vision-language representation for SOTA VL tracking; (ii) propose the embarrassingly simple yet effective ModaMixer for unified VL representation learning; (iii) we present ASS to adapt mixed VL representation for better tracking and (iv) using pure CNN architecture, we achieve SOTA results multiple benchmarks.
2 Related Work
Visual Tracking, Tracking has witnessed great progress in the past decades. Particularly, Siamese tracking 47], that aims to learn generic matching function, is representative branch and has rev- olutionized with numerous extensions [33, 61, 32, 57, 18, 25, 60, 62, 9]. Recently, Transformer [49] has been introduced to Siamese tracking for better interactions of visual features and greatly pushed the standard of state-of-the-arr performance [8, 52, 46, 10, 35J. From different perspective than using complex Transformer, we explore multimodal with simple CNN to achieve SOTA tracking.
Vision-Language Tracking. Natural language contains high-level semantics and has been leveraged foster vision-related tasks [21, 31, 2] including tracking [34, 19, 20]. The work [34] first introduces linguistic description to tracking and shows that language enhances the robustness of vision-based method. Most recently, SNLT [20] integrates linguistic information into Siamese tracking by fusing results respectively obtained by vision and language. Different from these VL trackers that regard vision and language as independent cues with weak connections only result fusion, we propose ModaMixer to unleash the power of VI. tracking by learning unified VI. representation.
NAS for Tracking. Neural architecture search (NAS) aims at finding the optimal design of deep network architectures [63, 42, 37, 26) and has been introduced to tracking [55, 60). LightTrack [55] tends search a lightweight backbone but is computationally demanding (about 40 V GPU days). AutoMatch uses DARTS [37] to find better matching networks for Siamese tracking. All these methods leverage NAS for vision-only tracking and search a symmetrical Siamese architecture. Differently, our work searches the network for multimodal tracking and tries to find a more general and flexible asymmetrical two-stream counterpart. addition, our search pipeline only takes 0.625 RTX-2080Ti GPU days, which is much more resource- frienddl
3 Unified-Adaptivy Vision-Language Tracking
This section details our mnified-adaptive vasion-language (VL) tracking as shown in Fig. 2. specific, we first describe the proposed modality mixer for generating unified multimodal representation and then asymmetrical network which searches for learning adaptive VI. VI. represetaation. Afterwards, we illustrate the proposed tracking framework, followed by theoretical analysis of our method.

---

3.1 Modality Mixer for Unified Representation
The essence of multimodal learning is a simple and effective modality fusion module. As discussed before, existing VL trackers simply use a later fusion way, in which different modalities are treated independently and processed distantly until merging their final results [20, 34]. Despite the effective- ness to some extent, the complementarity of different modalities in representation learning is largely unexplored, which may impede the multimodal learning to unleash its power for VL tracking. In this work, we propose the modality mixer (dubbed ModeMixer) to demonstrate compact way to learn unified vision-language representation for tracking.
ModaMixer considers language representation as selector to reweight channels of vision fea- tures. In specific, given the language descrip-
Langage
Linear
Block ASs
tion with N words of a video', a language model [15] is adopted to abstract the sentence to semantic features with size of (N + 2) x The extra "2" denotes the "[CLS][SEP]" char- acters in language model processing (see [15] for more details). Notably, descriptions for dif- ferent videos may contain various length NN. To
1x d
Fornst ModaMiixrr
Vision
Block Ass
ensure the ModaMixer applicable for all videos, we first average the features for all words along sequence length dimension "(N+2)" generate a unique language representation f E 1R for each description. Then a linear layer is followed to align the channel number of fi with the corresponding vision feature f. E 12 #x Channel selec- tor is expressed as Hadamard product operator, which point-wisely multiplies language representation (1 x C) embedding of each spatial position in the vision feature f.. Finally, a residual connection between the mixed feature f,,, and vision feature f, is conducted to avoid losing informative vision details. In a nutshell, the ModaMixer can be formulated as,
HxWxC
Figure 3: Illustration of the ModaMixer.
fm BlockASS (Linear (f) @ f.) + BlockASS (fu);
(I)
where @ denotes Hadamard product, Linear is a linear projection layer with weight matrix size of d x C' for channel number alignment, and Block indicates post-processing block before residual connection. Please note that, to enable adaptive feature modeling for different modalities, we search different BlockAss process features before and after fusion (see Sec. 3.2 for more details). The proposed ModaMixer is illustrated in Fig. 3. Akin to channel attention [44, 22], the high-level semantics in language representation dynamically enhance target-specific channels in vision features, and meanwhile suppress the responses of distractors belonging to both inter- and intra-classes.
3.2 Asymmetrical Search for Adaptive Vision-Language Representation
Besides the fusion module, the other crucial key for vision-language tracking is how to construct the basic modeling structure. The simplest strategy is to inherit symmetrical Siamese network from vision-based tracking (e.g.. [5, 32), as in current VL trackers [20]. But the performance gap still remains if using this manner, which is mostly blamed the neglect of the different intrinsic nature between VI.-based multimodal and vision-only single modality. To remedy this, we propose an asymmetrical searching strategy (dubbed ASS) to learn an adaptive modeling structure for pairing with ModaMixer.
The spirits of network search are originated from the field of Neural Architecture Search (NAS). We adopt a popular NAS model, in particular the single-path one-shot method SPOS [26], for searching the optimal structure of our purpose. Although SPOS has been utilized for tracking [55], our work significantly differs from it from two aspects: 1) Our ASS is tailored for constructing an asymmetrical two-stream network for multimodal tracking, while [55] is designed network for vision-only single-modality tracking. Besides, we search layers both in the backbone network and the post-processing BlockAss in the ModaMixer (see Eq. 1); 2) Our ASS reuses the
find symmetrical Siamese
'The language description in tracking is generated only by the initial target object in the first frame.

---

Table The asymmetrical architecture learned by ASS. is the stem convolution layer. (OCO0) represents the basic ASS unit, where the first three ones indicate Shuffle block [59] with kernel sizes of (3,5.7), respectively, and the last one denotes Shuffle Xception block [59] with kernel size of 3.
Moda
Moda
Moda
Moda
Stem
Stagel
Stage2
Stage3
Stage4
Mixer
Mixer
Mixer
Mixer
Template
Search
pre-trained supernet from SPOS, which avoids the burdensome re-training on ImageNet [14] (both for the supernet and found subnet) and thus reduces the time complexity of our search pipeline to 1/64 of that in LightTrack [55] (i.e., 0.625 RTX-2080Ti GPU days v.s. 40 V100 GPU days). Due to limited space, please refer to appendix for more details and comparison of our ASS and [55].
The search space and search strategy of ASS are kept consistent with the original SPOS [26]. In particular, the search pipeline is formulated as,
W = argmin EawI(A) [Lmmin [LminnNVa (()))),, W
a' argmax SUC (N(a,Wa())) Wa(a))),
aEA
where A represents the architecture search space of the network N, is sample from A and W denotes the corresponding network weights. Notably, the network / includes three components N = [ype, Pa. Pm J. where each indicates backbone for the template branch (t,, backbone for the search branch y., and layers in the ModaMixer fm. The whole pipeline consists of training supernet tracking datasets via random sampling I from search space A (Eq. 2) and finding the optimal subnet via evolutionary algorithms (Eq. 3). The SUC (success score) on validation data is used as rewards of evolutionary algorithms. Tab. demonstrates the searched asymmetrical networks in VL tracking. For more details of ASS, we kindly refer readers to appendix or [26].
3.3 Tracking Framework
With the proposed ModaMixer and the searched asymmetrical networks, construct new vision- language tracking framework, as shown in Fig. 2 and Tab. 1. Our framework matching- tracking. Both template and search backbone networks contain 4 stages with the maximum stride of the chosen blocks of each stage are denoted with different colors in Tab. 1. ModaMixer is integrated into each stage of the template and search networks learn informative mixed representation. is worth noting that, the asymmetry is revealed in not only the design of backbone networks, but also the ModaMixer. Each ModaMixer shares the same meta-structure as in Fig. 3, but comprises different post processing layers BlockAss to allow adaption to different semantic levels (i.e., network depth) and input signals (i.e., template and search, pure-vision and mixed feature each ModaMixer). With the learned unified-adaptive VI. representations from the template and search branches, we perform feature matching and target localization, the same in our baseline.
3.4 A Theoretical Explanation
This section presents a theoretical explanation of our method, following the analysis in [29]. Based on the Empirical Risk Minimization (ERM) principle [45], the objective of representation learning to find better network parameters 0 by minimizing the empirical risk, i.e.
min + (0 A
Σ L
s.t. OM
where L denotes loss function, M represents the modality set, n indicates sample number, X; {ot 22 /...MM/ } is the input mutimodal signal, y training label, and FF demotes optimization space
5

---

Table State-of-the-an comparisons on LaSOT [17], LaSOT Ext [16]. TNL2K [53]. GOT-1OK [28] and OTB99- (OTB99-L) [34]. TransT and SiamCAR are baselines of the proposed VLTTT and VLT SCAR. respectively. and denote the settings of "O-tensor" and "template" without language description. All metrics of performance are in % in tables unless otherwise specified.
LaSOT
LaSOT
TNL2K
GOT-IOK
OTB99-L
Type
Method
SUC
P
SUC
P
SUC
P
AO
SRo.s
SRg
SUC
SiamRCNN [5]]
64.8
68.4
52.3
52.8
64.9
72.8
59.7
70.0
89.4
PrDiMP [13]
59.8
60.8
47.0
45.9
63.4
73.8
54.3
69.5
89.5
AutoMatch [60]
58.3
59.9
37,6
43.0
47.2
43.5
65.2
76.6
54.3
71.6
93.2
Ocean [62]
56.0
56.6
38.4
37.7
72.1
47.3
68.0
KYS [6]
55.4
44.9
43.5
63.6
75.1
51.5
ATOM [12]
51.5
50.5
37.6
43.0
40.1
39.2
55.6
63.4
40.2
67.6
82.4
CNN-based
SiamRPN++ [32]
49.6
49.1
34.0
39.6
41.3
41.2
51.7
61.6
32.5
63.8
82.6
C-RPN [18]
45.5
42,5
27.5
32.0
SiamFC [S]
33.6
33.9
23.0
26.9
29.5
28.6
34.8
35.3
9.8
58.7
79.2
ECO [1]]
32.4
30.1
22.0
24.0
31.6
30.9
11.1
SiamCAR [23]
50.7
51.0
33.9
41.0
35.3
38.4
56.9
67.0
41.5
68.8
89.1
SNLT [20]
54.0
57.6
26.2
30.0
27.6
41.9
43.3
50.6
22.1
66.6
80.4
CNN-VL
VLAR (Ours)
65.2
69.1
41.2
47.5
48.3
46.6
61.4
72.4
52.3
72.7
88.8
VLT VLT& (Ours)
44.7
51.6
49.8
70.8
STARK [54]
66.4
71.2
47.8
55.1
68.0
77.7
62.3
69.6
91.4
Trans-based
TrDiMP [52]
63.9
66.3
67.1
77.7
58.3
70.5
92.5
TransT [8]
64.9
69.0
44.8
52.5
50.7
51.7
67.1
76.8
60.9
70.8
91.2
VLT (Our)
66.3
70.5
45.4
52.1
52.2
52.1
68.4
81.5
62.4
74.7
91.2
Trans-VL
VLT TT (Ours)
67.3
72.1
48.4
55.9
53.1
53.3
69.4
81.1
64.5
76.4
93.1
of 0. Given the empirical risk i (0.M). ). its corresponding population risk defined as,
(0m) = E(x...))Deraa. [ (0m))
(5)
Following [1, 48, 29], the population risk is adopted to measure the learning quality. Then the latent representation quality [29] is defined as,
m(0) inf [r (0) r((")]
(6)
where represents the optimal case, inf indicates the best achievable population risk. With the empirical Rademacher complexity Ri [4], we restate the conclusion in [29] with our definition.
Theorem ((29)). Assuming we have produced the empirical risk minimizers M and Ós, training with the [M] and [8] modalities separately (IM) |8l). Then, for all O, with probubility at
least 1 2
r (BM) -r (Bs) I SLD
4C
2ln(2/8)
+ 2C
Vn
where
TD(M,S) 2
Rr(FF) /Complexity((x))
(8)
(pM, computes the quality difference learned from multiple modalities M and single modality with dataset D. Theorem defines an upper bound of the population risk training with differ- ent number of modalities, which proves that more modalities could potentially enhance the representation quality. Furthermore, the Rademacher complexity R. (F) proportional to the network complexity, which demonstrates that heterogeneous network would theoretically rise
the upper bound of r (0x) -r (0s) and also exhibits that our asymmetrical design has larger optimization space when learning with |Ml modalities compared |S] modalities (IM) > (S)). The proof is beyond our scoop, and please refer to [29] for details.

---

Experiment
4.1 Implementation Details
We apply our method to both CNN-based SiamCAR [23] (dubbed SCTRCAR) and Transformer-bass TransT [8] (dubbed vLTrr). The matching module and localization head are inherited from the baseline tracker without any modifications.
Searching for VLT. The proposed ASS aims find more flexible modeling structure for vision- language tracking (VLT). Taking VCTSSAR as example, the supernet from SPOS [26] is used feature extractor to replace the ResNet [27] in SiamCAR. We train the trackers with supernet using training splits of COCO [36], Imagenet-VID VID [14], Imagenet-DET [14], Youtube-BB [43], GOT-10k [28], LaSOT [17] and TNL2K [53] for epochs, where each epoch contains 1.2 x 10 template-search pairs. Once finishing supernet training, evolutionary algorithms as in SPOS [26] applied search for optimal subnet and finally obtains VLTSCAR. The whole search pipeline consumes 15 hours on single RTX-2080Ti GPU. The search process of VLTrT is similar to VLTSCAR. We present more details in the appendix due to space limitation.
Optimizing VLTSCAR and VLTTT. The training protocol of VLTSCAR and VLTTT follows the corresponding baselines SiamCAR [23] and TransT [8]. Notably, for each epoch, half training pairs come from datasets without language annotations (i.e., COCO [36], Imagenet-VID [14], Imagenet- DET [14], Youtube-BB [43]). The language representation is set as O-tensor visual pooling feature under this circumstances (discussed in Sec. 4.4)*.
4.2 State-of-the-art Comparison
Tab. 2 presents the results and comparisons of our trackers with other SOTAs on LaSOT [17], LaSOT [16]. TNL2K [53], OTB99-LANG [34] and GOT-1OK [28]. The proposed VCTSCAR and VLTTT run at 43/35 FPS on single RTX-2080Ti GPU, respectively. Compared with the speeds of baseline trackers SiamCAR [23]/TransTT [8] with 52/32 FPS, the computation cost of our method is small. Moreover, VLTrT outperforms TransT in terms of both accuracy and speed.
Compared with SiamCAR [23], VLTSCAR achieves considerable SUC gains of 14.5% 11..%00...5..... on LaSOT/LaSOT BSN0T//LSSOTTs/NNNNN.N... /xe respectively, which demonstrates the effectiveness of the proposed VL tracker. Notably, our VLTSCAR outperforms the current best VI tracker SNLT [20] for 11.2% on LaSOT/LaSOT Ext, showing that the robust for VL tracking and is superior to simply fusing tracking results of different modalities. The advancement of our method is preserved across different benchmarks. What surprises us more is that the CNN-based VLTSCAR is competitive and even better than recent vision Transformer-based approaches. For example, VLTSCAR outperforms TransT [8] on LaSOT and meanwhile runs faster (43 FPS v.s. 32 FPS) and requires less training pairs (2.4 X 10" v.s. 3.8 x 10"). By applying our method to Trans', T, the new tracker VLTTT improves the baseline to 67.3% in SUC with 2.4% gains on LaSOT while being faster, showing its generality.
lision-language representation is more
4.3 Component-wise Ablation
We analyze the influence of each component in our method to show the effectiveness and rationality of the proposed ModaMixer and ASS. The ablation experiments are conducted on VLTSCAR with "O-tensor" setting (discussed in Sec. 4.4), and results are presented in Tab. 3. By directly applying the ModaMixer on the baseline SiamCAR [23] ("ResNetOO-ModaMixer")) it obtains SUC gains of 6.9% on LaSOT (@vxs.) This verifies that the unified VI. representation effectively improves tracking robustness. One interesting observation is that ASS improves vision-only baseline for 1.4% percents on LaSOT (@vss. @), but when equipping with ModaMixer, it surprisingly further brings 7.6% SUC gains (0v.s ()), which shows the complementarity of multimodal representation learning (ModaMixer) and the proposed ASS.
'GOT- 10k [28] provides simple descriptions for objecUmotioon/aaorrroo class, e.g., "dove, walking, bird, animal", in each video. We concatenate these words obtain pseudo language description.
7

---

Table Ablation on ModaMixer and asymmetrical searching strategy (ASS).
LaSOT
TN1.2K
#
Method
ModaMixer
ASS
SUC
P NNrr
P
SUC
P Norm
P
Bascline
50.7
60.0
51.0
35.3
43.6
38.4
2
VLTSCAR
57.6
65.8
61.1
41.5
49.2
43.2
VLTSCaR
52.1
59.8
50.6
40.7
47.2
40.2
VLTSCAR
65.2
74.9
69.1
48.3
55.2
46.6
Table Comparisons with two strategies (i.e., "O-tensor" and "template") and different language settings during inference.
LaSOT
TNL2K
Method
Settings
SUC
NNom
SUC
P NNomm
w/. language
65.2
74.9
69.1
48.3
55.2
46.6
O-tensor
w/o. language
50.8
57.9
52.6
39.5
47.1
41.2
Pse. language
53.1
60.4
55.0
38.1
45.7
39.6
VLTSSAR SCAR
w/. language
63.9
73.3
67.9
49.8
58.3
51.1
template
w/o. language
53,4
60.7
54.6
41.1
49.1
42.9
Pse. language
51.6
58.1
53.4
38.8
46.6
40.5
w/. language
66.3
77.0
70.5
52.2
58.6
52.1
O-tensor
w/o. language
60.7
71.1
63.1
48.2
54.6
46.8
Pse. language
59.3
68.6
62.2
49.3
55.7
49.2
VLTTT
w/. language
67.3
78.0
72.1
53.1
59.3
53.3
template
w/o. language
61.0
71.5
63.4
49.1
55.5
48.3
Pse. language
59.7
69.2
63.0
50.0
56.3
50.3
4.4 Further Analysis
Dealing with videos without language description during training. As mentioned above, language annotations not provided in several training datasets (e.g., YTB-BBox [43]). We design two strategies to handle that. One to use "O-tensor" as language embedding, and the other is to replace the language embedding with visual features which are generated by pooling template feature in the bounding box (dubbed as "template"). As shown in Tab. 2, the two strategies perform competitively, but the with visual feature (i.e., VLTS and VLT(T) is slightly better in average.
No/Pseudo description during inference. VI. trackers require the first frame of a video is annotated with language description. One may wonder that what if there is language description? Tab. presents the results by removing the description and using that generated with an recent advanced image-caption method [41] (in ICML2021) based on VLTSCAR and VLTTT with "O-tensor" and "template" settings. The results show that, without language description, tracking performances heavily degrade (e.g., 63.9% 53.4%, 67.3% + 61.0% SUC on LaSOT of VLT and VLT respectively), verifying that the high-level semantics in language do help in improving robustness. Even though, the performances the generated description, it doesn't 't show promising results (e.g., 51.6% of VLT and 59.7% of VL indicating that it still challenging generate accurate caption in real-world cases and noisy caption even brings negative effects to the model.
still better than the vision-only baseline. Surprisingly, when using
Symmetrical or Asymmetrical? The proposed asymmetrical scarching strategy is essential for achieving an adaptive vision-language representation. As illustrated in Tab. 5a, we experiment by searching for symmetrical network (including both backbone and Block in the ModaMixer) based on VLT&AR but it is inferior to the asymmetrical counterpart for 3.9% /..2% of success rate (SUC) and precision (P) LaSOT [17], respectively, which empirically proves our argument.
Asymmetry in Mixer The asymmetry used in not only the backbone network, but also the ModaMixer. In our work. the post-processing layers for different signals (visual and mixed features) are decided by ASS, which enables the adaption at both semantic levels (i.e., network depth) and different input signals (i.e., template and search, pure- vision and mixed feature in each

---

Table Evaluating different settings on LaSOT: (a) the influence of symmetrical and our asymmerri- cal design, (b) adopting fixed ShuffleNet block or searching the post- prot-proosssing block in ModaMixer, and (c) removing the residual connection (dubbed as "Res") of ModaMixer.
(b)
(c)
Settings
SUC
P
Settings
SUC
P
Settings
SUC
symmetrical
60.0
61.7
Shuffle
59.1
62.2
w/o. Res
61.1
63.6
asymmetrical
63.9
67.9
NAS-ModaMixer
63.9
67.9
w/. Res
63.9 67.9
Table 6: Comparing different data volumes and sources for training. "SiamCAR Four Datasets" consist of VID, YOUTUBEBB. DET and COCO, "SiamCAA Seven Datasets" consist of VID, YOUTUBEBB, DET, COCO, GOT-1OK, LaSOT and TNL2K, "TransT Four Datasets" consist of COCO, GOT- LaSOT and TrackingNet.
LaSOT
TNL.K
Method
Data Volume
Data Source
SUC
SUC
P
SiamCAR
60W 60Wx 20Epoc
SiamCAR Four Dalasets
50.7
51.0
35.3
38.4
SiamCAR
60W x 20Epoch
LaSOT
51.6
52.3
35.0
36.4
SiamCAR
120W 20Epohh
SiamCAR Seven Datasets
48.7
46.6
39.7
39.2
VLTSCAR
60W x2Epoch
LaSOT
57.0
58.6
39.0
39.8
VLTSCAR 120Wx20Epoch
SiamCAR Seven Datasets
63.9
67.9
48.3
46.6
TransT
3.00pohhh
TransT Four Datasets
64.9
69.0
50.7
51.7
TransT
3.8wx1000Eppe
TransT Four Datasets, TNL2K
62.2
65.2
51.2
52.3
VLTTT 3.8Wx 1000Epohhh
TransT Four Datascts, TNL2K
67.3
72.1
53.1
53.3
ModaMixer), As in Tab. 5b, when replacing the post-processing layers with fixed ShuffleNet block from SPOS [26] (i.e., inheriting structure and weights from the last block in each backbone stage), the performance of VLT SCAR drops from 63.9% to 59.1% in SUC on LaSOT. This reveals that the proposed ASS is important for building a better VL learner.
Residual Connection of ModaMixer The residual connection [27] is a commonly used trick to avoid information loss. In our VL representation learning, it provides more vision messages for better multimodal fusion. We experiment by removing the structure based on VLTSCAR with the "template" setting as shown in Tab. 5c. Compared to the default setting (i.e., w/. Res), the loss of additional vision details brings decreases for 2.7% 2.7%%1111 of SUC on LaSOT/TNL2K, respectively. Even though, the performance is still much higher than the baseline. This demonstrates the improvements are mainly attributed to the multimodal fusion.
Volume and Source of Training Data. As common wisdom, the tracking performance is deeply influenced by training data volume and source. As illustrated in Tab. 6, we experiment by comparing VLT VLT and their baselines (i.e., SiamCAR and TransT) with different data settings:
(1) We retrain SiamCAR with the same data setting of VLTSCAR (3 v.S. @). Compared to the default setting (W), double data volume and three more data sources contain different biases, which affect the trained model to produce biased outcomes, as illustrated [7, 30, 38]. From the results (3), the addition of TNL2K significantly improves the default @ with 4.4% gains in SUC on TNL2K, whereas the performance on LaSOT slightly decreases. Compared to VLTSCAR with the same setting (G), & is still suppressed for 8.6%/7.4%% of SUC and P on TNL.2K, respectively.
(2) We also retrain VLTSCAR with the only LaSOT (4), which keeps aligned with @ The SUC scores on LaSOT and TNL2K degrade heavily to 57.0%, 39.0% compared the default @, respectively. This is caused by the great reduction of the sanguage-annotale (LaSOT)+1300 (TNL2K)+9335 (GOT-10K) to 1120 (LaSOT). Our model is hard to learn good multimodal representation with the quite less intention. Even though, our VLTSCAR (Ø) still outperforms the baseline SiamCAR (0) for 5.4% 4..%%
training data, i.e., from 1120
annotated training data, which violates
/.0% of SUC scores on LaSOT/TNL2K.
(3) TransT is also retrained with the same data setting VLTTT (Φ v.s. @). More data sources bring similar biases and influence the performance as SiamCAR, compared to default TransT (6). Our VLTTT (8) still achieves superior scores both LaSOT and TNL2K.

---

Max. Score
after Fuslon
before fuslon
Min. Surr
(a) Channels visualization with max/min selection scores
(b) Activation maps
Figure 4: (a) feature channel with maximum/mminimum (top/bottomm) selection scores from ModaMixer
in 1-, (b) activation map before/after (top/bottom) multimodal fusion in ModaMixer.
Table 7: Results of applying ModaMixer and ASS to SiamRPN++.
LaSOT
TNL2K
Method
SUC
P Norm
P
SUC
Norm
P
RPN+
59.0
68.4
62.6
45.8
54.2
47.4
SiamRPN++
49.6
56.9
49.1
41.3
48.2
41.2
Channel selection by ModaMixer. ModaMixer translates the language description to a channel selector reweight visual features. As shown Fig. the channel activation maps with maximum selection scores always correspond to the target, while the surrounding distractors are successfully assigned with minimum scores (Fig. 4 (a)-bottom). Besides, with multimodal fusion channel selection), the network can enhance the response of target and meanwhile suppress the distractors (see Fig. 4 (b)). This evidences our argument that language embedding can identify semantics in visual feature channels and effectively select useful information for localizing targets. More visualization results are presented in appendix due to limited space.
Multimodal Vision-Language Tracking with SiamRPN++ [32] We apply our method to another pure CNN-based tracker PN++ (dubbed VLT and the results shown in Tab. 7. Com- pared with the baseline SiamRPN++, VLT + achieves considerable SUC gains of 9.4% on LaSOT/TNL2K, respectively. This demonstrates the effectiveness of multimodal representation learning (ModaMixer) and the proposed ASS.
Conclusion
In this work, we explore a different path achieve SOTA tracking without complex Transformer, i.e., multimodal VL tracking. The essence is unified-adaptive VL representation, learned by our ModaMixer and asymmetrical networks. experiments, our approach surprisingly boosts a pure CNN-based Siamese tracker to achieve competitive even better performances compared to recent SOTAs. Besides, we provide an theoretical explanation to evidence the effectiveness of our method. We hope that this work inspires more possibilities for future tracking beyond Transformer.
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant (62176020), the National Key Research and Development Program A0106600,))))00,, the Beijing Natural Science Foundation under Grant (Z180006, L211016), CAAl-Huawei MindSpore Open Fund and Chinese Academy of Sciences (OEIP-O-202004).
10

---

References
[1] Massih R Amini, Nicolas Usunier, and Cyril Goutte. Learning from multiple partially observed views-an application multilingual text categorization. Advances in Neural Information Processing Systems, 2009.
[2] Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. 3
[3] Stephanie Badde, Fangfang Hong. and Michael S Landy. Causal inference and the evolution of opposite neurons. Proceedings of the National Academy of Sciences, 2021. 2
[4] Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 2002.
[5] Luca Bertinetto, Jack Valmadre, João F Henriques. Andrea Vedaldi, and Philip H S Torr. Folly-convolutionall siamese networks for object tracking. In European Conference Computer Vision Workshops, 2016. 3,
[6] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Know your surroundings: Exploiting scene information for object tracking. In European Conference on Computer Vision, 2020.
[7] Haw-Shiuan Chang, Erik Learned-Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. Advances in Neural Information Processing Systems, 2017.
[8] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. 1, 2, 3, 6,
[9] Zedu Chen, Bineng Zhong, Guorong Li, Shengping Zhang, and Rongrong Ji. Siamese box adaptive network for visual tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020.
[10] Yutao Cui, Jiang Cheng, Limin Wang, and Gangshan Wu. Mixformer: End-to-end tracking with iterative mixed attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 2.3
[11] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsherg. ECO: Efficient convolution operators for tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
[12] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. ATOM: Accurate tracking by overlap maximization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2019.
[13] Martin Danelljan, Luc Van Gool, and Radu Timofte. Probabilistic regression for visual tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 6
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, 2009. 5.7
[15] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Fre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv: 18X1v:1800.04800 1810.04805, 2018. 001.. 2,
[16] Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Mingzhen Huang, Juehuan Liu, Yong Xu, et al. Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 2021. 6.
[17] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng. Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. LaSOT: A high-quality benchmark for large-scale single object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 7,8
[18] Heng Fan and Haibin I Ling. Siamese cascaded region proposal networks for real-time visual tracking. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, 2019. 3,6
[19] Qi Feng, Vitaly Ablavsky, Qinxun Bai, Guorong Li, and Stan Sclaroff. Real-time visual object tracking with natural language description. In IEEE Winter Conference on Applications of Computer Vision, 2020.
3

---

[20] Qi Feng. Vitaly Ablavsky, Qinxun Bai, and Stan Sclaroff. Siamese natural language tracker: Tracking by natural language descriptions with siamese trackers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2021. 2, 3, 4, 6,7
[21] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Conference on Empirical Methods in Natural Language Processing, 2016. 3
[22] Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng- zhong Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint 1810.0811,, 2018, 2, 4
[23] Dongyan Guo, Jun Wang, Ying Cui, Zhenhua Wang, and Shengyong Chen. SiamCAR: Siamese fully convolutional classification and regression for visual tracking. In Procedingss of the IEFE/CVF Conference
on Computer Vision and Pattern Recognition, 2020. 2, 6.7
[24] Mingzhe Guo, Zhipeng Zhang, Heng Fan, Liping Jing, Yilin Lyu, Bing Li, and Weiming Hu. Learn ing aware representation visual tracking via informative interactions. In International Joint Conference on Artificial Intelligence, 2022. 2
[25] Qing Guo, Wei Feng, Ce Zhou, Rui Huang, Liang Wan, and Song Wang. Learning dynamic siamese network for visual object tracking. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, 2017. 3
[26] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, 7echun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture scarch with uniform sampling. In European Conference Computer Vision, 2020. 3, 4, 5, 7,9
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016. 7,
[28] Lianghua Huang, Xin 7hao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2019. 6, 7
[29] Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 2021. 6,6
[30] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[31] JinHww Kim, Jachyun Jun, and Byoung-Tai Zhang. Bilinear attention networks. In Advances in Neural Information Processing Systems, 2018. 3
[32] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, 2019. 2, 3,4, 4, 6, 10
[33] Bo Li, Junjie Yan, Wei Wa, Zheng 7hu, and Xiaolin Hu. High performance visual tracking with siamese region proposal network. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[34] Zhenyang Li, Ran Tao, Efstratios Gavves, Cees GM Snoek, and Amold WM Smeulders. Tracking by natural language specification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2017. 2, 3, 6,7
[35] Liting Lin, Heng Fan, Yong Xu, and Haibin Ling. Swintrack: simple and strong baseline for transformer tracking. arXiv preprint arXiv:21 2021.
[36] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, 2014. 7
[37] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations, 2019. 3
[38] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and faimess in machine learning. ACM Computing Surveys (CSUR), 2021.
12

---

[39] Tim Meinhardt, Alexander Kirillov, Laura Taixe, and Christoph Feichtenhofer. Trackformer: Multi- object tracking with transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
[40] Juan-Manue Pérez-Rúa, Valentin Vielzeuf, Stéphane Pateux, Moez Baccouche, and Frédéric Jurie. Mfas: Multimodal fusion architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 2
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. 8
[42] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In The Association for the Advancement of Artificial Intelligence, 2019. 2,3
[43] Estehan Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent. Vanhoucke. Youtube- boundingboxes: A large high-precision human-annotated data set for object detection in video. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2017. 7,8
[44] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In JEEE Winter Conference Applicattion of Computer Vision, 2021. 4
[45] Ajit Singh. Foundations of machine learning, Available at SSRN 3399990, 2019. 5
[46] Peize Sun, Yi Jiang, Rufeng Zhang, Enze Xie, Jinkun Cao, Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple-object tracking with transformer. arXiv:2012. 15460, 2020. 3
[47] Ran Tao, Efstratios Gavves, and Arnold WM Smeulders. Siamese instance search for tracking. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2016. 3
[48] Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory transfer learning: The importance of task diversity. Advances in Neural Information Processing Syxtems, 2020.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. 3
[50] Paul Voigtlaender, Jonathon Luiten, Philip HS Torr, and Bastian Leibe. Siam r-cnn: Visual tracking by re-detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 6
[51] Naiyan Wang, Jianping Shi, Yan Yeung, and Jiaya Jia. Understanding and diagnosing visual tracking systems. In Proceedings of the IEEF/CVF International Conference on Computer Vision, 2015. 2
[52] Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li. Transformer meets tracker: Exploiting temporal context for robust visual tracking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021. 1, 1, 6,
[53] Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang. Yonghong Tian, and Feng Wu. To- wards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognittin, 2021. 6, 7
[54] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[55] Bin Yan, Houwen Peng, Kan Wu, Dong Wang, Jianlong Fu, and Huchuan Lu. Lighttrack: Finding lightweight neural networks for object tracking via one-shot architecture search. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021. 4,
[56] Bin Yang, Junjie Yan, Zhen Lei, and Stan Z Li. Convolutional channel features. In Proceedings of the
IEEE/CVF International Conference on Computer Vision, 2015. 2
[57] Yuechen Yu, Yilei Xiong. Weilin Huang, and Matthew R Scott. Deformable siamese attention networks for visual object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[58] Zhou Yu, Yuhao Cui. Jun Yu, Meng Wang, Dacheng Tao, and Qi Tian. Deep multimodal neural architecture search. In ACM International Conference Multimedia, 2020.
13

---

[59] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2018.
[60] Zhipeng Zhang. Yihao Liu, Xiao Wang, Bing Li, and Weiming Hu. Learn to match: Automatic matching network design for visual tracking. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 3,
[61] Zhipeng Zhang and Houwen Peng. Deeper and wider siamese networks for real-time visual tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 2, 3
[62] Zhipeng Zhang. Houwen Peng, Jianlong Fu. Bing Li, and Weiming Hu. Ocean: Object-aware anchor-Iree tracking. In European Conference on Computer Vision, 2020. 3,6 6
[63] Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. International Conference on Learning Representations, 2017. 2,
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Sec. 1.
(b) Did you describe the limitations of your work? [Yes] See Sec. 1.
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Sec. 3.4.
(b) Did you include complete proofs of all theoretical results? [Yes] See Sec. 3.4.
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes] See Sec. 4.1.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Sec. 4.1.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "e8pLsN2czLx", "reviewer": "Reviewer_nvbZ", "summary": "This paper proposes a tracking algorithm by designing a ModaMixer and asymmetrical networks to learn unified-adaptive VL representation. The combination of the two modules gives very good results. Applying language features to tracking tasks is a very reasonable and innovative multimodal learning algorithm.", "questions": "This paper doesn't clearly state how to get the language description of a template during inference. If the language description is from annotation or manual setting, it is not reasonable enough to compare results with algorithms that do not use language features.", "limitations": "The article does not include a discussion of solving common tracking problems such as target deformation and occlusion. The introduction of the template update module may have better results on some samples.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n•\tThe paper is clearly written.\n•\tResults greatly improve the performance and seems achieve SOTA.\n•\tThe paper has adequate ablation experiments.\n\nWeaknesses\n•\tI noticed that the results in Table 2 used different settings(0-Tensor vs Template). The author just lists the best results, the results of same algorithms should come from the exact same network structure, otherwise, they should be shown separately. For the same reason, the result of TNL2K in table 3(48.3/46.6) is different from it in Table 2(49.8/51.0).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LdZCnPe-EWs", "reviewer": "Reviewer_YvUr", "summary": "This paper proposes a method to learn the vision-language representation for achieving SOTA tracking by using pure ConvNets. It proposes a modality mixer as the fusion module and uses it in both low-level ConvStages and high-level ConvStages to explore the complementarity of different modalities. It also uses NAS to search an asymmetrical siamese network to learn adaptive vision-language representations. The proposed method is also used to improve a Transformer-based tracker.", "questions": "-The proposed trackers and the baselines should be trained with the same training set.\n-More quantitative studies about the modality mixer should be supplemented.\n-During inference, using the 0-tensor or the visual pooling feature for further analysis.\n-See weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n+The proposed method can make a CNN-based tracker and a Transformer-based tracker achieve SOTA performance on five benchmarks. The performance of vision-language tracking is improved largely.\n\n+This work uses NAS for tracking to search different model structures for template and search region respectively. This is a novel practice for siamese trackers.\n\n Weaknesses: \n-The proposed trackers and the baselines use different training data. The comparisons in this paper are unfair. For instance, both SiamCAR and TransT didn’t use TNL2K for model training, they shouldn’t be used as the baselines for comparison on TNL2K. SiamCAR only uses the LaSOT training set for model learning to achieve 50.7SUC. But the proposed trackers use lots of extra training sets. So the performance gains shown in Figure1 are debatable, and the current ablation study cannot show the effectiveness of the proposed method.\n\n-Modality Mixer is too simple to be the main contribution. It’s cross-modal channel attention. The authors attempted to make the structure more complex by designing a residual connection. But this modification brings some doubts about the effectiveness of the cross-modal channel attention. We don’t know whether the final improvements can be attributed to the modal-fusion or the pure vision feature. This paper didn’t provide any ablation analysis about it. The current experiments are not sufficient.\n\n-This paper proposes a VL representation learning method for object tracking. But half training pairs don’t have a language description, and the authors use the 0-tensor or the visual feature as the pseudo language description for training. It's doubtful that the learned representations are really multi-modal representations. The authors conducted an experiment by removing the description to prove the effectiveness of modal fusion. But how about using a 0-tensor or a visual pooling feature for inference? The current experiments are insufficient.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "-vEU8AuO3M_", "reviewer": "Reviewer_5MXU", "summary": "This paper proposes an interesting solution to show how to achieve state-of-the-art (sota) tracking performance without relying on the complex Transformer architecture in visual tracking. Specifically, the authors explore a unified multimodal learning of vision and language under the simple ConvNets for tracking. The authors first give an in-depth analysis of the limitations of previous solutions and then introduce an innovative unified framework that learns better vision-language (VL) representation by proposing ModaMixer and asymmetrical searching strategy (ASS). The former well enhances the interaction inside vision-language for discriminative representation, while the later improves the adaption of the learned VL representation by ModaMixer. Extensive experiments on five challenging datasets demonstrates that, the proposed method improves a pure CNN-based baseline to be competitive and even better than recent Transformer-based counterparts. Moreover, the proposed method is general and can also be applied to improve Transformer tracking architecture, as shown by experiments. In addition, some theoretical analysis is given to explain the effectiveness of the proposed method.", "questions": "See Weaknesses.", "limitations": "I don’t see the discussion related to the limitations and broader impact. I hope the author can provide a brief discuss on this part in the response and are encouraged to detail this part in the revision.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nOverall, I am rather positive on this paper. In particular, I really like the motivation of this work that aims at finding alternative path, instead of relying on Transformer architecture (although it is shown to be powerful), to achieve sota tracking by exploring the almost ignored multimodal learning, which I believe can inspire many other works on tracking and facilitates this field. The strengths in this work include:\n\n(1) Novelty. This paper introduces an innovative framework of multimodal vision-language learning (VL) for object tracking. The proposed ModaMixer enables a good interaction between the two modalities throughout the whole network, which is completely different from other works that only interact two modalities at the result fusion stage. In addition, another novel point is the proposed ASS that adapts the representation of different branches and modalities in an asymmetrical way. Compared with the conventional symmetrical way, ASS is the first time to show the asymmetrical architecture may be more suitable for representation learning in tracking and may provide new insights to future research. I like these novel ideas in this work.\n\n(2) Good performance. This paper demonstrates excellent performance in improving the simple pure CNN-based tracking baselines. In specific on the challenging LaSOT, the proposed approach improves the baseline from 50.7% to 65.2% in SUC with absolute 14.5% gains, which is competitive and even better than recent Transformer-based trackers. Compared to previous best VL method with 54.0% SUC, this paper shows 11.2% improvement. The performance on other benchmarks is also sota. This excellent performance clearly shows the effectiveness and advantages of the proposed method.\n\n(3) Generality. The proposed method is general and not limited to CNN-based framework. The authors show this point by applying their methods on a Transformer-based tracking framework and show promising improvements on all the five challenging benchmarks, which is consistent with the improvements shown for CNN-based framework.\n\n(4) Rich experiments and analysis. The authors provide extensive experiments and analysis for the proposed method. I appreciate this. The experimental analysis with various ablation studies allows a better understanding of each module and overall performance, and the theoretical analysis gives an explanation why the proposed method works well in improving tracking performance.\n\n(5) Good writing and organization. This paper is well written and organized. Each section has a clear motivation. It’s easy to follow the ideas. I enjoy reading the paper.\n\nOverall, I believe this paper is significant to the visual tracking community because it shows new insights and directions in designing simple but effective tracking framework with sota performance.\n\nWeaknesses\n\nAlthough this paper is technically sound and novel, I have some minor concerns or questions.\n(1) In this work, language is crucial. I noticed that in the experiment the pseudo language description generated by an image caption model does not show significant improvements. What do you think are the reasons causing this?\n(2) The proposed method is shown to be general, which is nice. However, it will be great if the authors can show more examples such as conducting more experiments on additional pure CNN-based frameworks like SiamRPN++ (Li et al, CVPR, 2019).\n(3) The authors show many comparisons with other sota methods. But I don’t see qualitative comparison with other approaches. It will be nice to see these results and comparison to other sota methods.\n(4) For experiments with partial language (50% and 75%) in the supplementary material, how do you determine the which 50% or 75% should be used? Randomly generating? If you randomly generate it, it will be better to do multiple (e.g., 3) times of experiments. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept