SwinTrack: A Simple and Strong Baseline for Transformer Tracking
Authors: Liting Lin
Heng Fan
Zhipeng Zhang
Yong Xu
Haibin Ling
Venue: NeurIPS 2022 Accept (2022)

Abstract: Recently Transformer has been largely explored in tracking and shown state-of-the-art (SOTA) performance. However, existing efforts mainly focus on fusing and enhancing features generated by convolutional neural networks (CNNs). The potential of Transformer in representation learning remains under-explored. In this paper, we aim to further unleash the power of Transformer by proposing a simple yet efficient fully-attentional tracker, dubbed SwinTrack, within classic Siamese framework. In particular, both representation learning and feature fusion in SwinTrack leverage the Transformer architecture, enabling better feature interactions for tracking than pure CNN or hybrid CNN-Transformer frameworks. Besides, to further enhance robustness, we present a novel motion token that embeds historical target trajectory to improve tracking by providing temporal context. Our motion token is lightweight with negligible computation but brings clear gains. In our thorough experiments, SwinTrack exceeds existing approaches on multiple benchmarks. Particularly, on the challenging LaSOT, SwinTrack sets a new record with 0.713 SUC score. It also achieves SOTA results on other benchmarks. We expect SwinTrack to serve as a solid baseline for Transformer tracking and facilitate future research. Our codes and results are released at https://github.com/LitingLin/SwinTrack.

SwinTrack: A Simple and Strong Baseline for Transformer Tracking
Liting Lin Heng Fan
Zhipeng Zhang*
Yong Xu Haibin Ling
School of Computer Science & Engineering, South China Univ. of Tech., Guangzhou, China
Peng Cheng Laboratory, Shenzhen, China
3 Department of Computer Science and Engineering, University of North Texas, Denton, USA
*DiDi Chuxing, Beijing, China
5 Department of Computer Science, Stony Brook University, Stony Brook, USA
l scut...ccedu en, heng. fan @uutteduu zhipeng @ @outlook.com yxu @ scut.edu.co edu.cn, hling hling@cs.stonnbrrott cs.stonybrook.edu
Abstract
Recently Transformer has been largely explored in tracking and shown state-of-the -he- art (SOTA) performance. However, existing efforts mainly focus on fusing and en- hancing features generated by convolutional neural networks (CNNs). The potential Transformer in representation learning remains under-explored. this paper, we aim to further unleash the power of Transformer by proposing simple yet efficient fully-attentional tracker, dubbed Swin TwinTracck, within classic Siamese framework. In particular, both representation learning and feature fusion in Swin leverage the Transformer architecture, enabling better feature interactions for tracking than pure CNN or hybrid ENN-Transforme frameworks. Besides, to further enhance robustness, we present a novel motion token that embeds historical target trajectory improve tracking by providing temporal context. Our motion token is lightweight with negligible computation but brings clear gains. In our thorough experiments, Trackrack exceeds existing approaches on multiple benchmarks. Particularly, on the challenging LaSOT, Swin Track sets a new record with 0.713 SUC score. It also achieves SOTA results on other benchmarks. We expect SwinTrack solid baseline for Transformer tracking and facilitate future research. Our codes and results are released at https //ggtttgtt com http:/Liinnnnn/iiiiiiinn
Introduction
Visual tracking has seen considerable progress since deep learning. In particular, the recent Trans- former [30] has significantly pushed the state-of-thhe-an in tracking owing to its ability in modeling long-range dependencies. However, existing methods usually leverage Transformer for fusing and enhancing features generated from convolutional neural networks (CNNs), e.g., ResNet [14]. The potential of exploiting Transformer for feature representation learning is largely under-explored.
Recently, Vision Transformer (ViT) [7] has exhibited great potential in robust feature representation learning. Particularly, its extension Swin Transformer [23] has achieved state-of-the -ht (SOTA) results on multiple tasks. Taking inspiration from this, we argue, besides the feature fusion, the representation learning in tracking can also benefit from Transformer via attention. Thus moti- vated, we propose to develop a fully attentional tracking framework based on Siamese architecture. Specifically, both the feature representation learning and the feature fusion of template and search region are realized by Transformer. More concretely, we borrow the architecture the powerful
Equal Contributions.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Visualization of Success Score and Speed on LaSOT
72
SwinTrack-B-33 (ours)
70%
SUC Threshold
Keep STARK
SwinTrack- T-2-4 (ours)
68
Success Score (%)
66
Simm CNN
TransT
64
DRMP super
TrDIMP
62
PrDiMP
60
AutoMatch
58
LTMU
DIMP
Ocaan
56
10
20
30
40
50
60
70
80
90
100
Speed (fps)
Figure I: Comparison LaSOT [9]. Our tracker SwwinTrack--3355 sets new record with 0.713 SUC score and still runs efficiently at around 45 fps. A lighter version T-22-22 achieves 0.672 SUC score and runs at around 96 fps, which is par with existing SOTAs in accuracy but much faster.
Swin Transformer [23] and adapt it to Siamese tracking. Note that, other Transformer architectures can be used. For feature fusion, introduce a simple homogeneous concatenation-based fusion architecture, without query-based decoder.
Moreover, taking into consideration that tracking is temporal task, we propose novel motion token to improve robustness. Inspired by that the target usually moves smoothly in a short period, motion token is represented by the historical target trajectory within a local temporal window. We incorporate the (single) motion token the decoder feature fusion to leverage motion information during tracking. Despite being conceptually simple, our motion token effectively boost tracking performance, with negligible computation.
We name our framework SwinTrack. As pure Transformer framework, SwinTrack enables better interactions inside the feature learning of template and search region and their fusion compared to pure CNN-based [1, 20] and hybrid CNN-TTansformer [5, 32, 36] frameworks, leading more robust performance (see Fig. 1). Fig. demonstrates the architecture of SwinTrack. We conduct extensive experiments
five large-scale benchmarks to verify the effectiveness of Swin Track, including LaSOT [9], LaSOText [8], TrackingNet [26], GOT- 10k [1S] and TNL2k [34]. On all benchmarks, SwinTrack achieves promising results and meanwhile runs fast at 45 fps. In particular, on the challenging LaSOT, Swin sets a new record of 71.3 SUC score, surpassing the strongest prior tracker [36] (to date) by 3.1 absolute percentage points and crossing the 0.7 SUC threshold for the first time (see Fig. again). also achieves 49.1 SUC, 84.0 SUC, 72.4 AO and 55.9 SUC scores LaSOT TrackingNet, G0T-10k and TNL2k respectively, which are better than on par with otate-of-the-arts (SoTAs). In addition, we provide a lighter version of Swin Track that obtains comparable results to SoTTAs but runs much faster around 98 fps.
In summary, our contributions are as follows: (i) We propose SwinTrack, simple and strong baseline for fully attentional tracking; (ii) We present simple yet effective motion token, enabling the integration of rich motion context during tracking, further boosting the robustness SwinTrack. with negligible computation; (iii) Our proposed Trick achieves state-of-the the-art performance multiple benchmarks. We believe SwinTrack further shows the potential of Transformer and expect to serve baseline for future research.
Related Work
Siamese Tracking. The Siamese tracking methods formulate tracking as matching problem and aim to offline leam generic matching function for this task. The seminal method of [1] introduces

---

Target Template
Faneda
weggt
SENT
SON 20R0
Result
cspy
Danodar
Modio Tokon Consination
Search Region.
Histodicll MejjctTaaaecaee
Transfomer- based Feature Representation Extraction
Transformer
Predicion
Figure 2: Architecture of SwinTrack, which contains three parts including Transformer-based feature representation extraction, Transformer-based feature fusion and prediction head. Our Swin Track simple and neat tracking framework without complex designs such multi-scale features or temporal template updating, yet demonstrating statc-of-the performance. Best viewed in color.
fully convolutional Siamese network for tracking and shows good balance between accuracy and speed. To improve Siamese tracking in handling scale variation, the work of [20] incorporates the region proposal network (RPN) [27] into the Siamese network and proposes the anchor-based tracker, showing higher accuracy with faster speed. Later, numerous extensions have been presented to improve Siamese tracking, including deeper backbone [19], multi-stage architecture [10, 11],
anchor-free Siamese trackers [41], deformable attention [37], to name a few.
Transformer in Vision. Transformer [30] originates from natural language processing (NLP) for machine translation and has been introduced to vision recently and shows great potential. The work of [3] first uses Transformer for object detection and achieved promising results. To explore the capability of Transformer in representation learning, the work of [7] applies Transformer to construct backbone network, and the resulting Vision Transformer (ViT) attains excellent performance compared to convolutional networks while requiring fewer training resources, which encourages many extensions upon VIT 4, 38, 33, 23J. Among them, the Swin Transformer [23] has received extensive attention. It proposes simple shifted window strategy to replace the fixed-patch method in VIT, which significantly improves efficiency and meanwhile demonstrates state-of-the-an results on multiple image tasks. Our work is inspired by Swin Transformer, but differently, we focus on the video task of visual tracking.
Transformer in Tracking. Inspired by the success in other fields, researchers have leveraged Transformer for tracking. The method of [5] applies Transformer to enhance and fuse features in the Siamese tracking for improvement. The approach of [32] uses Transformer to exploit temporal features to improve tracking robustness. The work of [36] introduces a new transformer architecture dedicated to visual tracking, explores the Spatio-temporal Transformer by integrating the model updating operations into Transformer module.
Our SwinTrack related but significantly different from the above Transformer-basee trackers. Specifically, the aforementioned methods mainly apply Transformer to fuse convolutional features and belong to the hybrid CNN-Transformer architecture. Unlike them, SwinTrack a pure Transformer- based tracking architecture where both representation learning and feature fusion are realized with Transformer, enabling the exploration of better features for robust tracking.
3 Tracking via Vision-Motion Transformer
We present SwinTrack, vision-motion integrated Transformer for object tracking, in Fig. 2. The proposed framework contains three main components, i.e., the Trin-Trrasfrmmer backbone for feature extraction, the encoder-decoder network mixing vision-motion cues, and the head network for localizing targets. In the following sections, we first shortly describe the backbone network, then elaborate on the proposed vision-motion encoder-decoder. Afterward, we give discussion about our method and shortly describe the network head and training loss.
3

---

3.1 Swin-Transformet for Feature Extraction
The deep convolutional neural network has significantly improved the performance of trackers. Along with the advancement of trackers, the backbone network has evolved twice: AlexNet [17] and ResNet [14]. Swin-Tranfformer [23]. in comparison ResNet, can give a more compact feature representation and richer semantic information to assist succeeding networks in better localizing the target objects (demonstrate in the ablation study demonstrated in the ablation study), which is thus chosen for basic feature extraction in our model.
Our tracker, following Siamese tracking framework [1], requires a pair of image patches as inputs, i.e. template image z E II xW,, x3 and search region image x E x x x3 As in the typical Swin- Transformer procedure, template and search region images are divided to non-overlapped patches and sent to the network, which generates template tokens (dubbed T-tokens) (o()) E R Ms Wa xC and search region tokens (dubbed S-tokens) pxx) IR xC, is the stride of the backbone network.
Since there is no dimension projection in our model, C' is the hidden dimension of the whole model.
3.2 Vision-Motion Representation Learning
The essential step for matching-based visual tracking is injecting the template information into the search region. In our framework, we adopt an encoder fuse the features from the template and the search region, meanwhile, decoder is arranged to achieve vision-motion representation learning, as illustrated in Fig.
Encoder for fusing template and search tokens. The encoder contains a sequence of Transformer blocks where each consists of multi-head self-attention (MSA) module and feed-forward network (FFN). FFN contains a two-layers multi-layer perceptron (MLP), GELJ activation layer is inserted after the first linear layer. Layer normalization (LN) is always conducted before every module (MSA and FFN). Residual connection is applied to MSA and FFN modules.
Before feeding the features into the encoder, the template and search region tokens are concatenated along spatial dimensions to generate a mixing representation fm. For each block, the MSA module computes self-attention over mixing union representation, which equals separately conducting self- self-attention on Stokens/S-tokeen and meanwhile performing cross-attentes S-tokens, but more efficient. FFN refines the features generated by MSA. When the tokens get of the encoder, a de-concatenation operation is arranged to decouple the template and search region tokens. The process of encoder be expressed
between T-tokens and
f! m
Concat(po(z), (xx))
P/ F/, (MS((LN(EE.))
(1)
pl+1 r. FFN(LN(C".)
r/ f DeConcat(F*))
where denotes the I-th layer and denotes the number of blocks.
Decoder for fusing vision and motion information. Before describing the architecture of decoder, we first detail how to generate a motion token (dubbed M-token). Motion token is the embedding of the historical trajectory of the target object. The past object trajectory is represented as set of target object box coordinates,
T [01 02 ..., ). where represents the frame index, o is the bounding box of target object. o is defined by the top-left and bottom-right corners of the target object, denotes as o = (o of "0",,,"" For flexible modeling, a sampling process is required to ensure the following properties: 1) fixed length, 2) focusing on the latest trajectories and 3) reducing redundancy. In our method, we sample object trajectory as:
T = (D.()). Ds(n)): where s(i) max (t- - - 1.
n is the number of sampled object trajectories, A is the fixed sampling interval. For Siamese tracker, the search region is cropped from the input image. In detail, cropping with resizing operation be

---

used to describe the process. Giving the point in the input image (x, the corresponding point in the search region as Yo), we can formulate the cropping process employed in pre- pre-processing (y; 1ylsy + Oy: where (Az, ky) is
of the Siamese Tracker the center of the cropping window the input image, (8x; 8y) the scaling factor, (0x, Qy) is
(x; 82 + and
the center of cropped and scaled window in the search region. We apply the same transformation on the sampled object trajectory to make the object trajectory invariant to the cropping, denoting T {1)( (b(()) .... (a(n)) J as the result.
Then, embed the transformed object trajectory into the network, we adopt four embedding matrices to embed the elements in box coordinates separately. We denotes the embedding matrix as W E ((++1)xd g controls the embedding granularity of the object trajectory, is the size of each embedding vector. The last entry of the embedding matrix used as the padding vector, indicating an invalid state, like object absence or out of the search region. Thus, we normalize the sampled target object box coordinates in the range [1, g]. and quantize integers to get the index of embedding vector:
t = (ó,(1), 02)))...,0)0) (,(n)))
where 0x(1) = [n(o)) ((1) w) n s(1y h), n6 2) A)))
(3)
1i
if valid,
n(o, ()
g +1 else,
(w, ht) is the size of search region feature map.
Finally, the motion token Emotion E RIxd is given by concatenation of all box coordinate embedding of the sampled object trajectory. FLOPs negligible because the construction of motion token is just composition of embedding lookups and token concatenation.
The decoder consists of multi-head Pross-attentionMMAA module and feed-forward (FFN). The decoder takes the outputs from the encoder and the motion token as input, generating the final vision- representation fum IR " XC of by computing aross-attention over f and Concat Concat(Ennttion f. f... The decoder is akin to a layer in the encoder, except that the correlation between the template tokens and the search tokens is dropped since do not need to update the features from the template image in the last layer. The process of the decoder is formulated as:
F/R Concat (Erotionn f. F%) P'm f + (C LN((()) fum f' FFN(LN((E,.))
fum will feed to the head network to generate a classification response map and bounding box regression map.
Positional encoding, Transformer requires a positional encoding to identify the position of the current processing token[30] because the self-attention module is coamutation-nvvaiianne We adopt the untied positional encoding [16] as our positional encoding method. The untied positional encoding enhances the expressiveness of the model through untie the positional embeddings from token embeddings with an isolated positional embedding matrix. It also considers the case of special tokens, like the motion token in this paper. We generalize the untied positional encoding to multi-dimensions multi-sources data to comply with concatenated- based fusion in tracker. See the appendix for the details.
3.3 Discussion
Why concatenated attention? To simplify the description, we call the method described above concatenation--uuvel fusion. To fuse and process features from multiple sources, it is intuitive to perform self-attention on the feature from each source separately and then compute cross-attention across features from different sources. We call this method cross-attention-husse fusion. Transformer makes fewer assumptions about the spatial structure of data, which provides great modeling flexibility.
5

---

In comparison to pross-attention-Dee -assed fusion, concatenation-bassee fusion can save computation cost through operation sharing and reduce model parameters through weight sharing. From the perspective of metric learning, weight sharing is an essential design ensure the metric between two branches of data symmetric. Through concatenation- -ased fusion, we implement this property not only in the feature extraction stage but also in the feature fusion stage. In general, concatenation- based fusion improves both efficiency and performance.
Why not window-based self/cross- atttetttssttttntion? Since we select stage 3 of the Swin-Trann Transformer as the output, the number of tokens involved is significantly reduced, window-based attention cannot save too many FLOPs. Furthermore, considering the extra latency introduced by the window partition and window reverse operations, window-base attention may even be the slower one.
Why not a query-based decoder? Derivated from vanilla Transformer decoder, many transformer- based models in vision tasks leverage learnable query to extract the desired objective features from the encoder, like object queries in [3], target query in [36]. However, in our experiment, query-based decoder suffers from slow convergence and inferior performance. Most Siamese trackers [20, 35, 13] formulate tracking as background information. The vanilla Transformer decoder generative model, the generative approaches are considered not suitable for the classification tasks. In another aspect, learning a general target query for any kind of object might cause a bottleneck. In terms of vanilla Transformer incoder-decode, architecture, SwinTrack is an "encoder" only model. Furthermore, quite little domain knowledge can be easily applied on a classic Siamese tracker to improve the performance, like introducing the smooth movement assumption by using Hanning penalty window on the response map.
classification problem, which can better exploit the
Are other forms of motion token feasible? Other forms to construct motion token are possible, such as constructing motion token by summing up the past box coordinate embeddings or representing past object trajectories by one token per box. In our early experiments, we find that the proposed motion token is more effective with the best performance. Summing up the past box coordinate embeddings may result in over parameterization on the coordinate embeddings. While adding temporal motion representation along with visual features to the single-layer decoder a multi- tokei form is ineffective, precise temporal modeling may be required in this form.
3.4 Head and Loss
Head. The head network is split into two branches: classification and bounding box regression. Each of them is a three-layer perceptron. And both of them receives the feature map from the decoder as input to predict the classification response map Tets E (H x and bounding box regression map Treg E (H x xx) respectively.
Classification loss. In classification branch, we employ the IoU-aware classification score as the training target and the varifocal loss [39] as the training loss function. IoU-aware design has been very popular recently, but most works consider IoU prediction as an auxiliary branch to assist classification or bounding box regression [41, 35]. To remove the gap between different prediction branches, [39] and [21] replace the hard classification target from the ground -trth value, (i.c., I for positive samples, 0 for negative samples), to the IoU between the predicted bounding box and the ground- one, which is named the loU-aware classification score (IACS). IACS can help the model select more accurate bounding box prediction candidate from the pool by trying to predict the quality the bounding box prediction in another branch at the same position. Along with the IACS, the varifocal loss was proposed in [39] to help the IACS approach outperform other IoU-aware designs.
The classification loss can be formulated as:
Lets = LVFL LoU B)),
(5)
where p denotes the predicted LACS, b denotes the predicted bounding box, and b denotes the ground-truth bounding box.
Regression loss. For bounding box regression, we employ the generalized IoU loss[28). The regression loss function can be formulated as:
Lreg Σ L (100
(b, (1o
(6)

---

The GloU loss is weighted by p to emphasize the high classification score samples. The training signals from the negative samples are ignored.
4 Experiments
4.1
Implementation
Model. We design two variants of SwinTrack with different configurations as follows:
SwinTrack-T-22 T-224.
Backbone: Swin Transformer-Tiny [23], pretrained with ImageNet-Ik:
Template size: [112 x 112]: Search region size: [224 x 224]; C' = 384; N = 4;
SwinTrack-B..
Backbone: Swin Transformer- Base [23], pretrained with ImageNet-22k:
Template size: [192 x 192); Search region size: [384 x 384]; C 512; N 8;
where C' and N are the channel number of the hidden layers in the first stage of Swin Transformer and the number of encoder blocks in feature fusion, respectively. In all variants, use the output after the third stage of Swin Transformer for feature extraction. Thus, the backbone stride is 16.
For motion token, the number of sampled object trajectory n is set 16, the fixed sampling interval is set to 15. If the frame rate of the video sequence is available, the sampling interval is adjusted according to the frame rate. Suppose the frame rate is f. the new sampling interval is getting by f. 30 fps is the standard frame rate we assumed. g, which controls the embedding granularity, set the same size the search region feature map, like 14 for SwinTrack-T--2 T-224,, and 24 for WwinTrack-B-35
For the model for GOT-10k sequences, n is set to 8, A is set to 8, and no frame rate adjustment is applied.
Training. We train SwinTrack using the training splits LaSOT [9]. Tracking [26], GOT- [15]
(1,000 videos are removed following [36] for fair comparison) and COCO 2017 [22]. In addition, we report the results of SwinTrack- Track-T-224 and Swin Track-B-338 with GOT- 10k training split only to follow the protocol described in [15].
The model is optimized with AdamW [24], with a learning rate of 5c-4, and a weight decay of le-4. The learning rate of the backbone is set to Se-5. We train the network on 8 NVIDIA V100 GPUs for 300 epochs with 131,072 samples per epoch. The learning rate dropped by factor of 10 after 210 epochs. A 3-epoch linear warmup is applied to stabilize the training process. DropPath [18] is applied on the backbone and the encoder with a rate 0.1. For the models trained for the GOT- 10k evaluation protocol, to prevent over-fitting the training epoch is set to 150, and the learning rate is dropped after 120 epochs.
For the motion token, the object trajectory for the Siamese training pair is generated with the method described above. The frames that object annotated as absent or out of the video sequence are marked as invalid, the corresponding box coordinates set to -oo. Since the coarse granularity of the coordinate embedding our setting already can be seen as an augmentation of historical object trajectory, no additional data augmentation is applied.
Inference. We follow the common procedures for Siamese network-based tracking [1]. The template image is cropped from the first frame of the video sequence. The target object is in the center of the image with a background area factor of 2. The search region is cropped from the current tracking frame, and the image center is the target center position predicted in the previous frame. The background area factor for the search region is 4.
Our SwinTrack takes the template image and search region as inputs and output classification map Tels and regression map Tegg To utilize positional prior in tracking, we apply hanning window penalty on Tela, and the final classification map I' is obtained via r'ds = (1 - 7) x Tels + x h, where is the weight parameter and is the Hanning window with the same size Tets- The target position is determined by the largest value in r'd and the scale is estimated based on the corresponding regression results in Freg-
For the motion token, the predicted confidence score and bounding box are collected on the fly. A confidence threshold o @conf is applied, if the confidence score given by the classification branch of the
7

---

Table Experiments and comparisons on five benchmarks: LaSOT, LaSOT
TrackingNet, GOT-
10k and TNL2k.
LaSOT
LaSOT ca [8]
TrackingNet [26]
GOT-TOK [15]
TNL2k
[34]
Tracker
SUC
SUC
SUC
P
AO
SRo.5
SRo.75
SUC
C-RPN [T0]
45.5
44.3
27.5
32.0
66.9
61.9
SiamPRN++ [19]
49.6
49.1
34.0
39.6
73.3
69.4
51.7
61.6
32.5
41.3
41.2
Ocean [41]
56.0
56.6
61.1
72.1
47.3
38.4
37.7
DiMP [2]
56.9
56.7
39.2
45.1
74.0
68.7
61.1
71.7
49.2
44.7
43.4
LTMU [6]
57.2
57.2
41.4
47.3
48.5
47.3
SiamR-CNe [31]
64.8
81.2
80.0
64.9
72.8
59.7
52.3
52.8
STMTrack [12]
60.6
63.3
76.7
64.2
73.7
57.5
AutoMatch [40]
58.3
59.9
37.6
43.0
76.0
72.6
65.2
76.6
54.3
TrDiMP [32]
63.9
61.4
78.4
73.1
67.1
77.7
58.3
TransT [5]
64.9
69.0
80.3
67.1
76.8
60.9
51.0
STARK [36]
67.1
82.0
68.8
78.1
64.1
KrepTrack [25]
67.1
70.2
48.2
SwinTrack--T-22
67.2
70.8
47.6
53.9
81.1
78.4
71.3
81.9
64.5
53.0
53.2
SwinTrack-B-384 B-384
71,3
76.5
49.1
55.6
84.0
82.8
72.4
80.5
67.8
55.9
57.1
head is lower than the threshold, the target object in the current frame is marked as lost by setting the collected bounding box to -oo. Pon is set to 0.4 for LaSOT, the rests are set to 0.3.
4.2 Comparisons to State-of-the-arts
We conduct experiments and compare SwinTrack with SoTA trackers on five benchmarks.
LaSOT. LaSOT [9] consists of 280 videos for test. Tab. 1 shows the results and comparisons with SoTAs. From Tab. 1, can observe that SwinTrack--T222 with light architecture reaches SoTA performance with 0.672 SUC and 0.708 PRE scores, which is competitive compared with other Transformer-based trackers, including STARK-ST101 (0.671 SUC score) and TransT (0.649 SUC), and other trackers using complicated designs such as KeepTrack (0.671 SUC) and SiamR-CNG (0.648 SUC score). With larger backbone and input size, our strongest variant SwinTrack-B-r --334 sets a new record with 0.713 SUC score, surpassing START-STI01 and KeepTrack by 4.2 absolute percentage points.
LaSOText The recent LaSOT [8] is an extension of LaSOT by adding 150 extra videos. These new sequences are challenging many similar distractors cause difficulties for tracking. The results of our tracker related to this dataset are an average of three times. KeepTrack uses a complex association technique handle distractors and achieves promising 0.482 SUC score in Tab. 1. Compared with complicated KeepTrack, Swin'rrck- T-224 is simple and neat, yet shows comparable performance with 0.476 SUC score. In addition, due complicated design, KeepTrack runs less than 20 fps, while SwinTrack--T222 runs in 98 fps, 5x faster than KeepTrack. When using larger model, SwwnTTrac--B-- shows the best performance with 0.491 SUC score.
TrackingNet. We evaluate different trackers on the test set of TrackingNet [26]. From Tab. 1, we observe that model and input size, SwinTrack- B-384 obtains the best performance with 0.840 SUC score, better than STARK-STI01 with 0.820 SUC score and TransT with 0.814 SUC score.
achieves comparable result with 0.81 SUC score. Using larger
GOT-10k. GOT-10k [15] offers 180 videos for test and it requires trackers be trained using B-384 achieves the best mAO of 0.724,
GOT-10k train split only. From Tab. 1, we see that and Track--24- obtains a mAO of 0.713. Both models outperform other Transformer- based counterparts significantly, including START-STI01 (0.688 mAO), TransT (0.671 mAO) and TrDiMP (0.671 mAO).
TNL2k. TNL2k [34] is a newly released tracking dataset with 700 videos for test. As reported in Tab. 1, both models surpass the others. SwinTrack-B -racck----384 set a new state-of-the-art with 0.559 SUC score.
Multiply "Multppy-accumular accumulate operation

---

Table 2: Comparison on running speed and # parameters with other Transformer based trackers.
Tracker
Speed (fps)
MACs (G)
Params (M)
TrDiMP [32]
26
TransT [5]
50
23
STARK-STSO [36]
42
10.9
24
STARK-STI01 [36]
32
18.5
42
SwinTrack--1222
98
6.4
23
SwinTrack-B-in
45
69.7
91
Table Ablation experiments of SwinTrack on four benchmarks. The experiments are conducted on Twackrack-T-22 without the motion token. O: baseline method, i.e., SwinTrack-T-224 without motion token; @: replacing Transformer backbone the baseline method with ResNet-50; 0: replacing our feature fusion with cross attention based fusion in the baseline method; Q replacing the decoder in baseline with a target query-based; 6 replacing united positional encoding with
absolute sine position encoding in the baseline method; @: replacing the loU-aware classification loss
with the plain binary cross entropy loss; removing the Hanning penalty window in the baseline method inference.
LaSOT SUC (%)
LaSOT ext SUC (%)
TrackingNet SUC (%)
GOT-1OK'
Speed fps
Params M
mAO (%)
66.7
46.9
80.8
70.9
98
22.7
64.2
41.8
79.5
68.2
121
20.0
66.6
45.4
80.2
69.3
72
34.6
66.6
43.2
79.6
69.0
91
25.3
65.7
45.0
80.0
70.0
103
21.6
66.2
46.7
79.4
68.2
98
22.7
65.7
46.0
80.0
69.6
98
22.7
Efficiency comparison. We report the comparisons of SwinTrack with other Transformer-based trackers in terms of efficiency and complexity. As displayed Tab. 2, SwinTrack-T-222. with small model runs the fastest with a speed of 98 fps. Especially, compared with STARK-ST101 and STARK-STT with 32 fps and 42 fps, SwinTrack- T-224 is 3x and 2x faster. Despite using larger model, our SwinTrack-B-355 is still faster than STARK-STI01 and STARK-ST50.
4.3 Ablation Experiment
Comparison with ResNet backbone. We compare the Trinn-rranfoomer backbone with popular ResNet-50 [14]. As shown in Tab. (0 vs. @). The Swin Transformer backbone significantly boosts the performance by 2.5% SUC score in LaSOT, 5. % SUC score in LaSOT The result shows that the strong appearance modeling capability provided by the Swin Transformer plays crucial role.
compared with the concatenation saasaaatioo---eeddddd fusion, the cross attention-base based fusion runs at slower speed, occupies much more memory, and also has an inferior performance on all datasets. Slower speed can be due the latency brought by the extra operations. The parameter-sharing strategy not only just reduces the number of parameters but also benefits metric learning.
Feature fusion. As displayed Tab.
v.s.
Comparison with the query-based decoder. Queries is commonly adopted in the decoder of Transformer network in vision tasks, e.g. object query [3] and target query [36]. Nevertheless, our empirical results in Tab. (O vs. ①) show that a target query-based decoder degrades the tracking performance all benchmarks, generative model is not suitable for classification. Besides, learning general target query for any kind of object may also be difficult.
with 2x training pairs. As discussed, one possible reason is the
Position encoding. We compare the united positional encoding used in SwinTrack and the original absolute position encoding in Transformer [30]. Notice, We make little modification to the original absolute position encoding: Except for the 2D embedding, the index of token source (e.g. for the
TThe GOT- 10k results in this column trained with full training datasets.

---

Table 4: Ablation experiments on our proposed motion token on the tracking performance on four benchmarks. The experiments are conducted on T-22 O: SwinTrack-T-22 0
B3: SwinTrack-T-224 without motion token; O: SwinTrack---366 without motion
token; @: replacing the motion token in SwinTrack-
with a learnable embedding token.
LaSOT
LaSOT
TrackingNet
GOT-1Ok
Speed
SUC (%)
SUC (%)
SUC (%)
mAO (%)
fps
67.2
47.6
81.1
71.3
96
71.3
49.1
84.0
72.4
45
66.7
47.0
80.8
70.0
98
70.2
48.5
84.0
70.7
45
66.3
45.2
81.2
70.0
96
tokens from the template patch, 2 for the tokens from the search region patch) is also embedded. As shown in Tab. 3 vs. 3), our method with united positional encoding obtains improvements with 0.8-1.9 absolute percentage points on the benchmarks with negligible loss in speed (98 vs. 103).
b), observe that the model trained with varifocal loss significantly outperforms the one with binary cross entropy (BCE) loss without loss of efficiency. This result indicates that the varifocal loss can assist the classification branch of the head generate an loU-aware response map, and thus help the tracker to improve the tracking performance.
Loss function. From Tab. 3
Post processing. One may wonder with highly discriminative Transformer architecture and IoU- aware classification loss does the hanning penalty window is still functional, which introduces a strong smooth movement assumption. In the experiments, post-processing, as shown Tab. (O vs. @), the performance dropped by 1.0 SUC for LaSOT, 1.3 AO for GOT-10k in absolute percentage, and less than 1% in the SUC metric of other datasets. This suggests that the strong smooth movement assumption is still applicable for our tracker. But compared with the former Transformer-batt based tracker [5], the performance gap between with and without penalty window post-processing is narrowing.
remove the hanning penalty window in
Effectiveness of motion token. We study the effectiveness of the motion token by conducting comparison experiments. As shown in Tab. (0 vs. and @ vs. 9), the models with motion token outperforms the models without motion token on all datasets, especially
LaSOT ext and GOT-10k. The results indicate that the motion token can assist the tracker to handle hard similar distractors in LaSOT and stabilize the short-term tracking like the sequences in GOT-10K test set. We also study whether the effectiveness of the motion token is simply from the extra embedding vector. We set up an experiment as in Tab. (O), which replaces the motion token with learnable embedding token. The result shows that the extra embedding vector has negative impacts indicating the effectiveness of the embedding of object trajectory.
Conclusion
In this work, we present SwinTrack, a simple and strong baseline for Transformer tracking. In SwinTrack, both representation learning and feature fusion are implemented with the attention mechanism. Extensive experiments demonstrate the effectiveness of such architecture. Besides, we propose the motion token to enhance the robustness of the tracker by providing the historical object trajectory, showing the flexibility of the Transformer model in architectural design. With the power of sequence-to sereencett--seuenee model architecture, context-rich tracker is possible, and more contextual cues can be incorporated. Finally, We hope this work can inspire and facilitate future research.
Acknowledgments and Disclosure of Funding
This work is supported by Peng Cheng Laboratory Research Project No. PCL2021 A07. Heng Fan and his employer receive no financial support for the research, authorship, and/or publication of this article.
10

---

References
Bertinetto, L., Valmadre. J., Henriques. J.F., Vedaldi, A., Tomr P.H., 2016. Fully-convolutional siamese networks for object tracking. ECCVW.
Bhat, G., Danelljan, M., Gool, L.V., Timofte, R., 2019. Learning discriminative model prediction for tracking, in: ICCV.
[3] Carion, N., Massa, F.. Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S., 2020. End-to-end object detection with transformers, in: ECCV.
[4] Chen, C.F.R., Fan, Q. Panda, R., 2021a. Crossvit: Cross- coss-attention multi-scale vision transformer for image classification, in: ICCV.
[5] Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., Lu, H., 2021b. Transformer tracking, in: CVPR.
[6] Dai, K., Zhang. Y., Wang. Di, Lu, H., Yang, X., 2020. fiigh-performnnce long-term tracking with meta uppate,, in: CVPR.
[7] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani. M., Minderer, M., Heigold, G., Gelly, S., et al., 2021. An image is worth 16x16 words: Transformers for image recognition scale, in: ICLR.
[8] Fan, H., Bai, H., Lin, L., Yang. F., Chu, P., Deng, G., Yu, S., Huang, M., Liu, J., Xu, Y., al., 2021. Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision 129, 439-461.
[9] Fan, H., Lin, L., Yang, F., Chu, P., Deng, G., Yu, S., Bai, H. Xu, Liao, C., Ling, H., 2019, Lasot: high-quality benchmark for large-scale single object tracking, in: CVPR.
[10] Fan, H., Ling, H., 2019. Siamese cascaded region proposal networks real-time visual tracking, in: CVPR.
[11] Fan, H., Ling, H., 2021. Cract: Cascaded aegressioonalignnelaaffcccoooo for robust visual tracking, in: IROS.
[12] Fu, Z., Liu, Q., Fu, Z., Wang, Y., 2021. Stmtrack: Template-free visual tracking with space-time memory networks, in: CVPR.
[13] Han, W., Dong, X., Khan, F.S., Shao, L., Shen, J., 2021. Learning to fuse asymmetric feature maps in siamese trackers, in: CVPR.
[14] He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition, in: CVPR.
[15] Huang, L., Zhao, X., Huang, K., 2019. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 1562-1577.
[16] Ke, G., He, D.. Liu. T.Y., 2021. Rethinking positional encoding language pre-training, in: ICLR.
[17] Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012. Imagenet classification with deep convolutional neural networks. NIPS.
[18] Larsson, G., Maire, M., Shakhnarovich, G., 2016. Fractalnet: Ultra-deep neural networks without residuals, ICLR.
[19] B.,
Wang, Q., Zhang, F., Xing, J., Yan, J.S., 2019, Evolution siamese visual tracking with very deep networks, in: CVPR.
[20] B., Yan. J.. W. Zhu, Hu, X., 2018. High performance visual tracking with siamese region proposal network. in: CVPR.
[21] Li, X., Wang, W., Wu, L., Chen, S., Hu, X., Li, J., Tang, J., Yang. J., 2020. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection, in: NeurIPS.
[22] Lin, T.Y., Maire, M., Belongie, S., Hays, J.. Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.,, 2014. Microsoft coco: Common objects in context, in: BCCV.
[23] Liu, Z., Lin, Y., Cao, Y. Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B., 2021. Swin transformer: Hierarchical vision transformer using shifted windows. ICCV

---

[24] Loshchilov, I., Hutter, F., 2019. Decoupled weight decay regularization. in: ICLR.
[25] Mayer, C., Danelljan, M., Paudel, D.P., Van Gool, L.,, 2021. Learning target candidate association to keep track of what not track, in: ICCV.
[26] Muller, M., Bibi, A., Giancola, S., Alsubaihi, S., Ghanem, B., 2018. Trackingnet: A large-scale dataset and benchmark for object tracking in the wild, in: ECCV.
[27] Ren, S., Hc, K., Girshick, R., Sun, 2015. Faster r-cnn: Towards real-time object detection with region proposal networks, in: NIPS.
[28] Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I,, Savarese, S., 2019. Generalized intersection over union.
[29] Touvron, H., Cord, M., Douze, M., Massa, F,, Sablayrolles, A., Jégou, H., 2021. Training data-efficient image transformers & distillation through attention, in: ICML.
[30] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A.N., Kaiser, F.,, Polosukhin, I., 2017. Attention is all you need, in: NeurIPS.
[31] Voigtlaender, P., Luiten, J., Torr, P.H., Leibe, B., 2020. Siam r-cmm: Visual tracking by re-detection, in: CVPR.
[32] Wang, N., Zhou, W., Wang, J., Li, H., 2021a. Transformer meets tracker: Exploiting temporal context for robust visual tracking, in: CVPR.
[33] Wang, W., Xie, E., Li, X., Fan, D.P. Song. K., Liang. D., Lu, T., Luo, P., Shao, L., 2021b. Pyramid vision transforeer: A versatile backbone for dense prediction without convolutions, in: ICCV.
[34] Wang, X., Shu, X., Zhang, Z., Jiang, B., Wang, Y., Tian, Y., Wu, F. 2021c. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark, in: CVPR.
[35] Xu, Y., Wang, Z., Li. Z., Yuan, Y. Yu, G., 2020. Siamfe++: Towards robust and accurate visual tracking with target estimation guidelines, in: AAAL
[36] Yan, Peng, H., Fu, J., Wang, D., Lu, H., 2021. Learning spatio-temporal transformer for visual tracking, in: ICCV.
[37] Yu. Y., Xiong. Y., Huang, W., Scott, M.R., 2020. Deformable siamese attention networks for visual object tracking, in: CVPR.
[38] Yuan, L., Chen, Y., Wang, T., Yu, W. Shi, Y., Jiang, Z.H., Tay, F.E., Feng, J., Yan, S., 2021. Tokens-to to-token vit: Training vision transformers from scratch imagenet, in: ICCV.
[39] Zhang, H. Wang, Y., Dayoub, F,, Sünderhauf, N., 2021a. in: CVPR.
Varifocalnet: An iou-aware dense object detector,
[40] Zhang, Z., Liu. Y., Wang, X., Li, B., Hu, W. 2021b, Learn to match: Automatic matching network design for visual tracking, in: ICCV.
[41] Zhang, Z., Peng, H., Fu, J. Li, B., Hu, W., 2020. Ocean: Object-aware anchor-free tracking, in: ECCV.
12


Reviews:
[{"review_id": "lh9DUjUPyD", "reviewer": "Reviewer_eHVZ", "summary": "This paper proposes a fully-attentional tracker where both the representation learning and the feature fusion modules are based on the Transformer. Besides, a motion token is introduced to embed the historical target trajectory, further improving the tracking performance. Extensive experiments and ablation study on several large-scale tracking benchmarks are performed to demonstrate the effectiveness and efficiency of the proposed approach.", "questions": "Please explain the aforementioned two weakness issues.", "limitations": "N/A", "rating": 9, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "For the strengths, 1) Compared with recent Transformer-based methods that use the Transformer to perform the feature fusion, the proposed tracking framework takes advantage of the Transformer on both the representation and feature fusion modules, which is more thorough and more reasonable.\n2) The design of introducing the motion token is novel and effective for tracking.\n3) Experiments, especially the ablation study are sufficient, the effectiveness of each designed module is proved.\n\nFor the weaknesses, 1) For the motion token, the template-search region pair is based on the single frame, while the object trajectory is sequential. To make the trajectory invariant to the cropping, do all bounding boxes in one trajectory perform the same transformation, i.e., based on the cropping of the current frame? This point is somewhat confusing. \n2) The fixed sampling interval of the trajectory is set to 15 and 8 for different datasets, is it the same setting for the training and inference phase? Besides, will it make a significant performance fluctuation when increasing or decreasing this value? I think the setting of this value needs to be explained. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "bRs70MHxi4s", "reviewer": "Reviewer_6BWX", "summary": "The paper addresses the problem of general object tracking, where the target object is given during test time. The authors propose a simple but strong baseline approach based on a Swin Transformer image encoder and a standard transformer decoder. Further, the paper introduces a motion token, which encodes past bounding boxes in order to improve the final prediction. The authors also revise the losses employed to train the tracker. Very strong results are reported on 5 popular benchmarks.", "questions": "See weaknesses.", "limitations": "I could not find discussion of negative social impact or limitations. It would be good to add these.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Before dwelling into the strengths and weaknesses, I want to discuss the novelty aspect of this paper. On one hand, the novelty is not remarkable: the authors are adopting the Swin backbone, while concatenation fusion is also used in Stark. However, the novelty is not limited to these aspects. On the contrary, I believe that this paper brings substantial value to the tracking field by consolidating existing techniques, while investigating important details in order to achieve a simple, yet highly powerful tracking framework. Importantly the authors provide valuable insights when motivating their approach and comparing to other techniques (modifications of fusion, transformer architecture, losses, etc.). Lastly, I find the motion token a very interesting novelty that could reopen a long-forgotten direction in tracking, namely exploiting motion prediction and other dynamic information. Although seemingly incremental at the first glance, I consider the novelty to be significant based on how much this paper advances the useful knowledge in the field.\n\nOther strong points of the paper are:\n\n• Very strong results, clearly SOTA.\n\n•\tSimple and elegant architecture.\n\n•\tInsightful discussions.\n\n•\tMethod and motivation are easy to follow.\n\n•\tInteresting ablative experiments on multiple datasets.\n\n•\tRelatively fast frame-rates.\n\n\nWeaknesses:\n\n1. Details regarding pre-training are missing. I assume that the authors use ImageNet-22k pre-training, while most trackers use ImageNet-1k. This has shown to give about 2-3% on LaSOT in previous papers. The authors should therefore analyze this in a separate experiment.\n\n2. By looking into the more detailed results in the supplementary material, it seems that the improvements mostly stem from increased accuracy. That is, better bounding box regression. The robustness (low overlap scores in the success plot) seems to be on par with recent trackers. While accuracy is also important, the major challenge in tracking is to improve the robustness. It is important to discuss these aspects in the paper in order to understand where and how SwinTrack performs better compared to other trackers. Moreover, please add more high-performing trackers to the success plots in the supplementary material.\n\n3. Unfortunately, the design of the motion token is motivated. For instance, why are the past box encodings concatenated in the channel dimension, and not processed in some other manner? Why add it as a single token in the transformer, instead of one per box?\n\n4. Results on UAV and VOT should be added, even if the tracker does not beat SOTA.\n\n5. There are quite a few language mistakes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QPIfsoTgeqm", "reviewer": "Reviewer_ZRLJ", "summary": "This paper presents a simple yet strong baseline tracker, named SwinTrack, for single object tracking (SOT). SwinTrack adopts the classic siamese structure, and uses Swin Transformer for both feature extraction and fusion of template and search region. A special motion token which encodes the past trajectory is introduced to further boost the tracking performance. SwinTrack is evaluated on major SOT benchmarks and achieves SOTA performance. ", "questions": "One main contribution of SwinTrack is to use Transformer network not only for feature fusion but for feature extraction. This is quite obvious since Transformer-based vision feature has shown its advantages over CNN-based feature in numerous scenarios and downstream tasks. One question I have been wondering is  why previous Transformer-based trackers adopt CNN-Transformer hybrid architecture? Are there any obstacles in using Transformer for feature extraction? Is there any secret weapon that the authors use to make it work?", "limitations": "This reviewer does not see potential negative societal impact of this work. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Single object tracking (SOT) is an important problem in computer vision and for video understanding in particular. The proposed method is technically sound and achieves SOTA performance on major SOT benchmarks. Although most components in SwinTrack are existing ones and cannot be said novel, I appreciate the system work which puts them together in a proper way and eventually achieves good results. In particular, the design choices to use concatenation-based fusion over cross-attention-based fusion and not to use query-based decoder are reasonable and well-justified by ablation studies. \nThe motion token design is new, based on my knowledge. According to the ablation, it achieves non-negligible gain, making it a good added component to the tracker. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "G-DWsV-PQhM", "reviewer": "Reviewer_iVQV", "summary": "The authors propose an interesting Siamese architecture for single object tracking based on the Transformer model. The encoder is divided into the two feature transforms and a transformer encoding step where correlations of spatiotemporal features on the concatenated branches are made. Instead of CNNs the authors suggest SwinTransformers as branches with shared weights. In the encoding part of the Transformer motion information is added to the stream of features by assuming smooth motion of the object and by maintaining a limited history of the BBs of the object. BBs are regressed with the output embeddings. The experiments show state-of-the-art results on large-scale tracking datasets.", "questions": "Could you explain in more detail why it is beneficial to neglect learnable queries in the model design?\nWhy did you not test your method on the VOT challenges? They give experimental results for the different challenges / attributes of tracking.\nHow would your method work when an object abruptly changes the direction of motion as you assume motion consistency?", "limitations": "No.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea to use SwinTransformer instead of a CNN in the branches seems to be novel and also seems to be fruitful at least in terms of algorithmic efficiency and surpassing the 70% limit in LASOT (Fig. 1). Concatenating the features of the branches and then applying a Transformer like attentional mechanism to extract embeddings for which BBs can be regressed is then straightforward [5, 36]. The use of non-learnable queries might be a novel change in the design and beneficial in terms of interpreting the tracking problem as classification problem.\nMotion consistency restricts the method to track objects properly e.g. when it comes to abrupt motions. Although the common experimental procedure on standard datasets including ablation studies has been conducted, the experiments show little insight into the effects of assumptions and design decisions. While overall benchmark scores and leave-one-out results give an answer to the usefulness of (parts of) the method compared to others, the methodology gives very little insight into the challenges and the potential causes of low performance, e.g. in case of rapid motion, occlusion, illumination changes, etc. I very much miss a deeper qualitative analysis of the model and understanding of its behaviour when it comes to the challenges of tracking.\nThe paper is well structured, however I recommend to proof-read the document as there are typo errors in the text, e.g. line 120. The English needs to be improved, e.g. line 163: “chopped off” sounds flappy. Technically the paper needs also to be improved, e.g. line 145, for n=0 T_traj is not defined properly, for n=1 T_traj = {o_c-delta, o_c, o_c-delta} which makes no sense.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept