MixFormerV2: Efficient Fully Transformer Tracking
Authors: Yutao Cui
Tianhui Song
Gangshan Wu
Limin Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Efficient Tracking
Fully Transformer
Distillation
Model Pruning
TL;DR: We propose an efficient fully transformer tracking framework MixFormerV2 and present a distillation-based model reduction paradigm for MixFormerV2.

Abstract: Transformer-based trackers have achieved strong accuracy on the standard benchmarks. However, their efficiency remains an obstacle to practical deployment on both GPU and CPU platforms. In this paper, to overcome this issue, we propose a fully transformer tracking framework, coined as \emph{MixFormerV2}, without any dense convolutional operation and complex score prediction module. Our key design is to introduce four special prediction tokens and concatenate them with the tokens from target template and search areas. Then, we apply the unified transformer backbone on these mixed token sequence. These prediction tokens are able to capture the complex correlation between target template and search area via mixed attentions. Based on them, we can easily predict the tracking box and estimate its confidence score through simple MLP heads. To further improve the efficiency of MixFormerV2, we present a new distillation-based model reduction paradigm, including dense-to-sparse distillation and deep-to-shallow distillation. The former one aims to transfer knowledge from the dense-head based MixViT to our fully transformer tracker, while the latter one is used to prune some layers of the backbone. We instantiate two types of MixForemrV2, where the MixFormerV2-B achieves an AUC of 70.6\% on LaSOT and AUC of 56.7\% on TNL2k with a high GPU speed of 165 FPS, and the MixFormerV2-S surpasses FEAR-L by 2.7\% AUC on LaSOT with a real-time CPU speed.

NixFormerV2: Efficient Fully Transformer Tracking
Limin Wang" State Key Laboratory for Novel Software Technology, Nanjing University, China
Yutao Cui Tianhui Song?
Gangshan Wu
Abstract
Transformer-bant based trackers have achieved high accuracy on standard benchmarks. However, their efficiency remains obstacle to practical deployment on both GPU and CPU platforms. In this paper, to mitigate this issue, we propose fully trans- former tracking framework based the successful MixFormer tracker [14], coined as MixFormerV2, without any dense convolutional operation or complex score prediction module. We introduce four special prediction tokens and concatenate them with those from target template and search area. Then, we apply a unified transformer backbone on these mixed token sequence. These prediction tokens are able to capture the complex correlation between target template and search area via mixed attentions. Based on them, we can easily predict the tracking box and estimate its confidence score through simple MLP heads. To further improve the efficiency of Former22, we present a new distillation- ratillation-based model reduction paradigm, including dense-to-sparse distillation and deep-to-shallow distillation. The former one aims to transfer knowledge from the dense-head based Mix ViT to our fully transformer tracker, while the latter one is for pruning the backhone layers. We instantiate two MixForemrV? trackers, where the MixFormerV2-1 achieves AUC of 70.6% LaSOT and AUC of 56.7% TNL2k with high GPU speed 165 FPS, and the MixFormer surpasses FEAR-L by 2.7% AUC on LaSOT with real-time CPU speed.
1
Introduction
Visual object tracking has been fundamental and long-standing task in computer vision, which aims locate the object in a video sequence, given its initial bounding box. has a wide range practical applications, which often require for low computational latency. So it is important to design more efficient tracking architecture while maintaining high accuracy.
Recently, the transformer-basee one-stream trackers [7, 14, 55] attain excellent tracking accuracy than the previous Siamese-based ones [2, 10, 11], due the unified modeling feature extraction and target integration within transformer block, which allows both components to benefit from the transformer development (e.g. ViT [18], self-supervised pre-training [24] or contrastive pre- training [43]). However for these trackers. inference efficiency, especially on CPU, is still the main obstacle to practical deployment. Taking the state-of-the-arr tracker Mix VIT [15] as an instance, its pipeline contains i) transformer backbone area, ii) dense corner head on the 2D search region for regression and iii) extra complex score prediction module for classification (i.c., estimating the box quality for reliable online samples selection). To achieve a high-efficiency tracker, there are still several issues on the design of ViT. First, the dense convolutional corner head still exhibits time-consuming design, implied in Tab 1. This is because it densely estimates the probability distribution the box corners through a total of
the token sequence from target template and search
Equal contribution. Corresponding author (anvang@nju.cduuco)
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Layer
Head
Score
GPU FPS
GFLOPs
75
SimTTack-B
Pyram. Comer
90
27.2
MixFormerV--8
Pyram. Comer
120tt 33.3%)
26.2
MaFchmerr
bSTrack
Token-based
166 84.4%)
22,5
Table I: Efficiency analysis on Mix VTT-B with dif-
LaSOT AUCOS)
STARK
TransT
MixFormerV2-5
ferent heads. 'Pyram. Corner' represents for the
60
pyramidal corner head [15].
E.r.Track
25-
FEAR-XS
Layer MI.P Ratio
Image Size
CPU FPS
GPU FPS
LighTrack
288
21
262
SiamPPN
224
30
280
4
288
12
255
45
150
250
350
224
15
275
Speedd(on)
288
180
Figure 1: Comparison with state-of-the-art trackers in terms of AUC performance, model Flops and GPU Speed on LaSOT. The cir- cle diameter is in proportion to model flops. Mix surpasses existing trackers by a large margin in terms of both accuracy and inference speed. MixFormerV2-S achieves ex- tremely high tracking speed of over 300 FPS while obtaining competitive accuracy com- pared with other efficient trackers [4, 5].
1
224
190
8
288
150
4
224
190
288
130
224
12
145
12
288
100
224
140
Table 2: Efficiency analysis on VixViT-B with dif- ferent backbone settings. The employed prediction head is plain corner head [14] for the analysis.
ten convolutional layers on the high-resolution 2D feature maps. Second, to deal with online template updating. an extra complex score prediction module composed of precise Rol pooling layer, two attention blocks, and a three-layer MLP is required for improving online samples quality, which largely hinders its efficiency and simplicity of ViT.
To avoid the dense corner head and complicated score prediction module, we propose a new fully transformer tracking Somnework-MMifoormeeer without any dense convolutional operation. Our MixFormerV2. yields a very simple and efficient architecture, which is composed of transformer backbone on the mixed token sequence and two simple MLP heads on the learnable prediction tokens. Specifically, we introduce four special Iearnable prediction tokens and concate them with the original tokens from target template and search area. Like the CLS token in standard ViT, these prediction tokens are able to capture the complex relation between target template and search area, serving as compact representation for subsequent regression and classification. Based on them, we can easily predict the target box and confidence score through simple MLP heads, which results in an efficient fully transformer tracker. Our MLP heads directly regress the probubility distribution of four box coordinates, which improves the regression accuracy without increasing overhead.
To further improve efficiency of MixFormerV2, we present new model reduction paradigm based on distillation, including kense-to-sparee distillation and seep-to-shallow distillation. The dense- to-sparse distillation aims transfer knowledge from the dense-head based Mix to our fully transformer tracker. Thanks to the distribution- -basdd regression design in our MLP head, we can easily adopt logits mimicking strategy for distilling Mix ViT trackers to our MixFormerV2. Based on the observation in Tab. 2, we also exploit the deep-to-shallow distillation to prune our MixFormerV2. We devise a new progressive depth pruning strategy by following a critical principle that constraining the initial distribution of student and teacher trackers to be similar as possible, which can augment the capacity transferring knowledge. Specifically, instructed by the frozen teacher model, some certain layers of copied teacher model are progressively dropped and we use the pruned model as our student initialization. For CPU-realtime tracking, we further introduce an intermediate teacher model to bridge the gap between the large teacher and small student, and prune hidden dim of MLP based on the proposed distillation paradigm.
Based on the proposed model reduction paradigm, we instantiate two types of MixFormerV2 trackers, MixFormerVV2B and MixFormerVV2S. As shown in Fig. 1, MixFormerV2 achieves better trade-off off between tracking accuracy and inference speed than previous trackers. Especially, MixFormerV2-8 achieves an AUC of 70.6% on LaSOT with a high GPU speed of 165 FPS, and MixFormerV2-S outperforms FEAR-L by 2.7% AUC on LaSOT with real-time CPU speed. Our contributions are two-fold: 1) We propose the first fully transformer tracking framework without any convolution operation, dubbed as MixFormer yielding more unified and efficient tracker. 2) We present

---

new distillation-basee model reduction paradigm to make MixFormerV? V2 more effective and efficient, which can achieve high- gegh-peforrance tracking on platforms with GPUs or CPUs.
2 Related Work
Efficient Visual Object Tracking. In recent decades, the visual object tracking task has witnessed
rapid development due the emergence of new benchmark datasets[20, 28, 41, 42, 48] and better trackers [2, 10, 12-14, 32, 52, 55]. Researchers have tried to explore efficient and effective track- ing architectures for practical applications, such as saaeesebbased trackers [2. 31, 32, 50], online trackers [3, 16] and transformer- based trackers [10, 37, 52]. Benefiting from transformer structure and attention mechanism, recent works [7. 14, 55] on visual tracking are gradually abandoning traditional three-stage model paradigm, i.e., feature extraction, information interaction and location head. They adopted more unified one-stream model structure to jointly perform feature extraction and interaction, which turned out to be effective for modeling visual object tracking task. However, some modern tracking architectures are too heavy and computational expensive, making it hard to deploy in practical applications. LightTrack [53] employed NAS to search the a light Siamese network, but its speed was not extremely fast on powerful GPUs. FEAR [5], HCAT [9], E.T.Track [4] designed more efficient framework, however were not suitable for one-stream trackers. We are the first to design efficient one one-stream tracker so as to achieve good accuracy and speed tradc-off.
Knowledge Distillation. Knowledge Distillation [27] was proposed to learn more effective student models with teacher model's supervision. In the beginning, KD is applied classification problem, where KL divergence is used for measuring the similarity of teacher's and student's predicted probability distribution. For regression problem like object detection, feature mimicking [1, 23, 33] is frequently employed. LD [57] operate logits distillation bounding box location by converting Dirac delta distribution representation to probability distribution representation of bounding box, which well unifies logits distillation and location distillation. In this work, we exploit some customized strategies to make knowledge distillation more suitable for our tracking framework.
Vision Transformer Compression. There exist many general techniques for the purpose of speed- ing up model inference, including model quantization [22, 47], knowledge distillation [27, 36), pruning [25], and neural architecture search [19]. Recently many works also focus compressing vision transformer models. For example, Dynamic ViT [44], [VT tried to prune tokens in attention mechanism. AutoFormer [8]. NASVIT [21], Slimming [6] employed NAS technique to explore delicate ViT architecture. ViTKD [54] provided several ViT feature distillation guidelines but it focused compressing the feature dimension instead of model depth. MiniVIT [56] applied weights sharing and multiplexing to reduce model parameters. Since one-stream trackers highly rely on training- rew.inng-eesuuue-oooooeo consuumnng pre-training, we resort to directly prune the layers of our tracker.
3 Method
In this section, we first present the MixFormerV2, which is a more efficient and unified fully transformer tracking framework. Then we describe the proposed iatiillation-basee model reduction, including dense-to-sparse distillation and deep-to-shallow distillation.
3.1 Fully Transformer Tracking: MixFormerV2
The proposed MixFormerV2 is a fully transformer tracking framework without any convolutional operation and complex score prediction module. Its backbone a plain transformer on the mixed token sequence of three types: target template token, search area token, and learnable prediction token. Then, simple MLP heads are placed on top for predicting probability distribution of the box coordinates and corresponding target quality score. Compared with other transformer- trackers (e.g. TransT [10], STARK [52], MixForrer [14], OSTrack [55] and SimTrack [?]). our MixFormerV2 streamlines the tracking pipeline by effectively removing the customized convolutional classification and regression heads for the first time, which yields a more unified, efficient and flexible tracker. The overall architecture is depicted in Fig. 2. With inputting the template tokens, the search area tokens and learnable prediction tokens, MixFormerV2 predicts the target bounding boxes and quality score in end-to-end manner.
3

---

Gropiin
Tracking
Result
Update
Template
E()
Distribution-baaal Localization Head
Score Head
Pred.
Predicted Coordinstes Distribution
Sooce
Search A rea
Skaring Weight
MLP
Special carmblle Prediction Tokans
Mean
b
K/V
p-Mixed Attention Modulee
Figure 2: MixFormerV2 Framework. MixFormerV2 is a fully transformer tracking framework, composed transformer backbone and simple MLP heads on the learnable prediction tokens.
Mixed Attention. Compared to original slimming mixed attention [15] in Mix, the key difference lies in the introduction of the special learnable prediction tokens, which are used to capture the correlation between the target template and search area. These prediction tokens can progressively compress the target information and used as a compact representations for subsequent regression and classification. Specifically, given the concatenated tokens of multiple templates, search and four learnable prediction tokens, we pass them into N layers of prediction- token-in token-involved involved mixed attention modules (P-MAM). We use qt, he and to represent template elements (i.e. query, key and value) of attention, 9.. k, and v, to represent search region, Jes hig and ve to represent learnable prediction tokens. The P-MAM can be defined as:
Prediction-
kzee = Concat(ke, ke, Ka ke),
Utse Concat (Uz, Vs, Ue),
Lake
)o,, Allen, = Softmax( gahits
Uuse, Atlen, Softmax Le gekite Utse
Atlen; Softmax
where d represents the dimension of each elements, Attent Attent, and Atten, are the attention output of the template, search and the learnable prediction tokens respectively. Similar to the original MixFormer, use the asymmetric mixed attention scheme for efficient online inference. Like the CLS tokens in standard ViT, the learnable prediction tokens automatically learn on the tracking dataset compress both the template and search information.
Direct Prediction Based on Tokens. After the transformer backbone, we directly use the prediction tokens to regress the target location and estimate its reliable score. Specifically, we exploit the distribution-basee regression based on the four special learnable prediction tokens. this sense, we regress the probability distribution of the four bounding box coordinates rather than their absolute positions. Experimental results in Section 4.2 also validate the effectiveness of this design. As the prediction tokens can compress target-aware information via the attention modules, we can simply predict the four box coordinates with a same MLP head as follows:
mixed
(x = MLP((ooenx), X E (T,
In implementation, we share the MLP weights among four prediction tokens. For predicted target quality assessment, the Score Head is simple MIP composed of two linear layers. Specifically, firstly we average these four prediction tokens to gather the target information, and then feed the token into the MLP-based Score Head to directly predict the confidence score s which is real number. Formally, we can represent it as:
MLP (mean (token , x E (T,C,
These token-based heads largely reduces the complexity for both the box estimation and quality score estimation, which leads to more simple and unified tracking architecture.

---

Backlous
Heal
Stagel
Backbone
Seart
ag make Depth
Cocdlaates
Theglie Tiree
Proows
tggul Edeic
Feunt Token
Prededes
Linnbating llacc
Heatrre
Nirricting
KL Dovergncce
Figure 3: Distillation-Based Model Reduction for Mix-
Figure 4: Progressive Depth Pruning Process for eliminating blocks. All weights in this block decay to zeros and finally only residual connection works, turning into an identity block.
FormerV2. The 'Stagel' represents for the dense-to-sparse distillation, while the 'Stage2' is the deep-to-shallow distil- lation. The blocks with orange arrows are be supervised and blocks with dotted line are to be eliminated.
3.2
Distillation-Basse Model Reduction
To further improve the efficiency and effectiveness of MiXFormerV2, we present a distillation-bast model reduction paradigm shown in Fig. 3, which first perform dense-to- to-rss distillation for better token-base prediction and then deep-to-sie -o-ssllww distillation for the model pruning.
3.2.1 Dense-to-Sparse Distillation
In Fix we directly regress the target bounding box based on the prediction tokens to the distribution of four random variables TT, L. B, R E R, which represents the box's top, left, bottom and right coordinate respectively. detail, we predict the probability density function of each coordinate: X (2), where X (T, C, B, R.J. The final bounding box coordinates can derived from the expectation over the regressed probability distribution:
Bx = Ex = xPx(x)dr.
(3)
Since the original Mix ViT's dense convolutional corner heads predict two-dimensional probability maps, i.c. the joint distribution Prc(x,8) y) and (BR(I, for top-left and bottom-right corners, the onc-dimensional version of box coordinates distribution can be deduced easily through marginal distribution:
Pr(x) / (rc(a,y)dg) Pc(v) /
Prc(x,
P5(x) = (aR((, yddy, PR(V) PER (æ. y)dz.
Herein, this modeling approach can bridge the gap between the dense corner prediction and our sparse token-based prediction, i.e., and the regression outputs of original Mix can be regarded as soft labels for dense-to-sparse distillation. Specifically, we use MixViT's outputs Px as Equation for supervising the four coordinates estimation Px of MixFormerV2, applying KI.-Divergence loss as follows:
Lioc
Σ
(Px Px).
(5)
KEIT,C,B,R)
In this way, the localization knowledge is transferred from the dense corner head of Mix ViT the head of MixFormerV2.
sparse
3.2.2 Deep-to-Shallow Distillation
For further improving efficiency, we focus on pruning the transformer backbone. However, designing new wiigh backbone is not suitable for fast one-stream tracking. A new backbone of one- stream trackers often highly relies on large-scale pre-training to achieve good performance, which requires for huge amounts of computation. Therefore, resort to directly cut down some layers
5

---

of MixFormerV2 backbone based on both the feature mimicking and logits distillation, can be seen Fig. 3:Stage2. Let P.S. F E Rxxxxe denote the feature map from student and teacher, the subscript represents the index of layers. For logits distillation, we use KI.-Divergence loss. For feature imitation, we apply 1.2 loss:
Leeat
Σ L2(F,, (.))
(6)
where M is the set of matched layer pairs need to be supervised. Specifically, we design progressive model depth pruning strategy for distillation.
Progressive Model Depth Pruning. Progressive Model Depth Pruning aims to compress Mix- FormerV2 backbone through reducing the number of transformer layers. Since directly removing some layers could lead to inconsistency and discontinuity, we explore progressive method for model depth pruning based the feature and logits distillation. Specifically, instead of letting teacher to supervise a smaller student model from scratch, we make the original student model a complete copy of the teacher model. Then, will progressively eliminate certain layers of student and make the remaining layers to mimic teacher's 's representation during training with supervision of teacher. This design allows the initial representation of student and teacher to keep as consistent as possible, providing smooth transition scheme and reducing the difficulty of feature mimicking.
Formally, let z; denote output of the i-th layer of MixFormerV2 backbone, the calculation of attention block can be represented as below Noryar-Normalizationn operation is omitted in equation):
z ATTN( + 2;-1; (FN +24
FFN(ATTN(Z:-1) + 2,-1) + (ITT((x;1)) -1-11
Let & be the set of layers to be eliminated in our student network, we apply decay rate on the weights of these layers:
T3 7 (FN(ATT((:-:)) + + ATTN(x;-1)) + +
(8)
During the first epochs of student network training, 7 will gradually decrease from in the manner of cosine function:
t TT )) m
0.5 x 1 cos
S m,
(t)
0,
m.
This means these layers in student network are gradually eliminated and finally turn into identity transformation, as depicted in Figure The pruned student model can be obtained by simply removing layers in E and keeping the remaining blocks.
Intermediate Teacher. For distillation of an extremely shallow model (4-layers MiXFormerV2), we introduce an intermediate teacher (8-layers MixFormerV2) for bridging the deep teacher (12-layers MixFonmerV2) and the shallow one. Typically, the knowledge of teacher may be too complex for small student model to learn. So we introduce an intermediate role serving as teaching assistant to relieve the difficulty of the extreme knowledge distillation. In this sense, divide the problem of knowledge distillation between teacher and small student into several distillation sub-problems.
MLP Reduction. As shown in Table 2. one key factor affecting the inference latency of tracker on CPU device is the hidden feature dim of MLP in Transformer block. In other words, becomes the bottleneck that limits the real-time speed on CPU device. In order to leverage this issue, we further prune the hidden dim of MLP based on the proposed distillation paradigm, i.e., feature mimicking and logits distillation. Specifically, let the shape of linear weights in the original model is E Rd xd2, and the corresponding shape in the pruning student model is w' E RR4 xdl, in which d' S di, d' S dz, we will initialize weights for student model as: w|[: di, dg). Then we apply distillation supervision for training, letting the pruned MLP to simulate original heavy MLP.

---

3.3 Training of MixFormerV2
The overall training pipeline demonstrated in Fig. 3. performing dense-to- tosse-to-ppare distillation and then deep-to-shallow distillation to yield our final efficient MixFormerV2 tracker. Then, we train the MIPP based score head for 50 epochs. Particularly, for CPU real-time tracking, we employ the intermediate teacher to generate shallower model (4-layer MixFormerV2) based on the proposed distillation. Besides, we also use the designed MIP reduction strategy for further pruning the CPU real-time tracker. The total loss of distillation training with student S and teacher I is calculated as:
X (B*, B + Le (B B + Xa La
(10)
where the first two terms are exactly the same as original MixFormer's 's location loss supervised by ground truth bounding box labels, and the rest term for aforementioned distillation.
Experiments
4.1 Implemented Details
Training and Inference. Our trackers are implemented using Python 3.6 and PyTorch 1.7. The distillation training is conducted on 8 NVidia Quadro RTX 8000 GPUs The inference process runs on one NVidia Quadro RTX 8000 GPU and Intel(R) (con(R) Gold 6230R CPU @ 2. 10GHz. The training datasets includes TrackingNet [42], LaSOT [20], GOT-1OK [28] and COCO [35] training splits., which are the same as MixFormer [14]. Each distillation training stage takes 500 epochs, where the first m 40 epochs are for progressively eliminating layers. We train the score prediction MLP for additional 50 epochs. The batch size is 256, each GPU holding 32 samples. We use AdamW optimizer with weight decay of 10-4, The initial learning rate is 10 and will be decreased to 10-5 after 400 epochs. We use horizontal flip and brightness jittering for data augmentation. We instantiate two types of MixFormerV2, including MixFormerV2-B of P-MAM layers for high-speed tracking GPU platform and MixFormer of 4 P-MAM layers with MLP ratio of 1.0 for real-time tracking on CPU platform. Their numbers of parameters are 58.8M 8M and 16.2M respectively. The resolutions of search and template images for MixFormerV2-BB are 288 x 288 and 128 x 128 respectively. While for MixFormer V2-S, the resolutions of search and template images 224 x 224 and 112 x 112 for real-time tracking CPU platform. The inference pipeline is the same as MixFormer [14]. We use the first template together with the current search region as input of MixFormerV2. The dynamic templates are updated when the update interval of 200 is reached by default, where the template with the highest score is selected as an online sample.
Distillation-Baset Reduction. For dense-to-sparse to-ssass distillation, we use Mix ViT-L as teacher for training MixFormer by default. We also try to use Mix VīT-B as the teacher in Tab 5. Particularly, we employ customized Mix of plain corner head and with search input size of 224 x 224 as the teacher for MixFormer-S. For deep-to-shallow distillation, we use the progressive model depth pruning strategy to produce the 8-layer MixFormer V2-B from a 12-layer one. For MixFormerV2--S we additionally employs intermediate teacher and MLP reduction strategies, and the process is '12-layers MixFormerV2 to 8-layers MixFormerV2, then 8-layers MixFormerV2 to 4-layers MixFormerV2, finally layers MLP-ratio-4.0 MixForme V2 to 4-layers MLP-ratio-1 1.0 MixFormerV2-S.
4.2
Exploration Studies
To verify the effectiveness of our proposed framework and training paradigm, we analyze different components of MixFormerV2 and perform detailed exploration studies on LaSOT [20] dataset.
4.2.1 Analysis on MixFormerV2 Framework
Token-based Distribution Regression. The design of distribution- based regression with special learnable prediction tokens the core of our MixFormerV2. We conduct experiments on different regression methods in Tab. 3a. All models employ ViT-B as backbone and are deployed without distillation and online score prediction. Although the pyramidal corner head obtains the best perfor- mance, the running speed is largely decreased compared with our token-based regression head in MixFormerV2. MixFormerV2 with four prediction tokens achieves good trade-off between perfor- mance and inference latency. Besides, compared to the direct box prediction with one token on the
7

---

Type
Layer
AUC
FPS
Method
Score.
AUC
FPS
Stu. Tea.
Tea-AUC
AUC
12
63.1%
112
Ours
68.9%
166
base
67.5%
T4
12
67.5%
110
Ours
70.6%
165
base hase 69.0%
68,99 9%,9%
Py-Corner.
12
69.0%
92
MixViT-B
69.0%
92
base large 71.5%
69.6%
Mix ViX-IT-B
69.6% 80
(a) Different regression methods. (b) Quality score prediction. In (c) Dense-to-Sparse distillation.
'TI' denotes direct box prediction MixFormerV2, we use token-based The "base" student denotes the based on one token, "T4' is the pro- MLP head for sample quality score
12-layers MixForeerV2 framework without score prediction. The 'base' teacher is the Mix MixViT-B and the 'large' teacher is the Mix ViT-L. "Tea- AUC' is the AUC of the teacher. Models without score prediction.
posed distribution-bass
prediction prediction. While Mix use
with 4 prediction tokens, and 'Py- an extra SPM for score prediction.
Conrer." is the pyramidal comner We employ MixFormerV2-B of 8
head as in VT.. Models are with- layers, with the ViT- as the dis-
out distillation and prediction. tillation teacher, for this analysis.
Log-dis. Feat-mim. AUC
Init. method AUC
Inter. teacher AUC
60.7%
MAE-fir4 62.9%
62.4%
Tea-skip4 64.4%
64.8%
62.9%
PMDP 64.8%
65.5%
(d) Feature mimicking & logits (e) Progressive model depth prun- (f) Intermediate teacher. For the 12-layers MixViT-B as the teacher, 8-layers
distillation. For distillation analysis, we use the Mix ViT-B of 12 layers with comerr head as the teacher, and
ing (PMDP). 'MAE-fir4' denotes the analysis,
using first layers of MAE-B
student initialization. "Tca-skip4' is MixViT as the intermediate teacher using 4 skipped layers of the teacher. and 4-layers MixViT ViT as the student.
ViX of 4 layers as the student.
blocks num. head MLP-r
AUC
Epoch m
AUC
blocks
head
AUC
12
Cor.
4
68.2%
30
68.3%
12
Py-Corner.
69.0%
12
T4
67.7%
40
68.5%
12
T4
68.9%
8
T4
4
66.6%
50
68.5%
8
T4
68.5%
4
T4
4
61.0%
4
T4
59.4%
Eliminating Epochs. 'Epoch (h) Model pruning route of (i) Model pruning route of Tra indicates the number of epochs MixFormerV2-BB. 'T4' denotes MixFormerV22S. 'Cor,' represents in progressive climinating process. the proposed distribation-based pre- for the plain corner head, which is The model architecture is based on diction with 4 prediction tokens. We used in the initial teacher model, MixFormerV2 B. Models are with- use the Mix ViT-B the distillation 'MLP-r' denotes the MLP ratio in out score prediction.
teacher this analysis.
attention blocks.
Table 3: Ablation studies on LaSOT.The default choice for our model is colored in gray
first line of Tab. 3a, which estimates the absolute target position instead of the probability distribution of four coordinates, the proposed distribution-based regression obtains better accuracy. Besides, this design allows to perform dense-to-sparse distillation so as to further boost performance.
Token-based Quality Score Prediction. The design of the prediction tokens also allows to perform more efficient quality score prediction via simple MLP head. As shown in Tab. 3b, the token-based score prediction component improves the baseline MixFormerV2-B by 1.7% with increasing quite little inference latency. Compared to ours, the score prediction module in ViT-B further decreases the running speed by 13.0%, which is inefficient. Besides, the SPM in Mix VIT requires precise Rol pooling, which hinders the migration to various platforms.
4.2.2 Analysis on Dense-to-Sparse Distillation
We verify the effectiveness of dense-to tonse-to-sparse distillation in Tab. 3c. When use MiXViT-B without its SPM (69.0% AUC) the teacher model, the MixFormerV2 of 12 P-MAM layers achieves an AUC score of 68.99 68.9%, increasing the baseline by 1.4%. This further demonstrate the significance of the design of four special prediction tokens, which allows to perform dense-to-sparse distillation. The setting of using ViX (71. 17.5% AUC) as the teacher model increases the baseline by an AUC score of 2.2%, which implies the good distillation capacity of the large model.
8

---

KCF
SiamPC
ATOM
D3S+2
DIMP
ToMP
TransT
SBT
SwinTrack [34]
Ours-S
Ours-B
[26]
[2]
[16]
[38]
[3]
[39]
[10]
[49]
EAO
0,239
0.255
0.386
0.356
0,430
0.511
0.512
0,522
0.524
0.431
0.556
Accuracy
0.542
0.562
0.668
0.521
0.689
0.752
0.781
0.791
0.788
0.715
0.795
Robustness
0.532
0.543
0,716
0.811
0.760
0.818
0,800
0.813
0.803
0,757
0.851
Table 4: State-of-the-art comparison on VOT2022 [30]. The best results are shown in bold font.
4.2.3
Analysis on Deep-to-Shallow Distillation
In the following analysis on deep-to-shallow distillation, we use the VixViT-B of 12 layers with plain corner head as the teacher, and ViT of layers with the same corner head as the student. The models are deployed without score prediction module.
Feature Mimicking Logits Distillation. To give detailed analysis on different distillation methods for tracking, we conduct experiments in Tab. 3d. The models are all initialized with the first 4-layers MAE pre-trained ViT-B weights. can be seen that logits distillation can increase the baseline by 1.7% AUC, and adding feature mimicking further improves by 0.4% AUC, which indicates the effectiveness of both feature mimicking and logits distillation for tracking.
Progressive Model Depth Pruning. We study the effectiveness of the progressive model depth pruning (PMDP) for the student initialization in Tab. 8b. It can observed that the PMDP improves the traditional initialization method of using MAE pre-trained first 4-layers ViT-B by 1.9%. This demonstrates that critical for constraining the initial distribution of student and teacher trackers to be as similar as possible, which can make the feature mimicking easier. Surprisingly, we find that even the initial weights of the four layers are not continuous, i.e., using the skipped layers (the 3,6,9,12-th) of the teacher for initialization, the performance is better than the baseline (62.9% vs. 64.4%), which further verifies the importance of representation similarity between the two ones.
Determination of Eliminating Epochs. We conduct experiments as shown in the Table 3g to choose the best number of epochs m in the progressive eliminating period. We find that when the epoch greater than 40, the choice of m seems hardly affect the performance. Accordingly we determine the epoch to be 40.
Intermediate Teacher. Intermediate teacher is introduced to promote the transferring capacity from a deep model shallow one. We conduct experiment as Table 3f. We can observe that the intermediate teacher can bring a gain of 0.7% AUC score which can verify that.
4.2.4 Model Pruning Route
We present the model pruning route from the teacher model Mix and MixFormerV2-S in Tab. 3h and Tab. 3i respectively. The models on the first line are corresponding teacher models. We can see that, through the dense-to-sparse distillation, our token-based MixFormerV2-B obtains comparable accuracy with the dense-comer-baaeed Mix VTT-B with higher running speed. Through the progressive model depth pruning based on the feature and logits distillation, MixFormerV2-B with layers only decreases little accuracy compared to the 12-layers one.
4.3 Comparison with the Previous Methods
Comparison with State-of-the-art Trackers. We evaluate the performance of our proposed track- ers on benchmark datasets: including the large-scale LaSOT [20], LaSOT ezt [20], Tracking [42], UAV 123 [41], TNL2K [48] and VOT2022 [30]. LaSOT a large-scale dataset with 1400 long videos in total and its test set contains 280 sequences. TrackingNet provides over 30K videos with more than 14 million dense bounding box annotations. UAV 123 is large dataset containing 123 aerial videos which is captured from low-altitude UAVs. VOT2022 benchmark has 60 sequences, which measures the Expected Average Overlap (EAO), Accuracy (A) and Robustness (R) metries. Among them, LaSOT ezt and TNL2K are two relatively recent benchmarks. LaSOT ext is a released ex- tension of LaSOT, which consists of 150 extra videos from 15 object classes. TNL2K consists of 2000 sequences, with natural language description for each. We evaluate our MixFormerV2 on the test set with 700 videos. The results are presented Tab. 4 and Tab. More results on other datascts will be present in supplementary materials. Only the trackers of similar complexity

---

LaSOT
LaSOT
TNL2K
TrackingNet
UAV123
Speed
Method
AUC
PNrrm
AUC
AUC
P
AUC
PYorm
AUC
GPU
MixFormerV221 V2-B
70,6
80.8
76.2
50.6
58.4
83.4
88.1
81.6
69.9
92.1
165
MixFormerV2 V
69.5
79.1
75.0
56.6
57.1
K2.9
87.6
81.0
70.5
91.9
165
MixFormer [14]
69.2
78.7
74.7
83.1
88.1
81.6
70.4
91.8
25
CTTrack-B [45]
67.8
77.8
74.0
82.5
80.3
68.8
89.5
40
OSTrick-256 [55]
69.1
78.7
75.2
47.4
53.3
54.3
83.1
87.8
82.0
68.3
105
SimTrack-B [7]
69.3
78.5
54.8
53.8
82.3
86.5
69.8
89.6
40
CSWinTT [46]
66.2
75.2
70.9
81.9
86.7
79.5
70.5
90.3
12
SBT-Base [49]
65.9
70.0
37
SwinTrack-T [34]
67.2
70.8
47.6
$3.9
53.0
53.2
81.1
78.4
98
ToMP101 [39]
68.5
79.2
68.5
81.5
86.4
78.9
66.9
20
STARK-STS0 [52]
66.4
42
KeepTruck [40]
67.1
77.2
70.2
48.2
69.7
19
TransT [10]
64.9
73.8
69.0
50.7
51.7
81.4
86.7
80.3
69.1
50
PrDiMP [17)
59.8
68.8
75.8
70.4
68.0
47
ATOM [16]
51.5
57.6
50.5
70.3
77.1
64.8
64.3
83
Table 5: State-of-the-art comparison on TrackingNet [42], LaSOT [20], LaSOT eant [20], UAV 123
[41] and TNL2K [48]. The best two results are shown in bold and underline fonts. *** denotes tracker
with Mix as the teacher during the dense-to-sparse distillation process. The default teacher
Mix ViT-L.. Only trackers of similar complexity are included.
Method
LaSOT
LaSOT
TNL2K
TrackingNet
UAV123
Speed
AUC
Parrm
P
AUC
P
AUC
P
ALC
Narrn 81.1
P
AUC
GPU
CPU
MixFormerVe V2-8
60.6
69.9
61.4
43.6
46.2
48.3
75.8
70.4
65.8
86.8
325
30
[S]
57.9
68.6
60.9
FEAR-XS [5]
53.5
64.1
54.5
26
HCAT[9]
59.0
68.3
60.5
76.6
82.6
72.9
63.6
45
F.T.Track [4]
59.1
74.5
80.3
70.6
62.3
150
42
LighrTrack-LargaA [53]
55,5
56.1
73.6
78.8
70.0
LighTTack-Mohile [53]
53.8
53.7
72.5
77.9
69.5
120
36
STARK-Lightnng
58.6
69.0
$7.9
DiMP [3]
56.9
65.0
56.7
74.0
80.1
68.7
65.4
15
SiamFC++ [50]
54.4
62.3
54.7
75.4
80.0
70.5
20
Table Comparison with CPU-realtime trackers on TrackingNet [42], LaSOT [20], LaSOT [20].
UAV 123 [41] and TNL2k [48]. The best results are shown in bold fonts.
included, i.e., the trackers with large-scale backbone or large input resolution are excluded. Our MixFormer V2-B achieves state-of-the-ard performance among these trackers with very fast speed, especially compared to transformer-basee one-stream tracker. For example, MixFormerV2-B without prssesocessing strategies surpasses OSTrack by 1.5% AUC on LaSOT and 2.4% AUC on TNL2k, running with quite faster speed (165 FPS vs. 105 FPS). Even the MixFormerV2-B with Mix ViT-B as the teacher model obtains better performance than existing SOTA trackers, such as MixFormer, OSTrack, ToMP101 and Sim Track, with much faster running speed on GPU.
Comparison with Efficient Trackers. For real-time running requirements on limited computing resources such CPU, explore lightweight model, i.e. MixFormerV2-S, which still reaches strong performance. And it is worth noting that this is the first time that transformer-based one- stream tracker is able run on CPU device with real-time speed. As demonstrated in Figure MixFormerV2-S surpasses all other architectures of CPU-real-tine trackers by large margin. We take comparison with other prevailing efficient trackers on multiple datasets, including LaSOT, TrackingNet, UAV122 and TNL2k, in Tab 6. We can see that our MixFormerV2-5 outperforms FEAR-L by an AUC score of 2.7% and STARK- Lightting by an AUC score of 2.0% on LaSOT.
Conclusion
In this paper, we have proposed fully transformer tracking framework MixFormerV2, composed of standard ViT backbones on the mixed token sequence and simple MLP heads for box regression and quality score estimation. Our MixFormerV2 streamlines the tracking pipeline by removing the dense convolutional head and the complex score prediction modules. We also present a distillation based model reduction paradigm for MixFormerV2 to further improve its efficiency. Our MixFormerV2 obtains a good trade-off between tracking accuracy and speed on both GPU and CPU platforms. We hope our MixFormerV2 can facilitate the development of efficient transformer trackers in the future.
10

---

Acknowledgement
This work is supported by National Key R&D Program of China (No. 20222D01 National Natural Science Foundation of China (No. 62076119, No. 61921006), Fundamental Research Funds for the Central Universities (No. 020214380099), and Collaborative Innovation Center of Novel Software Technology and Industrializztion.
References
[1] Romero Adriana, Ballas Nicolas, K Samira Ebrahimi, Chassang Antoine, Gatta Carlo, and B Yoshua. Fitnets: Hints for thin deep nets, Proc. ICLR, 2. 2015.
[2]
Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr. Fully-convohttional siamese networks for object tracking. In Proceedings of the European Conference on Computer Vision,
RCCV Workshops, 2016.
[3] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning discriminative model prediction for tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, pages 6182-6192 1221,191,222 2019.
[4] Philippe Blatter, Menelaos Kanakis, Martin Danelljan, and Luc Van Gool. Efficient visual tracking with exemplar transformers. arXiv preprint arXiv:2112. 09686, 2021.
[5] Vasyl Borsuk, Roman Vei, Orest Kupyn, Tetiana Martyniuk, Igor Krashenyi, and Jini Matas. Fear: Fast, efficient, accurate and robust visual tracker. arXiv preprint 1xXi::2122009955 2021.
[6] Arnav Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, and Eric P Xing. Vision transformer slimming: Multi-dimension searching in continuous optimization space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4931 1-144141 2022.
Boyu Chen, Peixia Li, Lei Bai, Lei Qiao, Qiuhong Shen, Bo Li, Weihao Gan, Wei Wu, and Wanli Ouyang. Backhone is all your need: A simplified architecture for visual object tracking. In Proceedings of the European Conference on Computer Vision ECCV, 2022.
[8] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
22700112800 2021.
[9] Xin Chen, Dong Wang, Dongdong Li, and Huchuan Lu. Efficient visual tracking via hierarchical cross- attention transformer. arXiv preprint arXiv:2203. 135377 2022.
[10] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan La. Transformer tracking. In IEFE Conference on Computer Vision and Pottern Recognition, CVPR, 2021.
[11] Zedu Chen, Bineng Zhong, Guorong Li. Shengping Zhang, and Rongrong Ji. Siamese box adaptive network for visual tracking. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
[12] Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu. Target transformed regression for accurate tracking. CoRR, 2021.
[13] Yutao Cui, Cheng Jiang. Limin Wang. and Gangshan Wu. Fully convolutional online tracking. Computer Vision and Image Understanding, 224: 103547, 2022.
[14] Yutao Cui, Cheng Jiang, Limin Wang. and Gangshan Wu. Mixformer: End-to-end tracking with iterative mixed attention. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2022.
[15] Yutao Cui, Cheng Jiang, Gangshan Wu, and Limin Wang. Mixformer: End-to-end tracking with iterative mixed attention, 2023.
[16] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg, ATOM: accurate tracking by overlap maxinization. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2019. [17] Martin Danelljan, Luc Van Gool, and Radu Timofte. Probabilistic regression for visual tracking. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, ICLR, 2021.
[19] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal
of Machine Learning Research, 1997:)9992001 2019.
[20] Heng Fan. Liting Lin, Fan Yang, Peng Chu. Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and benchmark for scale single object tracking. In IEEE Conference Computer Vision and Pattern Recognition, CVPR, 2019.
Haibin Ling. Lasot: A high-
[21] Chengyue Gong, Dilin Wang. Meng Li, Xinlei Chen. Zhicheng Yan, Yuandong Tian, Vikas Chandra, al. Nasvit: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training. In International Conference on Learning Representations, 2021.
[22] Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdey. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv;
2014.
[23] Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors via decoupled features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages
2021.

---

[24] Kaiming He. Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE Conference on Computer Vision and Pattern Recognition. CVPR, 2022.
[25] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
[26] João F. Henriques. Rui Caseiro. Pedro Martins, and Jorge Batista. High-speed tracking with kernelized corretation filters. IEEE Trans. Puttern Anal. Mach. Intell.. 37(3):583-55 59333359966660 2015.
[27] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network (2015), arXiv preprint 1 02531, 2, 2015.
[28]
Lianghua Huang. Xin Zhao, and Kaiqi Huang. Got-10k: A large hiihh-iverrsn benchmark for generic
object tracking in the wild. IEEE Trans. Pattern Anal. Mach. Intell., 1365):1562-1155 1577, 2021.
[29] Matej Kristan, Ales Leonardis, and ct. al. The eighth visual object tracking VOT2020 challenge results. In Adrien Bartoli and Andrea Fusiello, editors, Proceedings of the European Conference on Computer Vision, ECCV Workshops. 2020.
[30] Matej Kristan, Alcš Leonardis, Jifí Matas, Michael Felsberg, Roman Pftugfelder, -riitinn Kämäräinen, Hyung Jin Chang. Martin Danelljan, Luka Cehovin Zaje, Alan Lukežič, et al. The tenth visual object
tracking vor2022 challenge results. In ECCV 2022 Workshops, pages 431-460, 2023.
[31] Bo Li, Wei Wu, Qiang Wang. Fangyi Zhang, Junliang Xing, and Junjie Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. In IEEE Conference on Computer Vision and Pattern Recognition. CVPR, 2019.
[32] Bo Li, Junjic Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. High performance visual tracking with siamese region proposal network. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2018. prnpas [33] Quanquan Li, Shengying Jin, and Junjie Yan. Mimicking very efficient network for object detection. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 6356-6364, 2017. [34] Liting Lin, Heng Fan, Yong Xu, and Haibin Ling. Swintrack: A simple strong baseline for transformer tracking. Neural Information Procesiigg Systems, NIPS, 2022.
[35] Tsung- Yi Lin, Michael Maire. Serge J. Belongie, James Hays. Pietro Perona,
Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Proceedings of the European Conference on Computer Vision, ECCV, 2014.
[36] Benlin Liu, Yongming Rao, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Metadistiller: Network self-hoosting via meta-learned top-down distillation. In European Conference on Computer Vision, pages 694-709. Springer, 2020.
[37] Ze Liu, Yutong Lin, Yue Cao, Han HU, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IFEE/CVF International Conference on Computer Vision, ICCV, 2021.
[38] Alan Lukezic, Jiri Matas, and Matej Kristan. D3S A discriminative single shot segmentation tracker. In
IEEE Conference on Compuder Vision and Puttern Recognition, CVPR. 2020.
[39]
Christoph Mayer, Martin Danelljan, Goutam Bhat, Matthieu Paul, Danda Pani Paudel, Fisher Yu, and Luc Van Gool. Transforming model prediction for tracking. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2022.
[40] Christoph Mayer, Martin Danelljan. Danda Pani Paudel, and Luc Van Gool. Learning target candidate association to keep track of what not to track. In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2021.
[41] Matthias Mueller, Neil Smith, and Bernard Ghanem. A benchmark and simulator for UAV tracking. In Bastian Leibe, Jiri Matas, Nicu Sehe, and Max Welling, editors, Proceedings of the European Conference on Computer Vision, ECCV, 2016.
[42] Matthias Müller, Adel Bibi, Silvio Giancola, Salman Al-Subaihi, and Bernard Ghanem. Trackingnet:
A large-scale dataset and henchmark for object tracking in the wild. In Proceedings of the European Conference on Computer Vision, ECCV, 2018.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8788 PMLR, 2021.
[44] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamievit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems,
13949, 2021.
[45] Zikai Song, Run Luo, Junqing Yu, Yi-Ping Phoebe Chen, and Wei Yang. Compact transformer tracker with correlative masked modeling. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
February 2023.
[46] Zikai Song, Junqing Yu, Yi-Ping Phoebe Chen, and Wei Yang. Transformer tracking with eyelic shifting window attention. In JEEE Conference on Computer Vision and Pattern Recognition. CVPR, 2022. [47] Kuan Wang. Zhijian Liu, Yujun Lin, Ji Län, and Song Han. Haq: Hardware-awan automated quantization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8612-8620, 2019.
[48] Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Pro- ceedings of the IEEL/CVF Conference on Computer Vision and Pattern Recognition, pages 13763-13773,
12

---

2021.
[49] Fei Xie, Chunyu Wang, Guangting Wang, Yue Cao, Wankou Yang. and Wenjun Zeng. Correlation- -waaee deep tracking. In IEEE Conference Computer Vision and Pattern Recognition, CVPR, 2022.
[50]
Yinda Xu, Zeyu Wang, Zuoxin Li, Ye Yuan, and Gang Yu. Siamfe++: Towards robust and accurate visual tracking with target estimation guidelines. In Proceedings of the AAAI Conference on Artificial Intelligence,
AAAI 2020.
[51] Yifan Xu, Zhijie Zhang. Mengdan Zhang. Kekai Sheng, Li, Weiming Dong, Liqing Zhang. Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2964-2972, 2022.
[52] Bin Yan, Houwen Peng. Jianlong Fu, Dong Wang, and Huchuan Lu. Learning
transformer for visual tracking. In Proceedings of the JEEE/CVF International Conference on Computer Vision, ICCV,
2021.
[53] Bin Yan. Houwen Peng, Kan Wu, Dong Wang, Jianlong Fu, and Huchuan Lu. Lighttrack: Finding lightweight neural networks for object tracking via one-shot architecture search. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15180-15189, 2021.
[54] Zhendong Yang. Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. Vitkd: Practical guidelines for vit feature knowledge distillation. arXiv preprint arXiv:2209.024 2209.02432 2022.
[55] Botao Ye, Hong Chang, Bingpeng Ma, and Shiguang Shan. Joint feature learning and relation modeling for tracking: A one-stream framework. Proceedings of the European Conference on Computer Vision, ECCV. 2022.
[56] Jinnian Zhang, Houwen Peng, Kan Wu, Mengehen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vision transformers with weight multiplexing. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 12145-12154, 2022.
[57] Zhaohui Zheng, Rongguang Ye, Qibin Hou, Dongwei Ren, Ping Wang. Wangmeng Zuo, and Ming-Ming Cheng. Localization distillation for object detection. arXiv preprint 2rXiv:2204005957. 2022.
13

---

Appendix
Broader Impact
In this paper. we introduce MixFormerV2, a fully transformer tracking approach for efficiently and effectively estimating the state of an arbitrary target in video. Generic object tracking is one of the fundamental computer vision problems with numerous applications. For example, object tracking (and hence MixFormerV2) could be applied human-machine interaction, visual surveillance and unmanned vehicles. Our research could be used to improve the tracking performance while maintaining high running speed. Of particular concern is the use of the tracker by those wishing to position and surveil others illegally. Besides, if the tracker is used in unmanned vehicles, it may be challenge when facing the complex real-world scenarios. To mitigate the risks associated with using MixFormerV2, we encourage researchers to understand the impacts of using the trackers in particular real-world scenarios.
Limitations
The main limitation lies in the training overhead of MixFormerV22S, which performs multiple model pruning based on the dense-to-spars, distillation and deep-to- to-lloloo distillation. In detail, we first perform distillation from Mix ViT with 12 layers and plain corner head to MixFormerV2 12 layers. The 12-layers MixFormerV2 is pruned to 8-layers and then to 4-layers MixFormerV2 based on deep-to-shallow distillation. Finally, the MLP-ratio-4.0 4-layers MixFormerV2 pruned to MLP-ratio-4.0 4-yyyers MixFormerV2-S S for real-time tracking CPU. For each step, it requires training for 500 epochs which is time-consuming.
S.1 Details of Training Time
The models are trained on 8 Nvidia RTX8000 GPUs. The dense-to-sparyy stage takes about 43 hours. The deep-to-shallow stagel (12-10-8 layers) takes about 42 hours, and stage2 (8-to-4 layers) takes about 35 hours.
S.2 More Results on VOT2020 and GOT10k
VOT2020. We evaluate our tracker on VOT2020 [29] benchmark, which consists of 60 videos with several challenges including fast motion, occlusion, etc. The results reported in Table 7, with metrics Expected Average Overlap(EAOO) considering both Accuracy(A) and Robustness. Our MixFormerV2- obtains an EAO score of 0.322 surpassing CSWinTT by 1.8%. Besides, our MixFormerV2-S achieves an EAO score of 0.258, which is higher than the efficient tracker LightTrack, with real-time running speed on CPU.
GOT1OK. GOTTOK [28] is a large-scale dataset with over 10000 video segments and has 180 segments for the test set. Apart from generic classes of moving objects and motion patterns, the object classes in the train and test set are zero-overlapped. We evaluate MixFormer trained with the four datasets LaSOT, TrackingNet. COCO and GOTIOk-train on GOT Okkttest. We compare with MixFormer and TransT with the same training datasets for fair comparison. MixFormerV221 improves MixFormer and TransT by 0.7% and 1.6% on AO respectively with high running speed of 165 FPS.
KCF
SiamPC [2]
ATOM
LighTTrack
DIMP
STARK
TransT [10]
CSWinTT
MixFormer [14]
Ours-S
Ours-B
[26]
[16]
[53]
[3]
[52]
[46]
V0120ga0
0.154
0.179
0.258
0.322
GO
0.203
0.348
0.556
0.611
0,688
0.723
0.694
0.726
0.621
0.739
Table 7: saate-of-the-ane comparison on VOT2020 [29] and GOT1Ok [28]. * denotes training with four datasets including LaSOT [20]. TrackingNet [42]. GOT1Ok [28] and COCO [35]. The best results are shown in bold font.
S.3 More Ablation Studies
Design of Prediction Tokens. We practice three different designs of prediction tokens for the target localization in Tab, 8a. All the three methods use the formulation of estimating the probability
14

---

3+ token num.
MLP num.
AUC
Init. method LaSOT
LaSOT ex1 UAV123
67.1%
Tea-fir4 62.9%
45.2%
65.7%
4
4
67.3
Tea-skip4 64.4%
46.1%
66.6%
67.5%
PMDP 64.8%
47.1%
67.5%
(a) Different prediction designs, 'token num." indicates the number of the learnable prediction tokens, 'MLP num.' denotes the number of employed ML. layers localization.
(b) Progressive Model Depth Pruning (PMDP). 'Tea-fir4' denotes using first 4 layers of the teacher for student initialization. 'Tea- skip4' is using 4 skipped layers of the teacher. Table 8: More ablation studies. The default choice for our model is colored in gray.
distribution of the four coordinates of the bounding box. The model on the first line denotes using one prediction token and then predicting coordinates distribution with four independent MLP heads. can be observed that adopting separate prediction tokens for the four coordinates and a same MLP head retains the best accuracy.
More Exploration of PMDP Tea-skip4 is a special initialization method, which chooses the skiped four layers (layer-3/6/9/12) of the teacher (Mix VīT-B) for initialization. In other words, Tea-skip4 an extreme case of ours PMDP when the eliminating epoch m equal to O. So it is reasonable that Tea-skip4 performs better than the baseline Tea-fir4, which employs the first four layers of the teacher (MixViT-B) to initialize the student backbone. In Table 8b, we further evaluate the performance on more benchmarks. can be seen that ours PMDP surpasses Tea-skip4 by 1.0% on LaSOT_ess _xxt, which demonstrate its effectiveness.
Computation Loads of Different Localization Head We showcase the FLOPSs of different heads as follows. Formally, we denote C.n as input feature dimension, Cout as output feature dimen- sion, Hi, Win as input feature map shape of convolution layer, Hout, W out output feature map shape, and K as the convolution kernel size. The computational complexity of one linear layer is Couin. and that of one convolutional layer is O Cour Hout W out K²).
In our situation, for T4, the Localization Head contains four MLP predict four coordinates. Each MLP contains two linear layer, whose input and output dimensions are all 768. The loads can be calculated as:
Loadr4- = 4 X (768 x 768 + 768 x 72) = 2580480 2.5M
For Py-Corner, totally 24 convolution layers are used. The loads can be calculated as:
Load py Pyrrrr * (768 384 18
3+
384 * 192 * 18 18 *
384 = 192 + 18 * 18 3+
192 x 96 36 x * * 3+
384 96
48 72
48 72
192 - 96 x000011 3+
48
18
96 48 36 3+
48 1 36 * 36 x 3)
3902587776 3.9B
For simplicity, we not include some operations such as bias terms and Layer/Bacch-Normalzziiit which does not affect the overall calculation load level. Besides, the Pyramid Corner Head utilize additional ten interpolation operations. Obviously the calculation load of Py-Corner is still hundreds of times of T4.
15

---

S,4 Visualization Results
Visualization of Attention Map To explore how the introduced learnable prediction tokens work within the P-MAM, we visualize the attention maps of prediction-oken-tt--saace and prediction- token-to-template in Fig. and Fig. 6, where the prediction tokens are served as query and the others as keyval of the attention operation. From the visualization results, we can arrive that the four prediction tokens are sensitive to corresponding part of the targets and thus yielding a compact object bounding box. We suspect that the performance gap between the dense corner head based Mix VīT-B and our fully transformer MixFormer V2-B without distillation lies in the lack of holistic target modeling capability. Besides, the prediction tokens tend to extract partial target information in both the template and the search so as to relate the two ones.
Visualization of Predicted Probability Distribution We show two good cases and bad cases in Figure 7. In Figure 7a MixFormerV2 deals with occlusion well and locate the bottom edge correctly. As show in Figure 7b, the probability distribution of box representation can effectively alleviate issue of ambiguous boundaries. There still exist problems like strong occlusion and similar objects which will lead distribution shift, as demonstrated in Figure 7c and 7d.
MITM
HOT
E3
63 HOT
E3
E3 HOT
Search
Top
Left
Bottom
Right
Template
Top
Left
Bottom
Right
Visualization of prediction cenettionntokkn-teee search attention maps, where the prediction to- kens are served as query of attention operation.
Figure 5:
token-to-
Figure 6: Visualization of prediction-token-tt template attention maps, where the prediction to- kens are served query of attention operation.
(a)
(b)
(c)
(d)
Figure 7: In each figure, the left one is plot of the probability distribution of predicted box (red), which demonstrates how algorithm works. The right one is heatmap of attention weights in the backbone. The examples are from LaSOT test subset and the green boxes are ground truths.
16


Reviews:
[{"review_id": "KVnv6xAErI", "reviewer": "Reviewer_iyBU", "summary": "This paper proposes a fully transformer tracking framework without any dense convolutional operation and complex modules. The key contribution is the design of different input tokens and the distillation-based model reduction paradigm. Mixed attentions are performed between prediction tokens and the image generated tokens to capture abundant correlation. The distillation methods, dense-to-sparse distillation and deep-to-shallow distillation, improve the efficiency of the proposed MixFormerV2. Evaluations carried out on major tracking benchmarks show promising results.", "questions": "Please address the issues in the weakness section. ", "limitations": "The authors do not discuss the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed fully transformer tracking framework is interesting and makes sense.\n- The proposed method achieves favorable performance in both terms of accuracy and run speed.\n", "weaknesses": "Some important details are missing.\n- The details of some components are missing, such as the details of the Score Head.\n- In the first ablation experiment, it shows that the run speed of the Py-Corner version is slower than that of T4. The calculation loads of T4 (under the VIT framework) is not light. It is suggested to provide a detailed calculation comparison between T4 and Py-Corner.\n- Based on equation (8) and (9), some layers in the student network are gradually eliminated and finally turn into identity transformation. How to decide which layer to be eliminated? How to determine the epoch m in the training process. Are these eliminated parameters removed or just not involved in the process flow of the network.\n- It is suggested to provide intermediate results, such as heatmaps, to better show how the proposed algorithm works.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***C1:  The details of some components are missing, such as the details of the Score Head.***\n\n**R1:** We will add the detailed structure of the heads in our revision. The Score Head is a simple MLP composed of `two linear layers with the hidden dimension of 768.` \nSpecifically, firstly we average these four prediction tokens to gather the target information, and then feed the token into the MLP-based Score Head to directly predict the confidence score $s$ which is a real number. Formally, we can represent it as:\n$$ s = \\mathrm{MLP}(\\mathrm{mean}(\\mathrm{token}_X)), X \\in \\{\\mathcal{T}, \\mathcal{L}, \\mathcal{B}, \\mathcal{R}\\}. $$\n\n***C2: The calculation loads of T4 (under the VIT framework) is not light. It is suggested to provide a detailed calculation comparison between T4 and Py-Corner.***\n\n**R2:** Thanks for your suggestion, we will add the detailed comparison between Py-Corner head and T4 head in terms of FLOPs in our revised manuscrip. Since `the four prediction tokens can be concatenated to the search tokens as a whole for mixed attention in our implementation, the computational latency can be omitted`, which means only the four MLP heads are ought to calculate loads (FLOPs). Based on that, we showcase the FLOPs as follows.\n\nFormally, we denote $C_{in}$ as input feature dimension, $C_{out}$ as output feature dimension, $H_{in}, W_{in}$ as input feature map shape of convolution layer, $H_{out}, W_{out}$ as output feature map shape, and $K$ as the convolution kernel size. \nThe computational complexity of one linear layer is $O(C_{in}C_{out})$, and that of one convolutional layer is $O(C_{in}C_{out}H_{out}W_{out}K^2)$.\n\nIn our situation, for **T4**, the Localization Head contains four MLP to predict four coordinates. Each MLP contains two linear layer, whose input and output dimensions are all 768. The loads can be calculated as:\n$$Load_{T4} = 4 \\times (768 \\times 768 + 768 \\times 72)= 2580480 \\sim 2.5 M$$\nFor **Py-Corner**, totally 24 convolution layers are used. The loads can be calculated as: \n$$ \\begin{aligned}\nLoad_{Py-Corner} = 2 * (&768 * 384 * 18 * 18 * 3 * 3 + \\\\\\\\ \n                            &384 * 192 * 18 * 18 * 3 * 3 + \\\\\\\\ \n                            &384 * 192 * 18 * 18 * 3 * 3 + \\\\\\\\\n                            &192 * 96 * 36 * 36 * 3 * 3 + \\\\\\\\\n                            &384 * 96 * 18 * 18 * 3 * 3 + \\\\\\\\\n                            &96 * 48 * 72 * 72 * 3 * 3 + \\\\\\\\\n                            &48 * 1 * 72 * 72 * 3 * 3 + \\\\\\\\\n                            &192 * 96 * 18 * 18 * 3 * 3 + \\\\\\\\\n                            &96 * 48 * 18 * 18 * 3 * 3 + \\\\\\\\\n                            &48 * 1 * 18 * 18 * 3 * 3 + \\\\\\\\\n                            &96 * 48 * 36 * 36 * 3 * 3 + \\\\\\\\\n                            &48 * 1 * 36 * 36 * 3 * 3) \\\\\\\\\n                      = & 3902587776 \\sim 3.9 B \n\\end{aligned}$$\nFor simplicity, we do not include some operations such as bias term and Layer/Batch-Normalization, which does not affect the overall calculation load level. Besides, the Pyramid Corner Head utilize additional ten interpolation operations. Obviously the calculation load of Py-Corner is still hundreds of times of T4. In fact, `the FLOPs is not equal to running latency due to the parallel computing`, so we directly provide comparison of their running time in the manuscript.\n\nAnother advantage of using T4 is the flexibility to allow us to easily predict the target confidence score with a simple MLP head, which is quite efficient compared to the original score prediction decoder (i.e., SPM) in MixFormer and is more easily deployed on different platforms (since the Precise RoI Pooling in SPM is not supported on CPU). \n\n***C3: How to decide which layer to be eliminated? How to determine the epoch m in the training process. Are these eliminated parameters removed or just not involved in the process flow of the network?***\n\n**R3:** To embody multi-level representations and also reduce the difficulty of feature mimicking during elimination, we choose to eliminate layers uniformly. We have also experimented with eliminating all layers in high level or all layers in low level, but it turned out to perform worse than the employed choice. \n\nAs shown in the table, we find that when the epoch $m$ greater than 40, the choice of $m$ seems hardly affect the performance. So we determine the epoch $m$ to 40.\n|     Arch      | Online | Epoch $m$ | AUC  |\n| :-----------: | :----: | :-------: | :--: |\n| MixFormerV2-B |   no   |    30     | 68.3 |\n| MixFormerV2-B |   no   |    40     | 68.5 |\n| MixFormerV2-B |   no   |    50     | 68.5 |\n\nThese eliminated parameters can be removed directly when the drop rate decreased to zero in both training and inference processes.\n\n***C4: It is suggested to provide intermediate results, such as heatmaps, to better show how the proposed algorithm works.***\n\n**R4:** Thanks for your advice, we will add the intermediate visualization results in our revised manuscript. We have provided the visualization results in the common response pdf.\n\nMixFormerV2 predict the probability distribution of bounding box coordinates, so we plot the output probability distribution as show in Fig. 1.\nWe also visualize the attention heatmaps of four prediction tokens in the last layer of the backbone.\nIt can be observe that the four prediction tokens focus on the four boundaries of the target object, and the coordinates probability distribution highly consists with the corresponding attention heatmaps.\n\nAs shown in Fig. 1(c)(d), there still exist some cases the model is not able to handle perfertly, such as occlusion and similar objects which may cause distribution shift.\n\n***C5: The authors do not discuss the limitations.***\n\n**R5:** We have discussed the limitations in the supplementary material, please check it."}, {"review_id": "jOgEJ4fDVl", "reviewer": "Reviewer_GURa", "summary": "This paper focuses on designing efficient transformer-based single object tracking algorithm. The basic idea is to replace the original corner-based convolution head with a more lightweight MLP head input w/ only 4 learnable tokens.  Moreover, both dense-to-sparse and deep-to-shallow distillations are designed for further improving the efficiency. Experimental results show that the proposed approaches achieve better speed-accuracy trade-off compared to existing transformer-based tracking approaches.", "questions": "No. See Weaknesses.", "limitations": "See Weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and organized, which is easy to understand and follow.\n- Using the distillation technique to achieve better speed-accuracy trade-off is technically sound.\n- The proposed approaches are verified on various large-scale tracking benchmarks. \n- The paper is solid in terms of both illustration and experimental results, which brings something new about how to perform distillation for speeding up existing transformer-based trackers.", "weaknesses": "- What’s the effect of  w/o applying the progressive strategy for depth pruning?\n- It seems that the fast running speed mainly comes from the decrease of layers used in the transformer backbone as illustrated in Table 2. What’s the tracking performance and speed can be achieved if we only apply progressive model depth pruning and a corner prediction head (w/o using dense-to-sparse distillation)? \n- Using the distillation technique to speed up deep models is not a very novel idea, since many attempts are working on distillation for speed up. One good thing is that the authors made some specifical design in VOT like a progressive pruning strategy in PMDP.\n- The speed evaluation is somewhat questionable. In Tables 5-6, is the FPS comparison fair? is the speed (FPS) evaluated on the same platform? What is the GPU platform used for GPU speed and GFLOPs evaluation? Please detail these information in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***C1: What’s the effect of w/o applying the progressive strategy for depth pruning?***\n\n**R1:** As shown in the table, the baseline is the **most usual initialization** method, which employs the first four layers of the teacher (MixViT-B) to initialize the student backbone, denoted as *Tea-fir4*. On the challenging LaSOT_ext dataset, ours PMDP strategy improves it by 1.9% of AUC. \n\nBesides, we also explore an **special initialization** method, named *Tea-skip4*, which chooses the skiped four layers (layer-3/6/9/12) of the teacher (MixViT-B) for initialization. In fact, *Tea-skip4* is an extreme case of ours PMDP when the eliminating epoch $m$ equal to 0. It can be seen that ours PMDP also surpasses *Tea-skip4* by 1.0%, which demonstrate its effectiveness.\n\n|  Method   | LaSOT_ext | UAV123 | LaSOT  |\n| :-------: | :-------: | :----: | :----: |\n| Tea-fir4  |    45.2   |  65.7  |  63.0  |\n| Tea-skip4 |    46.1   |  66.6  |  64.4  |\n|   PMDP    |   **47.1**   |  **67.5**  |  **64.8**  |\n\n***C2: It seems that the fast running speed mainly comes from the decrease of layers used in the transformer backbone as illustrated in Table 2. What’s the tracking performance and speed can be achieved if we only apply progressive model depth pruning and a corner prediction head (w/o using dense-to-sparse distillation)?***\n\n**R2:** As illustrated in Table 1, the proposed token-based head also improves the baseline by 84% of GPU running speed when using score prediction head for temporal-spatial tracking.\n\nThe tracking performance and speed of using PMDP and a plain corner head (not pyramidal corner head) is showcased in the following table. It can be observed that MixFormerV2-B obtains comparable performance with higher running speed (+45 FPS).\n\n|  Backbone |  Head    | LaSOT | UAV123 | FPS(GPU)  |\n| :---------: | :---: | :---: | :----: | :-------: |\n| MixViT-8  |    Plain Corner   |  69.4  |  70.0 | 120 |\n| MixFormerV2-B  |    T4   |  69.5  |  70.5 | 165 |\n\nExcepting for the difference between corner head and the MLP head of MixFormerV2, our T4 design is more flexible. It allows us to easily predict the target confidence score with a simple MLP head, which is quite efficient compared to the original score prediction decoder (i.e., SPM) in MixFormer and is more easily deployed on different platforms (since the Precise RoI Pooling in SPM is not supported on CPU). \n\n***C3: Using the distillation technique to speed up deep models is not a very novel idea.***\n\n**R3:** As you pointed, we have made some specifical design in VOT like the progressive pruning strategy and demonstrated its effectiveness. We think the exploration for distillation in VOT can bring some insprition to the tracking community.\n\n***C4: In Tables 5-6, is the speed (FPS) evaluated on the same platform? What is the GPU platform used for GPU speed and GFLOPs evaluation?***\n\n**R4:** Thanks for your suggestions, and we will revise that in the next version of the manuscript. In Tables 5-6, the speed is evaluated on different platforms since we directly use the reported FPS in their original papers. In the table, we present the FPS of some transformer-based trackers on the same platform.\n\n|  Method |  FPS(GPU)    |\n| :---------: | :---: |\n| STARK-50 | 45 |\n| OSTrack-256 | 95 |\n| MixFormerV2-B | 165 |\n\nWe use a NVidia Quadro RTX 8000 GPU for evaluation."}, {"review_id": "2q8akIZXe7", "reviewer": "Reviewer_fR9F", "summary": "Since main-stream tracking methods are somewhat limited in efficiency, this paper is motivated to develop efficient trackers. The proposed method (named MixFormerV2) is based on the recent tracker MixFormer. The key improvement of architecture is to use a token-based prediction head to replace the corner-based head. Four prediction tokens are used to predict the four coordinates of the bounding box, like the CLS token in ViT. In training, a dense-to-sparse distillation and a deep-to-shallow distillation are employed to further improve trackers’ efficiency. The dense-to-sparse distillation improve the performance of the token-based prediction head, while the deep-to-shallow distillation prune some layers of the backbone. Finally, the proposed MixFormerV2-B and MixformerV2-S achieve good trade-off of performance and speed on GPU and CPU.", "questions": "In rebuttal, I hope to see the response to question 1) in weaknesses. Besides, I suggest revising the paper according to 2) and 3) in weaknesses.", "limitations": "The paper does not provide a discussion on limitations. I recommend providing some failure cases, improvement aspects, etc.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The architecture of the model is simple. The main architecture is a ViT-based backbone with four prediction tokens.\n2) The performance and speed is good. Compared with existing high-performance trackers, MixFormerV2-B achieves competitive performance with higher speed. Compared with existing high-speed trackers, MixformerV2-S achieves state-of-the-art performance.\n3) Employing distillation is novel for tracking. The proposed dense-to-sparse distillation and deep-to-shallow distillation help the development of efficient tracking methods.", "weaknesses": "1) The description of Table 3(e) is not clear. Does Tea-skip4 use PMDP? If not, does its higher performance mean that PMDP is not necessary? If not, why can it achieve similar performance to PMDP?\n2) The LaSOT extension benchmark is not included in comparison. The HCAT method is not included in the comparison. \n3) The amount of parameters is not reported. \n\n[1] Fan H, Bai H, Lin L, Yang F, Chu P, Deng G, Yu S, Huang M, Liu J, Xu Y, et al.. LaSOT: A High-quality Large-scale Single Object Tracking Benchmark. IJCV, 2021\n\n[2] Chen X, Kang B, Wang D, Li D, Lu H. Efficient Visual Tracking via Hierarchical Cross-Attention Tracking via Hierarchical Cross-Attention Transformer. In ECCVW. 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***C1: Does Tea-skip4 use PMDP? If not, does its higher performance mean that PMDP is not necessary? If not, why can it achieve similar performance to PMDP?***\n\n**R1:** *Tea-skip4* is a **special initialization** method, which chooses the skiped four layers (layer-3/6/9/12) of the teacher (MixViT-B) for initialization. In other words, *Tea-skip4* is `an extreme case of ours PMDP when the eliminating epoch m equal to 0`. So it is reasonable that *Tea-skip4* performs better than the baseline *Tea-fir4*, which employs the first four layers of the teacher (MixViT-B) to initialize the student backbone, in the following table. \n\nWe further evaluate the performance on the more challenging LaSOT_ext dataset. It can be seen that ours PMDP surpasses *Tea-skip4* by 1.0%, which demonstrate its effectiveness.\n|  Method   | LaSOT_ext | UAV123 | LaSOT  |\n| :-------: | :-------: | :----: | :----: |\n| Tea-fir4  |    45.2   |  65.7  |  63.0  |\n| Tea-skip4 |    46.1   |  66.6  |  64.4  |\n|   PMDP    |   **47.1**   |  **67.5**  |  **64.8**  |\n\n***C2: The LaSOT extension benchmark is not included in comparison. The HCAT\nmethod is not included in the comparison.***\n\n**R2:** Thanks for your advice, we will add the following LaSOT_ext results and the HCAT comparison in our revision. As in the table, MixFormerV2-B surpasses SwinTrack-B and OSTrack-256 by a large margin with a high running speed.\n\n|  Method   | AUC | Norm P | P  |\n| :-------: | :-------: | :----: | :----: |\n| MixFormerV2-B  |   **50.6**  |  **61.0**  |  **56.9**  |\n| MixFormerV2-S  |  43.6   |  52.7  |  46.2  |\n| SwinTrack-B |    47.6   |  58.2  |  54.1  |\n| OSTrack-256  |   47.4  |  57.3  |  53.3  |\n\n***C3: The amount of parameters is not reported.***\n\n**R3:** We will add the amount of parameters in the revised version. \n\n***C4: The paper does not provide a discussion on limitations. I recommend providing some failure cases, improvement aspects, etc.***\n\n**R4:** We have provided the limitation in the supplementary material. In our revision, we will provide the visualization results of the failure cases and some improvement aspects. In the common response pdf file, we have also provided some visualization examples of the good cases and failure cases."}, {"review_id": "GgirHz58dP", "reviewer": "Reviewer_eZL2", "summary": "This paper proposes an efficient pure-Transformer-based tracking framework MixFormerV2. By replacing dense prediction heads and complex updating-score mapping modules with simple four-box tokens, the tracking pipeline is streamlined with high efficiency. Besides, Dense-to-Sparse Distillation and Deep-to-Shallow Distillation further alleviate the computation burden of transformer architecture. Experiments show that this method achieves a good trade-off between tracking accuracy and speed.", "questions": "- The training time of different distillation strategies is not provided, which reflects the practicability of the proposed method.\n- Experiments evidence the effectiveness of teacher supervision with a similar architecture (i.e., MixFormer). I wonder if it works with the supervision of other Transformer trackers (e.g., OSTrack)?\n", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation that achieving efficient and effective tracking with Transformer is crucial and significant for the balance between performance and speed.\n- Experimental results show that the proposed method effectively improves the inference speed while keeping excellent performance.\n- The writing is clear and easy to follow.\n", "weaknesses": "- The proposed method is only applied on one baseline tracker MixFormer. The generality is not well proved on other Transformer trackers.\n- More detailed analysis about the pruned parameters should be provided, e.g., the curve between the performance and prune ratio.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***C1: The proposed method is only applied on one baseline tracker MixFormer. The generality is not well proved on other Transformer trackers.***\n\n**R1:** Thanks for your suggestion. To demonstrate the generality in rebuttal, we conduct experiments of **dense-to-sparse distillation** (i.e. the prediction token based head) and **deep-to-shallow distillation** on SimTrack[7]. Besides, we also validate the effect of deep-to-shallow distillation on OSTrack. we find that our proposed dense-to-sparse distillation can not be empolyed to OSTrack since the search tokens are gradually eliminated in some layers, which makes the four learnable prediction token hard to capture the object position information.\n\n\n|     Tracker     |   Head   |   Init   | LaSOT_ext | LaSOT |\n| :-------------: | :------: | :------: | :-------: | :---: |\n|  Sim-B/16(12)   | original |    -     |   48.8    | 68.6  |\n| Sim-B/16-T4(12) |  our T4  |    -     |   48.3    | 68.1  |\n| Sim-B/16-T4(8)  |  our T4  | our PMDP |   47.9    | 67.5  |\n| Sim-B/16-T4(8)  |  our T4  | Tea-fir8 |   46.2    | 66.0  |\n\n|     Tracker      |   Head   |   Init   | LaSOT_ext | LaSOT |\n| :--------------: | :------: | :------: | :-------: | :---: |\n| OSTrack-256 (12) | original |    -     |   47.4    | 68.8  |\n| OSTrack-256 (8)  | original | our PMDP |   46.9    | 67.9  |\n| OSTrack-256 (8)  | original | Tea-fir8 |   45.1    | 66.3  |\n\nThrough the results in the above table (without carefully tuning hyper-parameters), we can derive that the proposed prediction token based architecture and the distillation method is effective on other one-stream transformer trackers. \n\n***C2: More detailed analysis about the pruned parameters should be provided, e.g., the curve between the performance and prune ratio.***\n\n**R2:** Thanks for your useful advice, we provide the curve between the performance and prune ratio in the common response  pdf file. It can be observed that the proposed method can consistently improve the performance over baseline.\n\n***C3: The training time of different distillation strategies is not provided.***\n\n**R3:**  Thanks for your suggestion. We will add the training time of different models in our revision. The models are trained on 8 Nvidia RTX 8000 GPUs. The dense-to-sparse stage takes about 43 hours. The deep-to-shallow stage1 (12 to 8) takes about 43 hours, stage2 (8 to 4) takes about 35 hours.\n\n***C4: I wonder if it works with the supervision of other Transformer trackers (e.g., OSTrack)?***\n\n**R4:** We guess you means whether **dense-to-sparse** distillation could be supervised by other transformer trackers.\n\nFor dense-to-sparse distillation, the proposed distribution-based regression is `ought to be supervised with corner prediction head, since it can provide the probability maps for top-left and bottom-right corners`, which has been illustrated in Section 3.2.1. OSTrack is not applicable as it does not have a coerner head . To respond to your concern, we use SimTrack of corner head as the teacher for this analysis. (Note that we did not carefully tune the hyper-parameters due to the limited time.)\n\n|  Tracker   | Teacher |  LaSOT |\n| :-------: | :-------:| :-------: |\n| MixFormerV2(12) | Sim-B/16 | 67.9 |\n| MixFormerV2(12) | MixViT-B | 68.9 |\n\nIt concludes that SimTrack as a teacher can provide a comparable performance to MixViT-B."}]

Author rebuttal:
We thank all reviewers' efforts in reviewing our paper and giving insightful comments and valuable suggestions. We have provided the visualization of intermediate results and curve between performance and pruned depth in the PDF file.

Meta-review: The paper describes an approach for visual object tracking based on transformers. The motivation behind the paper is clear, that is redesigning the network to achieve a better balance between performance and speed. The reviewers mostly asked clarifications about the different components of the method and a few experimental results. The authors provided all the necessary clarifications. All reviewers agree that the paper can be accepted.
Decision: Accept (poster)