SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The overall training pipeline is demonstrated in Fig. 3, performing dense-to-sparse distillation and then deep-to-shallow distillation to yield our final efficient MixFormerV2 tracker. Then, we train the MLP based score head for 50 epochs. Particularly, for CPU real-time tracking, we employ the intermediate teacher to generate a shallower model (4-layer MixFormerV2) based on the proposed distillation. Besides, we also use the designed MLP reduction strategy for further pruning the CPU real-time tracker. The total loss of distillation training with student  $S$  and teacher  $T$  is calculated as:
$$
L = \lambda_ {1} L _ {1} \left(B ^ {S}, B ^ {g t}\right) + \lambda_ {2} L _ {\text {c i o u}} \left(B ^ {S}, B ^ {g t}\right) + \lambda_ {3} L _ {\text {d i s t}} (S, T), \tag {10}
$$
where the first two terms are exactly the same as original MixFormer's location loss supervised by ground truth bounding box labels, and the rest term is for aforementioned distillation.
4.1 Implemented Details
Training and Inference. Our trackers are implemented using Python 3.6 and PyTorch 1.7. The distillation training is conducted on 8 NVidia Quadro RTX 8000 GPUs. The inference process runs on one NVidia Quadro RTX 8000 GPU and Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz. The training datasets include TrackingNet [42], LaSOT [20], GOT-10k [28] and COCO [35] training splits., which are the same as MixFormer [14]. Each distillation training stage takes 500 epochs, where the first  $m = 40$  epochs are for progressively eliminating layers. We train the score prediction MLP for additional 50 epochs. The batch size is 256, each GPU holding 32 samples. We use AdamW optimizer with weight decay of  $10^{-4}$ . The initial learning rate is  $10^{-4}$  and will be decreased to  $10^{-5}$  after 400 epochs. We use horizontal flip and brightness jittering for data augmentation. We instantiate two types of MixFormerV2, including MixFormerV2-B of 8 P-MAM layers for high-speed tracking on GPU platform and MixFormerV2-S of 4 P-MAM layers with MLP ratio of 1.0 for real-time tracking on CPU platform. Their numbers of parameters are 58.8M and 16.2M respectively. The resolutions of search and template images for MixFormerV2-B are  $288 \times 288$  and  $128 \times 128$  respectively. While for MixFormerV2-S, the resolutions of search and template images are  $224 \times 224$  and  $112 \times 112$  for real-time tracking on CPU platform. The inference pipeline is the same as MixFormer [14]. We use the first template together with the current search region as input of MixFormerV2. The dynamic templates are updated when the update interval of 200 is reached by default, where the template with the highest score is selected as an online sample.
Distillation-Based Reduction. For dense-to-sparse distillation, we use MixViT-L as teacher for training MixFormerV2-B by default. We also try to use MixViT-B as the teacher in Tab 5. Particularly, we employ a customized MixViT-B of plain corner head and with search input size of  $224 \times 224$  as the teacher for MixFormer-S. For deep-to-shallow distillation, we use the progressive model depth pruning strategy to produce the 8-layer MixFormerV2-B from a 12-layer one. For MixFormerV2-S, we additionally employs intermediate teacher and MLP reduction strategies, and the process is '12-layers MixFormerV2 to 8-layers MixFormerV2, then 8-layers MixFormerV2 to 4-layers MixFormerV2, finally 4-layers MLP-ratio-4.0 MixFormerV2 to 4-layers MLP-ratio-1.0 MixFormerV2-S'.
4.2 Exploration Studies
To verify the effectiveness of our proposed framework and training paradigm, we analyze different components of MixFormerV2 and perform detailed exploration studies on LaSOT [20] dataset.
4.2.1 Analysis on MixFormerV2 Framework
Token-based Distribution Regression. The design of distribution-based regression with special learnable prediction tokens is the core of our MixFormerV2. We conduct experiments on different regression methods in Tab. 3a. All models employ ViT-B as backbone and are deployed without distillation and online score prediction. Although the pyramidal corner head obtains the best performance, the running speed is largely decreased compared with our token-based regression head in MixFormerV2. MixFormerV2 with four prediction tokens achieves good trade-off between performance and inference latency. Besides, compared to the direct box prediction with one token on the
Type	Layer	AUC	FPS
T1	12	63.1%	112
T4	12	67.5%	110
Py-Corner.	12	69.0%	92
Method	Score.	AUC	FPS
Ours	-	68.9%	166
Ours	✓	70.6%	165
MixViT-B	-	69.0%	92
MixViT-B	✓	69.6%	80
Stu.	Tea.	Tea-AUC
base	-	-	67.5%
base	base	69.0%	68.9%
base	large	71.5%	69.6%
(a) Different regression methods. 'T1' denotes direct box prediction based on one token, 'T4' is the proposed distribution-based prediction with 4 prediction tokens, and PyConrer.' is the pyramidal corner head as in MixViT. Models are without distillation and score prediction.
(b) Quality score prediction. In MixFormerV2, we use token-based MLP head for sample quality score prediction. While in MixViT, it uses an extra SPM for score prediction. We employ MixFormerV2-B of 8 layers, with the MixViT-L as the distillation teacher, for this analysis.
(c) Dense-to-Sparse distillation. The 'base' student denotes the 12-layers MixFormerV2 framework without score prediction. The 'base' teacher is the MixViT-B and the 'large' teacher is the MixViT-L. 'TeaAUC' is the AUC of the teacher. Models are without score prediction.
Log-dis.	Feat-mim.	AUC
-	-	60.7%
✓	-	62.4%
✓	✓	62.9%
Init. method	AUC
MAE-fir4	62.9%
Tea-skip4	64.4%
PMDP	64.8%
Inter. teacher	AUC
-	64.8%
✓	65.5%
(d) Feature mimicking & logits distillation. For distillation analysis, we use the MixViT-B of 12 layers with corner head as the teacher, and MixViT of 4 layers as the student.
(e) Progressive model depth pruning (PMDP). 'MAE-fir4' denotes using first 4 layers of MAE-B for student initialization. 'Tea-skip4' is using 4 skipped layers of the teacher.
(f) Intermediate teacher. For the analysis, we use the 12-layers MixViT-B as the teacher, 8-layers MixViT as the intermediate teacher and 4-layers MixViT as the student.
Epoch m	AUC
30	68.3%
40	68.5%
50	68.5%
blocks num.	head	AUC
12	Py-Corner.	69.0%
12	T4	68.9%
8	T4	68.5%
blocks num.	head	MLP-r	AUC
12	Cor.	4	68.2%
12	T4	4	67.7%
8	T4	4	66.6%
4	T4	4	61.0%
4	T4	1	59.4%
(g) Eliminating Epochs. 'Epoch  $m$  indicates the number of epochs in progressive eliminating process. The model architecture is based on MixFormerV2-B. Models are without score prediction.
(h) Model pruning route of MixFormerV2-B*. 'T4' denotes the proposed distribution-based prediction with 4 prediction tokens. We use the MixViT-B as the distillation teacher for this analysis.
(i) Model pruning route of MixFormerV2-S. 'Cor.' represents for the plain corner head, which is used in the initial teacher model. 'MLP-r' denotes the MLP ratio in attention blocks.
Table 3: Ablation studies on LaSOT.The default choice for our model is colored in gray
first line of Tab. 3a, which estimates the absolute target position instead of the probability distribution of four coordinates, the proposed distribution-based regression obtains better accuracy. Besides, this design allows to perform dense-to-sparse distillation so as to further boost performance.
Token-based Quality Score Prediction. The design of the prediction tokens also allows to perform more efficient quality score prediction via a simple MLP head. As shown in Tab. 3b, the token-based score prediction component improves the baseline MixFormerV2-B by  $1.7\%$  with increasing quite little inference latency. Compared to ours, the score prediction module in MixViT-B further decreases the running speed by  $13.0\%$ , which is inefficient. Besides, the SPM in MixViT requires precise RoI pooling, which hinders the migration to various platforms.
4.2.2 Analysis on Dense-to-Sparse Distillation
We verify the effectiveness of dense-to-sparse distillation in Tab. 3c. When use MixViT-B without its SPM (69.0% AUC) as the teacher model, the MixFormerV2 of 12 P-MAM layers achieves an AUC score of 68.9%, increasing the baseline by 1.4%. This further demonstrates the significance of the design of four special prediction tokens, which allows to perform dense-to-sparse distillation. The setting of using MixViT-L (71.5% AUC) as the teacher model increases the baseline by an AUC score of 2.2%, which implies the good distillation capacity of the large model.
KCF [26]	SiamFC [2]	ATOM [16]	D3Sv2 [38]	DiMP [3]	ToMP [39]	TransT [10]	SBT [49]	SwinTrack [34]	Ours-S	Ours-B
EAO	0.239	0.255	0.386	0.356	0.430	0.511	0.512	0.522	0.524	0.431	0.556
Accuracy	0.542	0.562	0.668	0.521	0.689	0.752	0.781	0.791	0.788	0.715	0.795
Robustness	0.532	0.543	0.716	0.811	0.760	0.818	0.800	0.813	0.803	0.757	0.851
Table 4: State-of-the-art comparison on VOT2022 [30]. The best results are shown in bold font.
4.2.3 Analysis on Deep-to-Shallow Distillation
In the following analysis on deep-to-shallow distillation, we use the MixViT-B of 12 layers with plain corner head as the teacher, and MixViT of 4 layers with the same corner head as the student. The models are deployed without score prediction module.
Feature Mimicking & Logits Distillation. To give detailed analysis on different distillation methods for tracking, we conduct experiments in Tab. 3d. The models are all initialized with the first 4-layers MAE pre-trained ViT-B weights. It can be seen that logits distillation can increase the baseline by  $1.7\%$  AUC, and adding feature mimicking further improves by  $0.4\%$  AUC, which indicates the effectiveness of both feature mimicking and logits distillation for tracking.
Progressive Model Depth Pruning. We study the effectiveness of the progressive model depth pruning (PMDP) for the student initialization in Tab. 8b. It can be observed that the PMDP improves the traditional initialization method of using MAE pre-trained first 4-layers ViT-B by  $1.9\%$ . This demonstrates that it is critical for constraining the initial distribution of student and teacher trackers to be as similar as possible, which can make the feature mimicking easier. Surprisingly, we find that even the initial weights of the four layers are not continuous, i.e., using the skipped layers (the 3,6,9,12-th) of the teacher for initialization, the performance is better than the baseline ( $62.9\%$  vs.  $64.4\%$ ), which further verifies the importance of representation similarity between the two ones.
Determination of Eliminating Epochs. We conduct experiments as shown in the Table 3g to choose the best number of epochs  $m$  in the progressive eliminating period. We find that when the epoch  $m$  greater than 40, the choice of  $m$  seems hardly affect the performance. Accordingly we determine the epoch to be 40.
Intermediate Teacher. Intermediate teacher is introduced to promote the transferring capacity from a deep model to a shallow one. We conduct experiment as in Table 3f. We can observe that the intermediate teacher can bring a gain of  $0.7\%$  AUC score which can verify that.
4.2.4 Model Pruning Route
We present the model pruning route from the teacher model to MixFormerV2-B* and MixFormerV2-S in Tab. 3h and Tab. 3i respectively. The models on the first line are corresponding teacher models. We can see that, through the dense-to-sparse distillation, our token-based MixFormerV2-B obtains comparable accuracy with the dense-corner-based MixViT-B with higher running speed. Through the progressive model depth pruning based on the feature and logits distillation, MixFormerV2-B with 8 layers only decreases little accuracy compared to the 12-layers one.
4.3 Comparison with the Previous Methods
Comparison with State-of-the-art Trackers. We evaluate the performance of our proposed trackers on 6 benchmark datasets: including the large-scale LaSOT [20],  $\mathrm{LaSOT}_{ext}$  [20], TrackingNet [42], UAV123 [41], TNL2K [48] and VOT2022 [30]. LaSOT is a large-scale dataset with 1400 long videos in total and its test set contains 280 sequences. TrackingNet provides over 30K videos with more than 14 million dense bounding box annotations. UAV123 is a large dataset containing 123 aerial videos which is captured from low-altitude UAVs. VOT2022 benchmark has 60 sequences, which measures the Expected Average Overlap (EAO), Accuracy (A) and Robustness (R) metrics. Among them,  $\mathrm{LaSOT}_{ext}$  and TNL2K are two relatively recent benchmarks.  $\mathrm{LaSOT}_{ext}$  is a released extension of LaSOT, which consists of 150 extra videos from 15 object classes. TNL2K consists of 2000 sequences, with natural language description for each. We evaluate our MixFormerV2 on the test set with 700 videos. The results are presented in Tab. 4 and Tab. 5. More results on other datasets will be present in supplementary materials. Only the trackers of similar complexity are
Method	LaSOT	LaSOText	TNL2K	TrackingNet	UAV123	Speed
AUC	PNorm	P	AUC	P	AUC	P	AUC	P	P	AUC	P	GPU
MixFormerV2-B	70.6	80.8	76.2	50.6	56.9	57.4	58.4	83.4	88.1	81.6	69.9	92.1	165
MixFormerV2-B*	69.5	79.1	75.0	-	-	56.6	57.1	82.9	87.6	81.0	70.5	91.9	165
MixFormer [14]	69.2	78.7	74.7	-	-	-	-	83.1	88.1	81.6	70.4	91.8	25
CTTrack-B [45]	67.8	77.8	74.0	-	-	-	-	82.5	87.1	80.3	68.8	89.5	40
OSTrack-256 [55]	69.1	78.7	75.2	47.4	53.3	54.3	-	83.1	87.8	82.0	68.3	-	105
SimTrack-B [7]	69.3	78.5	-	-	-	54.8	53.8	82.3	86.5	-	69.8	89.6	40
CSWinTT [46]	66.2	75.2	70.9	-	-	-	-	81.9	86.7	79.5	70.5	90.3	12
SBT-Base [49]	65.9	-	70.0	-	-	-	-	-	-	-	-	-	37
SwinTrack-T [34]	67.2	-	70.8	47.6	53.9	53.0	53.2	81.1	-	78.4	-	-	98
ToMP101 [39]	68.5	79.2	68.5	-	-	-	-	81.5	86.4	78.9	66.9	-	20
STARK-ST50 [52]	66.4	-	-	-	-	-	-	81.3	86.1	-	-	-	42
KeepTrack [40]	67.1	77.2	70.2	48.2	-	-	-	-	-	-	69.7	-	19
TransT [10]	64.9	73.8	69.0	-	-	50.7	51.7	81.4	86.7	80.3	69.1	-	50
PrDiMP [17]	59.8	68.8	60.8	-	-	-	-	75.8	81.6	70.4	68.0	-	47
ATOM [16]	51.5	57.6	50.5	-	-	-	-	70.3	77.1	64.8	64.3	-	83
Table 5: State-of-the-art comparison on TrackingNet [42], LaSOT [20],  $\mathrm{LaSOT}_{ext}$  [20], UAV123 [41] and TNL2K [48]. The best two results are shown in **bold** and **underline** fonts. * denotes tracker with MixViT-B as the teacher during the dense-to-sparse distillation process. The default teacher is MixViT-L. Only trackers of similar complexity are included.
Method	LaSOT	LaSOText	TNL2K	TrackingNet	UAV123	Speed
AUC	PNorm	P	AUC	P	AUC	P	AUC	PNorm	P	AUC	P	GPU	CPU
MixFormerV2-S	60.6	69.9	60.4	43.6	46.2	48.3	43	75.8	81.1	70.4	65.8	86.8	325	30
FEAR-L [5]	57.9	68.6	60.9	-	-	-	-	-	-	-	-	-	-	-
FEAR-XS [5]	53.5	64.1	54.5	-	-	-	-	-	-	-	-	-	80	26
HCAT[9]	59.0	68.3	60.5	-	-	-	-	76.6	82.6	72.9	63.6	-	195	45
E.T.Track [4]	59.1	-	-	-	-	-	-	74.5	80.3	70.6	62.3	-	150	42
LightTrack-LargeA [53]	55.5	-	56.1	-	-	-	-	73.6	78.8	70.0	-	-	-	-
LightTrack-Mobile [53]	53.8	-	53.7	-	-	-	-	72.5	77.9	69.5	-	-	120	36
STARK-Lighting	58.6	69.0	57.9	-	-	-	-	-	-	-	-	-	200	42
DiMP [3]	56.9	65.0	56.7	-	-	-	-	74.0	80.1	68.7	65.4	-	77	15
SiamFC++ [50]	54.4	62.3	54.7	-	-	-	-	75.4	80.0	70.5	-	-	90	20
Table 6: Comparison with CPU-realtime trackers on TrackingNet [42], LaSOT [20], LaSOT<sub>ext</sub> [20], UAV123 [41] and TNL2k [48]. The best results are shown in **bold** fonts.
included, i.e., the trackers with large-scale backbone or large input resolution are excluded. Our MixFormerV2-B achieves state-of-the-art performance among these trackers with a very fast speed, especially compared to transformer-based one-stream tracker. For example, MixFormerV2-B without post-processing strategies surpasses OSTrack by  $1.5\%$  AUC on LaSOT and  $2.4\%$  AUC on TNL2k, running with quite faster speed (165 FPS vs. 105 FPS). Even the MixFormerV2-B with MixViT-B as the teacher model obtains better performance than existing SOTA trackers, such as MixFormer, OSTrack, ToMP101 and SimTrack, with much faster running speed on GPU.
Comparison with Efficient Trackers. For real-time running requirements on limited computing resources such as CPU, we explore a lightweight model, i.e. MixFormerV2-S, which still reaches strong performance. And it is worth noting that this is the first time that transformer-based one-stream tracker is able to run on CPU device with a real-time speed. As demonstrated in Figure 6, MixFormerV2-S surpasses all other architectures of CPU-real-time trackers by a large margin. We take a comparison with other prevailing efficient trackers on multiple datasets, including LaSOT, TrackingNet, UAV123 and TNL2k, in Tab 6. We can see that our MixFormerV2-S outperforms FEAR-L by a an AUC score of  $2.7\%$  and STARK-Lightning by an AUC score of  $2.0\%$  on LaSOT.
In this paper, we have proposed a fully transformer tracking framework MixFormerV2, composed of standard ViT backbones on the mixed token sequence and simple MLP heads for box regression and quality score estimation. Our MixFormerV2 streamlines the tracking pipeline by removing the dense convolutional head and the complex score prediction modules. We also present a distillation based model reduction paradigm for MixFormerV2 to further improve its efficiency. Our MixFormerV2 obtains a good trade-off between tracking accuracy and speed on both GPU and CPU platforms. We hope our MixFormerV2 can facilitate the development of efficient transformer trackers in the future.
This work is supported by National Key R&D Program of China (No. 2022ZD0160900), National Natural Science Foundation of China (No. 62076119, No. 61921006), Fundamental Research Funds for the Central Universities (No. 020214380099), and Collaborative Innovation Center of Novel Software Technology and Industrialization.
2021.
[49] Fei Xie, Chunyu Wang, Guangting Wang, Yue Cao, Wankou Yang, and Wenjun Zeng. Correlation-aware deep tracking. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2022.
[50] Yinda Xu, Zeyu Wang, Zuoxin Li, Ye Yuan, and Gang Yu. Siamfc++: Towards robust and accurate visual tracking with target estimation guidelines. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, 2020.
[51] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2964–2972, 2022.
[52] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2021.
[53] Bin Yan, Houwen Peng, Kan Wu, Dong Wang, Jianlong Fu, and Huchuan Lu. Lighttrack: Finding lightweight neural networks for object tracking via one-shot architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180-15189, 2021.
[54] Zhendong Yang, Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. Vitkd: Practical guidelines for vit feature knowledge distillation. arXiv preprint arXiv:2209.02432, 2022.
[55] Botao Ye, Hong Chang, Bingpeng Ma, and Shiguang Shan. Joint feature learning and relation modeling for tracking: A one-stream framework. Proceedings of the European Conference on Computer Vision, ECCV, 2022.
[56] Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vision transformers with weight multiplexing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12145-12154, 2022.
[57] Zhaohui Zheng, Rongguang Ye, Qibin Hou, Dongwei Ren, Ping Wang, Wangmeng Zuo, and Ming-Ming Cheng. Localization distillation for object detection. arXiv preprint arXiv:2204.05957, 2022.
In this paper, we introduce MixFormerV2, a fully transformer tracking approach for efficiently and effectively estimating the state of an arbitrary target in a video. Generic object tracking is one of the fundamental computer vision problems with numerous applications. For example, object tracking (and hence MixFormerV2) could be applied to human-machine interaction, visual surveillance and unmanned vehicles. Our research could be used to improve the tracking performance while maintaining a high running speed. Of particular concern is the use of the tracker by those wishing to position and surveil others illegally. Besides, if the tracker is used in unmanned vehicles, it may be a challenge when facing the complex real-world scenarios. To mitigate the risks associated with using MixFormerV2, we encourage researchers to understand the impacts of using the trackers in particular real-world scenarios.
The main limitation lies in the training overhead of MixFormerV2-S, which performs multiple model pruning based on the dense-to-sparse distillation and deep-to-shallow distillation. In detail, we first perform distillation from MixViT with 12 layers and plain corner head to MixFormerV2 of 12 layers. The 12-layers MixFormerV2 is pruned to 8-layers and then to 4-layers MixFormerV2 based on the deep-to-shallow distillation. Finally, the MLP-ratio-4.0 4-layers MixFormerV2 is pruned to the MLP-ratio-4.0 4-layers MixFormerV2-S for real-time tracking on CPU. For each step, it requires training for 500 epochs which is time-consuming.
S.1 Details of Training Time
The models are trained on 8 Nvidia RTX8000 GPUs. The dense-to-sparse stage takes about 43 hours. The deep-to-shallow stage1 (12-to-8 layers) takes about 42 hours, and stage2 (8-to-4 layers) takes about 35 hours.
S.2 More Results on VOT2020 and GOT10k
VOT2020. We evaluate our tracker on VOT2020 [29] benchmark, which consists of 60 videos with several challenges including fast motion, occlusion, etc. The results is reported in Table 7, with metrics Expected Average Overlap(EAO) considering both Accuracy(A) and Robustness. Our MixFormerV2-B obtains an EAO score of 0.322 surpassing CSWinTT by  $1.8\%$ . Besides, our MixFormerV2-S achieves an EAO score of 0.258, which is higher than the efficient tracker LightTrack, with a real-time running speed on CPU.
GOT10k. GOT10k [28] is a large-scale dataset with over 10000 video segments and has 180 segments for the test set. Apart from generic classes of moving objects and motion patterns, the object classes in the train and test set are zero-overlapped. We evaluate MixFormerV2 trained with the four datasets of LaSOT, TrackingNet, COCO and GOT10k-train on GOT10k-test. We compare it with MixFormer and TransT with the same training datasets for fair comparison. MixFormerV2-B improves MixFormer and TransT by  $0.7\%$  and  $1.6\%$  on AO respectively with a high running speed of 165 FPS.
KCF [26]	SiamFC [2]	ATOM [16]	LightTrack [53]	DiMP [3]	STARK [52]	TransT [10]	CSWinTT [46]	MixFormer [14]	Ours-S	Ours-B
VOT20EAO	0.154	0.179	0.271	0.242	0.274	0.280	-	0.304	-	0.258	0.322
GOT10kAO	0.203	0.348	0.556	-	0.611	0.688	0.723*	0.694	0.726*	0.621*	0.739*
Table 7: State-of-the-art comparison on VOT2020 [29] and GOT10k [28]. * denotes training with four datasets including LaSOT [20], TrackingNet [42], GOT10k [28] and COCO [35]. The best results are shown in bold font.
S.3 More Ablation Studies
Design of Prediction Tokens. We practice three different designs of prediction tokens for the target localization in Tab. 8a. All the three methods use the formulation of estimating the probability
token num.	MLP num.	AUC
1	4	67.1%
4	4	67.3
4	1	67.5%
(a) Different prediction designs. 'token num.' indicates the number of the learnable prediction tokens, 'MLP num.' denotes the number of employed MLP layers for localization.
Init. method	LaSOT	LaSOT_ext	UAV123
Tea-fir4	62.9%	45.2%	65.7%
Tea-skip4	64.4%	46.1%	66.6%
PMDP	64.8%	47.1%	67.5%
(b) Progressive Model Depth Pruning (PMDP). 'Tea-fir4' denotes using first 4 layers of the teacher for student initialization. 'Teaskip4' is using 4 skipped layers of the teacher.
Table 8: More ablation studies. The default choice for our model is colored in gray .
distribution of the four coordinates of the bounding box. The model on the first line denotes using one prediction token and then predicting coordinates distribution with four independent MLP heads. It can be observed that adopting separate prediction tokens for the four coordinates and a same MLP head retains the best accuracy.