SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yutao Cui† Tianhui Song† Gangshan Wu Limin Wang*  
State Key Laboratory for Novel Software Technology, Nanjing University, China
https://github.com/MCG-NJU/MixFormerV2
Transformer-based trackers have achieved high accuracy on standard benchmarks. However, their efficiency remains an obstacle to practical deployment on both GPU and CPU platforms. In this paper, to mitigate this issue, we propose a fully transformer tracking framework based on the successful MixFormer tracker [14], coined as MixFormerV2, without any dense convolutional operation or complex score prediction module. We introduce four special prediction tokens and concatenate them with those from target template and search area. Then, we apply a unified transformer backbone on these mixed token sequence. These prediction tokens are able to capture the complex correlation between target template and search area via mixed attentions. Based on them, we can easily predict the tracking box and estimate its confidence score through simple MLP heads. To further improve the efficiency of MixFormerV2, we present a new distillation-based model reduction paradigm, including dense-to-sparse distillation and deep-to-shallow distillation. The former one aims to transfer knowledge from the dense-head based MixViT to our fully transformer tracker, while the latter one is for pruning the backbone layers. We instantiate two MixForemrV2 trackers, where the MixFormerV2-B achieves an AUC of  $70.6\%$  on LaSOT and AUC of  $56.7\%$  on TNL2k with a high GPU speed of 165 FPS, and the MixFormerV2-S surpasses FEAR-L by  $2.7\%$  AUC on LaSOT with a real-time CPU speed.
Visual object tracking has been a fundamental and long-standing task in computer vision, which aims to locate the object in a video sequence, given its initial bounding box. It has a wide range of practical applications, which often require for low computational latency. So it is important to design a more efficient tracking architecture while maintaining high accuracy.
Recently, the transformer-based one-stream trackers [7, 14, 55] attain excellent tracking accuracy than the previous Siamese-based ones [2, 10, 11], due to the unified modeling of feature extraction and target integration within a transformer block, which allows both components to benefit from the transformer development (e.g. ViT [18], self-supervised pre-training [24] or contrastive pretraining [43]). However for these trackers, inference efficiency, especially on CPU, is still the main obstacle to practical deployment. Taking the state-of-the-art tracker MixViT [15] as an instance, its pipeline contains i) transformer backbone on the token sequence from target template and search area, ii) dense corner head on the 2D search region for regression and iii) extra complex score prediction module for classification (i.e., estimating the box quality for reliable online samples selection). To achieve a high-efficiency tracker, there are still several issues on the design of MixViT. First, the dense convolutional corner head still exhibits a time-consuming design, as implied in Tab 1. This is because it densely estimates the probability distribution of the box corners through a total of
† Equal contribution. * Corresponding author (lmwang@nju.edu.cn).
Layer	Head	Score	GPU FPS	GFLOPs
8	Pyram. Corner	✓	90	27.2
8	Pyram. Corner	-	120(↑ 33.3%)	26.2
8	Token-based	-	166(↑ 84.4%)	22.5
Table 1: Efficiency analysis on MixViT-B with different heads. 'Pyram. Corner' represents for the pyramidal corner head [15].
Layer	MLP Ratio	Image Size	CPU FPS	GPU FPS
4	1	288	21	262
224	30	280
4	288	12	255
224	15	275
8	1	288	12	180
224	16	190
4	288	7	150
224	8	190
12	1	288	8	130
224	12	145
4	288	4	100
224	6	140
Table 2: Efficiency analysis on MixViT-B with different backbone settings. The employed prediction head is plain corner head [14] for the analysis.
Figure 1: Comparison with state-of-the-art trackers in terms of AUC performance, model Flops and GPU Speed on LaSOT. The circle diameter is in proportion to model flops. MixFormerV2-B surpasses existing trackers by a large margin in terms of both accuracy and inference speed. MixFormerV2-S achieves extremely high tracking speed of over 300 FPS while obtaining competitive accuracy compared with other efficient trackers [4, 5].
ten convolutional layers on the high-resolution 2D feature maps. Second, to deal with online template updating, an extra complex score prediction module composed of precise RoI pooling layer, two attention blocks, and a three-layer MLP is required for improving online samples quality, which largely hinders its efficiency and simplicity of MixViT.
To avoid the dense corner head and complicated score prediction module, we propose a new fully transformer tracking framework—MixFormerV2 without any dense convolutional operation. Our MixFormerV2 yields a very simple and efficient architecture, which is composed of a transformer backbone on the mixed token sequence and two simple MLP heads on the learnable prediction tokens. Specifically, we introduce four special learnable prediction tokens and concatenate them with the original tokens from target template and search area. Like the CLS token in standard ViT, these prediction tokens are able to capture the complex relation between target template and search area, serving as a compact representation for subsequent regression and classification. Based on them, we can easily predict the target box and confidence score through simple MLP heads, which results in an efficient fully transformer tracker. Our MLP heads directly regress the probability distribution of four box coordinates, which improves the regression accuracy without increasing overhead.
To further improve efficiency of MixFormerV2, we present a new model reduction paradigm based on distillation, including dense-to-sparse distillation and deep-to-shallow distillation. The dense-to-sparse distillation aims to transfer knowledge from the dense-head based MixViT, to our fully transformer tracker. Thanks to the distribution-based regression design in our MLP head, we can easily adopt logits mimicking strategy for distilling MixViT trackers to our MixFormerV2. Based on the observation in Tab. 2, we also exploit the deep-to-shallow distillation to prune our MixFormerV2. We devise a new progressive depth pruning strategy by following a critical principle that constraining the initial distribution of student and teacher trackers to be as similar as possible, which can augment the capacity of transferring knowledge. Specifically, instructed by the frozen teacher model, some certain layers of a copied teacher model are progressively dropped and we use the pruned model as our student initialization. For CPU-realtime tracking, we further introduce an intermediate teacher model to bridge the gap between the large teacher and small student, and prune hidden dim of MLP based on the proposed distillation paradigm.
Based on the proposed model reduction paradigm, we instantiate two types of MixFormerV2 trackers, MixFormerV2-B and MixFormerV2-S. As shown in Fig. 1, MixFormerV2 achieves better trade-off between tracking accuracy and inference speed than previous trackers. Especially, MixFormerV2-B achieves an AUC of  $70.6\%$  on LaSOT with a high GPU speed of 165 FPS, and MixFormerV2-S outperforms FEAR-L by  $2.7\%$  AUC on LaSOT with a real-time CPU speed. Our contributions are two-fold: 1) We propose the first fully transformer tracking framework without any convolution operation, dubbed as MixFormerV2, yielding a more unified and efficient tracker. 2) We present a
new distillation-based model reduction paradigm to make MixFormerV2 more effective and efficient, which can achieve high-performance tracking on platforms with GPUs or CPUs.
Efficient Visual Object Tracking. In recent decades, the visual object tracking task has witnessed rapid development due to the emergence of new benchmark datasets[20, 28, 41, 42, 48] and better trackers [2, 10, 12-14, 32, 52, 55]. Researchers have tried to explore efficient and effective tracking architectures for practical applications, such as siamese-based trackers [2, 31, 32, 50], online trackers [3, 16] and transformer-based trackers [10, 37, 52]. Benefiting from transformer structure and attention mechanism, recent works [7, 14, 55] on visual tracking are gradually abandoning traditional three-stage model paradigm, i.e., feature extraction, information interaction and location head. They adopted a more unified one-stream model structure to jointly perform feature extraction and interaction, which turned out to be effective for modeling visual object tracking task. However, some modern tracking architectures are too heavy and computational expensive, making it hard to deploy in practical applications. LightTrack [53] employed NAS to search the a light Siamese network, but its speed was not extremely fast on powerful GPUs. FEAR [5], HCAT [9], E.T.Track [4] designed more efficient framework, however were not suitable for one-stream trackers. We are the first to design efficient one-stream tracker so as to achieve good accuracy and speed trade-off.
Knowledge Distillation. Knowledge Distillation [27] was proposed to learn more effective student models with teacher model's supervision. In the beginning, KD is applied in classification problem, where KL divergence is used for measuring the similarity of teacher's and student's predicted probability distribution. For regression problem like object detection, feature mimicking [1, 23, 33] is frequently employed. LD [57] operate logits distillation on bounding box location by converting Dirac delta distribution representation to probability distribution representation of bounding box, which well unifies logits distillation and location distillation. In this work, we exploit some customized strategies to make knowledge distillation more suitable for our tracking framework.
Vision Transformer Compression. There exist many general techniques for the purpose of speeding up model inference, including model quantization [22, 47], knowledge distillation [27, 36], pruning [25], and neural architecture search [19]. Recently many works also focus on compressing vision transformer models. For example, Dynamic ViT [44], Evo-ViT [51] tried to prune tokens in attention mechanism. AutoFormer [8], NASViT [21], SlimmingViT [6] employed NAS technique to explore delicate ViT architecture. ViTKD [54] provided several ViT feature distillation guidelines but it focused on compressing the feature dimension instead of model depth. MiniViT [56] applied weights sharing and multiplexing to reduce model parameters. Since one-stream trackers highly rely on training-resource-consuming pre-training, we resort to directly prune the layers of our tracker.
In this section, we first present the MixFormerV2, which is a more efficient and unified fully transformer tracking framework. Then we describe the proposed distillation-based model reduction, including dense-to-sparse distillation and deep-to-shallow distillation.
3.1 Fully Transformer Tracking: MixFormerV2
The proposed MixFormerV2 is a fully transformer tracking framework without any convolutional operation and complex score prediction module. Its backbone is a plain transformer on the mixed token sequence of three types: target template token, search area token, and learnable prediction token. Then, simple MLP heads are placed on top for predicting probability distribution of the box coordinates and corresponding target quality score. Compared with other transformer-based trackers (e.g. TransT [10], STARK [52], MixFormer [14], OSTrack [55] and SimTrack [7]), our MixFormerV2 streamlines the tracking pipeline by effectively removing the customized convolutional classification and regression heads for the first time, which yields a more unified, efficient and flexible tracker. The overall architecture is depicted in Fig. 2. With inputting the template tokens, the search area tokens and learnable prediction tokens, MixFormerV2 predicts the target bounding boxes and quality score in an end-to-end manner.
Figure 2: MixFormerV2 Framework. MixFormerV2 is a fully transformer tracking framework, composed of a transformer backbone and two simple MLP heads on the learnable prediction tokens.
Prediction-Token-Involved Mixed Attention. Compared to original slimming mixed attention [15] in MixViT, the key difference lies in the introduction of the special learnable prediction tokens, which are used to capture the correlation between the target template and search area. These prediction tokens can progressively compress the target information and used as a compact representations for subsequent regression and classification. Specifically, given the concatenated tokens of multiple templates, search and four learnable prediction tokens, we pass them into  $N$  layers of prediction-token-involved mixed attention modules (P-MAM). We use  $q_{t}$ ,  $k_{t}$  and  $v_{t}$  to represent template elements (i.e. query, key and value) of attention,  $q_{s}$ ,  $k_{s}$  and  $v_{s}$  to represent search region,  $q_{e}$ ,  $k_{e}$  and  $v_{e}$  to represent learnable prediction tokens. The P-MAM can be defined as:
$$
k _ {t s e} = \operatorname {C o n c a t} \left(k _ {t}, k _ {s}, k _ {e}\right), \quad v _ {t s e} = \operatorname {C o n c a t} \left(v _ {t}, v _ {s}, v _ {e}\right),
$$
$$
\operatorname {A t t e n} _ {\mathrm {t}} = \operatorname {S o f t m a x} \left(\frac {q _ {t} k _ {t} ^ {T}}{\sqrt {d}}\right) v _ {t}, \operatorname {A t t e n} _ {s} = \operatorname {S o f t m a x} \left(\frac {q _ {s} k _ {t s e} ^ {T}}{\sqrt {d}}\right) v _ {t s e}, \operatorname {A t t e n} _ {e} = \operatorname {S o f t m a x} \left(\frac {q _ {e} k _ {t s e} ^ {T}}{\sqrt {d}}\right) v _ {t s e} \tag {1}
$$
where  $d$  represents the dimension of each element,  $\mathrm{Attent}_t$ ,  $\mathrm{Attent}_s$  and  $\mathrm{Attene}_e$  are the attention output of the template, search and the learnable prediction tokens respectively. Similar to the original MixFormer, we use the asymmetric mixed attention scheme for efficient online inference. Like the CLS tokens in standard ViT, the learnable prediction tokens automatically learn on the tracking dataset to compress both the template and search information.
Direct Prediction Based on Tokens. After the transformer backbone, we directly use the prediction tokens to regress the target location and estimate its reliable score. Specifically, we exploit the distribution-based regression based on the four special learnable prediction tokens. In this sense, we regress the probability distribution of the four bounding box coordinates rather than their absolute positions. Experimental results in Section 4.2 also validate the effectiveness of this design. As the prediction tokens can compress target-aware information via the prediction-token-involved mixed attention modules, we can simply predict the four box coordinates with a same MLP head as follows:
$$
\hat {P} _ {X} (x) = \operatorname {M L P} \left(\operatorname {t o k e n} _ {X}\right), X \in \{\mathcal {T}, \mathcal {L}, \mathcal {B}, \mathcal {R} \}. \tag {2}
$$
In implementation, we share the MLP weights among four prediction tokens. For predicted target quality assessment, the Score Head is a simple MLP composed of two linear layers. Specifically, firstly we average these four prediction tokens to gather the target information, and then feed the token into the MLP-based Score Head to directly predict the confidence score  $s$  which is a real number. Formally, we can represent it as:
$$
s = \operatorname {M L P} \left(\operatorname {m e a n} \left(\operatorname {t o k e n} _ {X}\right)\right), X \in \{\mathcal {T}, \mathcal {L}, \mathcal {B}, \mathcal {R} \}.
$$
These token-based heads largely reduces the complexity for both the box estimation and quality score estimation, which leads to a more simple and unified tracking architecture.
Figure 3: Distillation-Based Model Reduction for Mix-FormerV2. The 'Stage1' represents for the dense-to-sparse distillation, while the 'Stage2' is the deep-to-shallow distillation. The blocks with orange arrows are to be supervised and blocks with dotted line are to be eliminated.
Figure 4: Progressive Depth Pruning Process for eliminating blocks. All weights in this block decay to zeros and finally only residual connection works, turning into an identity block.
3.2 Distillation-Based Model Reduction
To further improve the efficiency and effectiveness of MixFormerV2, we present a distillation-based model reduction paradigm as shown in Fig. 3, which first perform dense-to-sparse distillation for better token-based prediction and then deep-to-shallow distillation for the model pruning.
3.2.1 Dense-to-Sparse Distillation
In MixFormerV2, we directly regress the target bounding box based on the prediction tokens to the distribution of four random variables  $\mathcal{T},\mathcal{L},\mathcal{B},\mathcal{R}\in \mathbb{R}$ , which represents the box's top, left, bottom and right coordinate respectively. In detail, we predict the probability density function of each coordinate:  $X\sim \hat{P}_X(x)$ , where  $X\in \{\mathcal{T},\mathcal{L},\mathcal{B},\mathcal{R}\}$ . The final bounding box coordinates  $B$  can be derived from the expectation over the regressed probability distribution:
$$
B _ {X} = \mathbb {E} _ {\hat {P} _ {X}} [ X ] = \int_ {\mathbb {R}} x \hat {P} _ {X} (x) \mathrm {d} x. \tag {3}
$$
Since the original MixViT's dense convolutional corner heads predict two-dimensional probability maps, i.e. the joint distribution  $P_{\mathcal{T}\mathcal{L}}(x,y)$  and  $P_{\mathcal{B}\mathcal{R}}(x,y)$  for top-left and bottom-right corners, the one-dimensional version of box coordinates distribution can be deduced easily through marginal distribution:
$$
P _ {\mathcal {T}} (x) = \int_ {\mathbb {R}} P _ {\mathcal {T L}} (x, y) \mathrm {d} y, \quad P _ {\mathcal {L}} (y) = \int_ {\mathbb {R}} P _ {\mathcal {T L}} (x, y) \mathrm {d} x \tag {4}
$$
$$
P _ {\mathcal {B}} (x) = \int_ {\mathbb {R}} P _ {\mathcal {B R}} (x, y) \mathrm {d} y, \quad P _ {\mathcal {R}} (y) = \int_ {\mathbb {R}} P _ {\mathcal {B R}} (x, y) \mathrm {d} x.
$$
Herein, this modeling approach can bridge the gap between the dense corner prediction and our sparse token-based prediction, i.e., and the regression outputs of original MixViT can be regarded as soft labels for dense-to-sparse distillation. Specifically, we use MixViT's outputs  $P_{X}$  as in Equation 4 for supervising the four coordinates estimation  $\hat{P}_{X}$  of MixFormerV2, applying KL-Divergence loss as follows:
$$
L _ {\mathrm {l o c}} = \sum_ {X \in \{\mathcal {T}, \mathcal {L}, \mathcal {B}, \mathcal {R} \}} L _ {\mathrm {K L}} \left(\hat {P} _ {X}, P _ {X}\right). \tag {5}
$$
In this way, the localization knowledge is transferred from the dense corner head of MixViT to the sparse token-based head of MixFormerV2.
3.2.2 Deep-to-Shallow Distillation
For further improving efficiency, we focus on pruning the transformer backbone. However, designing a new light-weight backbone is not suitable for fast one-stream tracking. A new backbone of one-stream trackers often highly relies on large-scale pre-training to achieve good performance, which requires for huge amounts of computation. Therefore, we resort to directly cut down some layers
of MixFormerV2 backbone based on both the feature mimicking and logits distillation, as can be seen in Fig. 3:Stage2. Let  $F_{i}^{S}, F_{j}^{T} \in \mathbb{R}^{h \times w \times c}$  denote the feature map from student and teacher, the subscript represents the index of layers. For logits distillation, we use KL-Divergence loss. For feature imitation, we apply  $L_{2}$  loss:
$$
L _ {\text {f e a t}} = \sum_ {(i, j) \in \mathcal {M}} L _ {2} \left(F _ {i} ^ {S}, F _ {j} ^ {T}\right), \tag {6}
$$
where  $\mathcal{M}$  is the set of matched layer pairs need to be supervised. Specifically, we design a progressive model depth pruning strategy for distillation.
Progressive Model Depth Pruning. Progressive Model Depth Pruning aims to compress Mix-FormerV2 backbone through reducing the number of transformer layers. Since directly removing some layers could lead to inconsistency and discontinuity, we explore a progressive method for model depth pruning based on the feature and logits distillation. Specifically, instead of letting teacher to supervise a smaller student model from scratch, we make the original student model a complete copy of the teacher model. Then, we will progressively eliminate certain layers of student and make the remaining layers to mimic teacher's representation during training with supervision of teacher. This design allows the initial representation of student and teacher to keep as consistent as possible, providing a smooth transition scheme and reducing the difficulty of feature mimicking.
Formally, let  $x_{i}$  denote output of the  $i$ -th layer of MixFormerV2 backbone, the calculation of attention block can be represented as below (Layer-Normalization operation is omitted in equation):
$$
\begin{array}{l} x _ {i} ^ {\prime} = \operatorname {A T T N} \left(x _ {i - 1}\right) + x _ {i - 1}, \\ x _ {i} = \operatorname {F F N} \left(x _ {i} ^ {\prime}\right) + x _ {i} ^ {\prime} \tag {7} \\ = \operatorname {F F N} (\operatorname {A T T N} (x _ {i - 1}) + x _ {i - 1}) + \operatorname {A T T N} (x _ {i - 1}) + x _ {i - 1}, \\ \end{array}
$$
Let  $\mathcal{E}$  be the set of layers to be eliminated in our student network, we apply a decay rate  $\gamma$  on the weights of these layers:
$$
x _ {i} = \gamma (\operatorname {F F N} (\operatorname {A T T N} (x _ {i - 1}) + x _ {i - 1}) + \operatorname {A T T N} (x _ {i - 1})) + x _ {i - 1}, i \in \mathcal {E}. \tag {8}
$$
During the first  $m$  epochs of student network training,  $\gamma$  will gradually decrease from 1 to 0 in the manner of cosine function:
$$
\gamma (t) = \left\{ \begin{array}{l l} 0. 5 \times \left(1 + \cos \left(\frac {t}{m} \pi\right)\right), & t \leq m, \\ 0, & t > m. \end{array} \right. \tag {9}
$$
This means these layers in student network are gradually eliminated and finally turn into identity transformation, as depicted in Figure 4. The pruned student model can be obtained by simply removing layers in  $\mathcal{E}$  and keeping the remaining blocks.
Intermediate Teacher. For distillation of an extremely shallow model (4-layers MixFormerV2), we introduce an intermediate teacher (8-layers MixFormerV2) for bridging the deep teacher (12-layers MixFormerV2) and the shallow one. Typically, the knowledge of teacher may be too complex for a small student model to learn. So we introduce an intermediate role serving as teaching assistant to relieve the difficulty of the extreme knowledge distillation. In this sense, we divide the problem of knowledge distillation between teacher and small student into several distillation sub-problems.
MLP Reduction. As shown in Table 2, one key factor affecting the inference latency of tracker on CPU device is the hidden feature dim of MLP in Transformer block. In other words, it becomes the bottleneck that limits the real-time speed on CPU device. In order to leverage this issue, we further prune the hidden dim of MLP based on the proposed distillation paradigm, i.e., feature mimicking and logits distillation. Specifically, let the shape of linear weights in the original model is  $w \in \mathbb{R}^{d_1 \times d_2}$ , and the corresponding shape in the pruning student model is  $w' \in \mathbb{R}^{d_1' \times d_2'}$ , in which  $d_1' \leq d_1$ ,  $d_2' \leq d_2$ , we will initialize weights for student model as:  $w' = w[: d_1', : d_2']$ . Then we apply distillation supervision for training, letting the pruned MLP to simulate original heavy MLP.
3.3 Training of MixFormerV2