General Compression Framework for Efficient Transformer Object Tracking

## Abstract

Abstract Transformer-based trackers have established a dominant role in the field of visual object tracking. While these trackers exhibit promising performance, their deployment on resource-constrained devices remains challenging due to inefficiencies. To improve the inference efficiency and reduce the computation cost, prior approaches have aimed to either design lightweight trackers or distill knowledge from larger teacher models into more compact student trackers. However, these solutions often sacrifice accuracy for speed. Thus, we propose a general model compression framework for efficient transformer object tracking, named CompressTracker, to reduce the size of a pre-trained tracking model into a lightweight tracker with minimal performance degradation. Our approach features a novel stage division strategy that segments the transformer layers of the teacher model into distinct stages, enabling the student model to emulate each corresponding teacher stage more effectively. Additionally, we also design a unique replacement training technique that involves randomly substituting specific stages in the student model with those from the teacher model, as opposed to training the student model in isolation. Replacement training enhances the student model’s ability to replicate the teacher model’s behavior. To further forcing student model to emulate teacher model, we incorporate prediction guidance and stage-wise feature mimicking to provide additional supervision during the teacher model’s compression process. Our framework CompressTracker is structurally agnostic, making it compatible with any transformer architecture. We conduct a series of experiment to verify the effectiveness and generalizability of CompressTracker. Our CompressTracker-4 with 4 transformer layers, which is compressed from OSTrack, retains about 𝟗𝟔 % percent 96 \mathbf{96\%} bold_96 % performance on LaSOT ( 66.1 % percent 66.1 \mathbf{66.1\%} bold_66.1 % AUC) while achieves 2.17 × \mathbf{2.17\times} bold_2.17 × speed up.

## 1 Introduction

Figure: Figure 1: We apply our framework to OSTrack under several different layer configurations. (a) We implement each enhancement into our CompressTracker step by step. The training time is calculated by using 8 NVIDIA RTX 3090 GPUs. Notably, our CompressTracker-4 accelerates OSTrack by $\mathbf{2.17\times}$ while preserving approximately $\mathbf{96\%}$ of its original accuracy, thereby demonstrating the effectiveness of our framework. (b) Performance and speed comparison of CompressTracker variants with different numbers of layers. CT-x refers to a version of CompressTracker with ’x’ layers. (c) FLOPs and parameters comparison of CompressTracker variants with different numbers of layers.
Refer to caption: x1.png

Visual object tracking is tasked with continuously localizing a target object across video frames based on the initial bounding box in the first frame. Transformer-based trackers have achieved promising performance on well-established benchmarks, their deployment on resource-restricted device remains a significant challenge. Developing a strong tracker with high efficiency is of great significance.

To reduce the inference cost of models, previous works attempt to design lightweight trackers or transfer the knowledge from teacher models to student trackers. Despite achieving increased speed, these existing methods still exhibit notable limitations. (1) Inferior Accuracy. Certain works propose lightweight tracking models [, , , , ] or employ neural architecture search (NAS) to search better architecture []. Due to the limited number of parameters, these models often suffer from underfitting and inferior performance. (2) Complex Training. Some works [] aim to enhance the accuracy of fast trackers through transferring the knowledge from a teacher tracker to a student model. Despite the improved performance,  [] introduces a complex multi-stage training strategy, which is time-consuming. Any suboptimal performance in these individual stages can cumulatively result in suboptimal performance in the final model. (3) Stucture Limitation. Additionally, the model reduction paradigm in [] severely restricts the structure of student models to be consistent only with the teacher’s model.

Thus, we introduce CompressTracker, a novel and general model compression framework to enhance the efficiency of transformer tracking models. The current dominant trackers are one-stream models [, , , ] characterized by a series of sequential transformer encoder layers, each designed to refine the temporal matching features across frames. The output of each layer is a critical temporal matching result that is refined as the layers get deeper. Given this layer-wise refinement, it becomes a natural progression to consider the model not as a single entity but as a series of interconnected stages and encourage student tracker to align teacher model at each stage. We propose the stage division strategy, which involves partitioning the teacher model, a complex pretrained transformer-based tracking model, into distinct stages that correspond to the layers of a simpler student model. This is achieved by dividing the teacher model into a number of stages equivalent to the student model’s layers. Each stage in the student model is then tasked with learning and replicating the functional behavior of its corresponding stage in the teacher model. This division is not merely a structural alteration but a strategic educational approach. By focusing each stage of the student model on mimicking a specific stage of the teacher, we enable a targeted and efficient transfer of knowledge. The student model learns not just the ’what’ of tracking—i.e., the raw matching of features—but also the ’how’—i.e., the strategies developed by the teacher model at each layer of processing.

Contrary to conventional practices that isolate the training of student models, we employ a replacement training methodology that strategically intertwines the teacher and student models. The core of this methodology is the dynamic substitution of stages during training. we randomly select stages from the student model and replace them with the corresponding stages from the teacher model. By doing so, we situate the teacher model and the student model within a collaborative environment. This arrangement permits the unaltered stages of the teacher model to collaboratively inform and enhance the learning of the substituted stages in the student model rather than supervising the entire student model as a single entity. The student model is not merely learning in parallel but is directly engaging with the teacher’s learned behaviors. After training, we can just combine each stage of student model for inference. The replacement training leads to a more authentic replication of the teacher’s tracking strategies and helps to prevent the student model from overfitting to specific stages of the teacher model, promoting a more stable training.

To augment the learning process, we introduce prediction guidance, which serves as a supervisory signal for the student model by leveraging the teacher model’s predictions. By using the predictions of the teacher model as a reference, the student model can converge more quickly. Furthermore, to enhance the similarity of the temporal matching features across corresponding stages, we have developed a stage-wise feature mimicking strategy. This approach systematically aligns the feature representations learned at each stage of the student model with those of the teacher model, thereby promoting a more accurate and consistent learning. In Figure 1 (a), we show the procedure and the results we are able to achieve with each step toward an efficient transformer tracker.

Figure: Figure 2: CompressTracker Framework. (a) In the training phase, we divide both the teacher model and student model into an identical number of stages. We implement a series of training strategies including replacement training, prediction guidance, and stage-wise feature mimicking, to enhance the student model’s ability to emulate the teacher model. The dotted lines represent the randomly selected paths for replacement training, with black dotted lines indicating the chosen path, while grey dotted lines denote paths not selected in a specific training iteration. (b) During inference process, we simply combine each stage of the student model for testing purposes.
Refer to caption: x2.png

Compared to previous works, our CompressTracker holds many merits. (1) Enhanced Mimicking and Performance. CompressTracker enables the student model to better mimic the teacher model, resulting in better performance. As shown in Figure 1, our CompressTracker-4 achieves $\mathbf{2.17\times}$ speed up while maintaining about $\mathbf{96\%}$ accuracy. (2) Simplified Training Process. Our CompressTracker streamlines training into a single but efficient step. This simplification not only reduces the time and resources required for training but also minimizes the potential for sub-optimal performance associated with complex procedures. The training process for CompressTracker-4 requires merely 20 hours on 8 NVIDIA RTX 3090 GPUs.
(3) Heterogeneous Model Compression. Our stage division strategy gives a high degree of flexibility in the design of the student model. Our framework supports any transformer architecture for student model, which is not restricted to the same structure of teacher tracker. The number of layers and their structure are not predetermined but can be tailored to fit the specific computational constraints and requirements of the deployment environment.

Our contribution can be summarized as follows: (1) We introduce a novel and general model compression framework, CompressTracker, to facilitate the efficient transformer-based object tracking. (2) We propose a stage division strategy that enables a fine-grained imitation of the teacher model at the stage level, enhancing the precision and efficiency of knowledge transfer. (3) We propose the replacement training to improve the student model’s capacity to replicate the teacher model’s behavior. (4) We further incorporate the prediction guidance and feature mimicking to accelerate and refine the learning process of the student model. (5) Our CompressTracker breaks structural limitations, adapting to various transformer architectures for student model. It outperforms existing models, notably accelerating OSTrack [] by $\mathbf{2.17\times}$ while preserving approximately $\mathbf{96\%}$ accuracy.

## 2 Related Work

Visual Object Tracking. Visual object tracking aims to localize the target object of each frame based on its initial appearance. Previous tracking methods [, , , , , , , , , , ] utilize a two-stream pipeline to decouple the feature extraction and relation modeling. Recently, the one-stream pipeline hold the dominant role.  [, , , , , , , , , , ] combine feature extraction and relation modeling into a unified process. These models are built upon vision transformer, which consists of a series of transformer encoder layers. Thanks to a more adequate relationship modeling between template and search frame, one-stream models achieve impressive performance. However, these models suffer from low inference efficiency, which is the main obstacle to practical deployment.

Efficient Tracking.
Some works have attempted to speed up tracking models.  [] utilizes neural architecture search (NAS) to search a light Siamese network, and the searching process is complex.  [, , , ] design a lightweight tracking model, but the small number of parameters restricts the accuracy to a large degree. MixFormerV2 [] propose a complex multi-stage model reduction strategy. Although MixFormerV2-S achieves real-time speed on CPU, the multi-stage training strategy is time consuming, which requires about 120 hours (5 days) on 8 Nvidia RTX8000 GPUs, even several times the original training time of MixFormer []. Any suboptimal performance during these stages impact the final model’s performance negatively. Besides, the reduction paradigm imposes constraints on the design of student models. To address these shortcuts, we propose the general model compression framework, CompressTracker, to explore the roadmap toward an end-to-end and traininig-efficient model compression for lightweight transformer-based tracker. Our CompressTracker break the structure restriction and achieves balance between speed and accuracy.

Transformer Compression. Model compression aims to reduce the size and computational cost of a large model while retaining as much performance as possible, and recently many attempts have been made to speed up a large pretrained tranformer model.  [] reduced the number of parameters through pruning technique, and [] accomplished the quantization of BERT to 2-bits utilizing Hessian information. [, , , ] leverage the knowledge distillation to transfer the knowledge from teacher to student model and exploit pretrained model. Beyond language models, considerable focus has also been placed on compressing vision transformer models.  [, , , , , , , ] utilize multiple model compression techniques to compress vision transformer models. MixFormerV2 [] proposed a two-stage model reduction paradigm to distill a lightweight tracker, relying on the complex multi-stage distillation training. However, our CompressTracker propose an end-to-end and efficient compression training to achieve any transformer structure compression, which speed up OSTrack $\mathbf{2.17\times}$ while maintaining about $\mathbf{96\%}$ accuracy.

## 3 CompressTracker

In this section, we will introduce our proposed general model compression framework, CompressTracker. The workflow of our CompressTracker in illustrated in Figure 2.

### 3.1 Stage Division

Recently, transformer-based one-stream tracking models [, , , ] surpass conventional Siamese trackers [, , ], becoming the dominant manner in the field of visual object tracking. These models consist of several transformer encoder layers, each generating and progressively refining temporal matching features. Building upon this layer-wise refinement, we introduce the stage division strategy, which segments the model into a series of sequential stages. This approach encourages the student model to emulate the teacher model’s behavior at each individual stage. Specifically, we denote the pretrained tracker and the compressed model as teacher and student model, with $N_{t}$ and $N_{s}$ layers, respectively. Both teacher and student models are then divided into $N_{s}$ stages, where each stage in the student model encompasses a single layer, and each corresponding stage in the teacher model may aggregate multiple layers. For a specific stage $i$, we establish a correspondence between the stages of the teacher and student models. The objective of stage division is to enforce each stage of the student model to replicate its counterpart in the teacher model. This stage division strategy breaks the traditional approach that treats the model as an indivisible whole [, , , ]. Instead, it enables a fine-grained learning process where the student model transfers knowledge from the teacher in a more detailed, stage-specific manner.

Unlike the reduction paradigm adopted in [], which confines itself to pruning within identical structures, our CompressTracker framework facilitates support for arbitrary transformer structures of the student tracker, thanks to our innovative stage-wise division design. To align the size and channel dimensions of the student model’s temporal matching features with those of the teacher model, we implement input and output projection layers before and after the student layers, respectively. These projection layers serve as an adjustment mechanism to ensure compatibility between the teacher and student models and allow for a broader range of architectural possibilities for the student model. During the inference process, these input and output injection layers are omitted.

### 3.2 Replacement Training

During the training process, we adopt the replacement training to integrates teacher model and student models, diverging from the conventional practice of training the student model in isolation. In a specific training iteration, we implement a stochastic process to determine which stages of the student model are to be replaced by the corresponding stages of the teacher model. For the specific stage $i$, we decide whether to replace or not by random Bernoulli sampling $b_{i}$ with probability $p$, where $b_{i}\in\{0,1\}$. If $b_{i}$ equals $1$, the output from the preceding stage $i-1$ is directed to the $i$ student stage, otherwise, we channel the output into the $i$ frozen teacher stage. This replacement training creates a collaborative learning environment where the teacher model dynamically supervises the student model. The unreplaced stages of teacher provide valuable contextual supervision for a specific stage in the student model. Consequently, the student model is not operating in parallel but is actively engaged with and learning from the teacher’s established behaviors. For the optimization of student model, we only require the groundtruth box and denote the loss as $L_{track}$. Upon completion of the training process, the student model’s stages are harmoniously combined for inference. We show the pseudocode code in Appendix A.1.

### 3.3 Prediction Guidance & Stage-wise Feature Mimicking

Replacement training enables the student model to learn the behavior of each individual stage, resulting in enhanced performance. However, merely forcing student model to emulate teacher model may be overly challenging for a smaller-sized student. Thus, we employ the teacher’s predictions to further guide the learning of compressed tracker. We apply the same loss as $L_{track}$ for prediction guidance, which is denoted as $L_{pred}$. With the aid of prediction guidance, student benefits from a quicker and stable learning process, assimilating knowledge from teacher model more effectively.

While prediction guidance accelerates the convergence, the student tracker might not entirely match the complex behavior of the teacher model. We introduce the stage-wise feature mimicking to further synchronize the temporal matching features between corresponding stages of the teacher and student models. This alignment is quantified by calculating the $L_{2}$ distance between the outputs of these stages, which is referred as $L_{feat}$. It is worth noting that any metric assessing the discrepancy in feature distributions can serve as the loss function. However, we choose a simple $L_{2}$ distance rather than a complex loss to highlight the effectiveness of our stage division and replacement training strategies. The stage-wise feature mimicking not only promotes a closer similarity in the feature representations of corresponding stages but also enhances the overall coherence between the teacher and student models.

**Table 1: Compress OSTrack. We compress OSTrack multiple configurations with different layer settings. CompressTracker-x denotes the compressed student model with ’x’ layers. We report the performance on 5 benchmarks and calculate the performance gap in comparison to the original OSTrack. Our CompressTracker effectively achieves the balance between performance and speed.**
| Method | LaSOT | LaSOText | TNL2K | TrackingNet | UAV123 | FPS |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AUC | PNorm | P | AUC | P | AUC | P | AUC | PNorm | P | AUC | P |  |  |
| OSTrack-256 [] | 69.1 | 78.7 | 75.2 | 47.4 | 53.3 | 54.3 | - | 83.1 | 87.8 | 82.0 | 68.3 | - | 105 |
| CompressTracker-2 | 60.4 87% | 68.5 | 61.5 | 40.4 85% | 43.8 | 48.5 89% | 45.0 | 78.2 94% | 83.3 | 74.8 | 62.5 92% | 82.5 | 346 3.30× |
| CompressTracker-3 | 64.9 94% | 74.0 | 68.4 | 44.6 94% | 49.6 | 52.6 97% | 50.9 | 81.6 98% | 86.7 | 79.4 | 65.4 96% | 88.3 | 267 2.54× |
| CompressTracker-4 | 66.1 96% | 75.2 | 70.6 | 45.7 96% | 50.8 | 53.6 99% | 52.5 | 82.1 99% | 87.6 | 80.1 | 67.4 99% | 88.0 | 228 2.17× |
| CompressTracker-6 | 67.5 98% | 77.5 | 72.4 | 46.7 99% | 52.5 | 54.7 101% | 54.3 | 82.9 99% | 87.8 | 81.5 | 67.9 99% | 88.7 | 162 1.54× |
| CompressTracker-8 | 68.4 99% | 78.0 | 73.1 | 47.2 99% | 53.1 | 55.2 102% | 54.8 | 83.3 101% | 88.0 | 81.9 | 68.2 99% | 89.0 | 127 1.21× |

**Table 2: Compress MixFormerV2. We compress MixFormerV2 into CompressTracker-M-S with 4 layers, which is the same as MixFormerV2-S including the dimension of MLP layer. We report the performance on 5 benchmarks and calculate the performance gap in comparison to the origin MixFormerV2-B. Our CompressTracker-M-S outperforms MixFormerV2-S under the same setting.**
| Method | LaSOT | LaSOText | TNL2K | TrackingNet | UAV123 | FPS |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AUC | PNorm | P | AUC | P | AUC | P | AUC | PNorm | P | AUC | P |  |  |
| MixFormerV2-B [] | 70.6 | 80.8 | 76.2 | 50.6 | 56.9 | 57.4 | 58.4 | 83.4 | 88.1 | 81.6 | 69.9 | 92.1 | 165 |
| MixFormerV2-S [] | 60.6 | 69.9 | 60.4 | 43.6 | 46.2 | 48.3 | 43.0 | 75.8 | 81.1 | 70.4 | 65.8 | 86.8 | 325 |
| CompressTracker-M-S | 62.0 88% | 70.9 | 63.2 | 44.5 88% | 47.1 | 50.2 87% | 47.8 | 77.7 93% | 82.5 | 73.0 | 66.9 96% | 87.1 | 325 1.97× |

### 3.4 Progressive Replacement

In Section 3.2, we describe the replacement training strategy. Although setting the Bernoulli sampling probability $p$ as a constant value can realize the compression, these stages have not been trained together at the same time and there may be some dissonance. A further finetuning step is necessary to achieve better harmony among the stages. Thus, we introduce a progressive replacement strategy to bridges the gap between the two initially separate training phases, fostering an end-to-end easy-to-hard learning process. By adjusting the value of $p$, we can control the number of stages to be replaced. The value of $p$ gradually increases from $p_{init}$ to $1.0$, allowing for a more incremental and coherent training progression:

$$ $p=\left\{\begin{array}[]{ll}p_{init},&0<=t<\alpha_{1}m,\\ p_{init}+p_{init}\frac{t-\alpha_{1}m}{(1-\alpha_{1}-\alpha_{2})m},&\alpha_{1}m <=t<=(1-\alpha_{2})m,\\ 1.0,&(1-\alpha_{2})m<t<=m,\end{array}\right.$ (1) $$

where $m$ represents the total number of training epochs, and $t$ is a specific training epoch, $\alpha_{1}$ and $\alpha_{2}$ are hyper parameters to modulate the training process. Specifically, $\alpha_{1}$ controls the duration of warmup process, whereas $\alpha_{2}$ determines the length of final finetuning process. The mathematical expectation of $p$ for each layer is:

$$ $E(p)=\int_{0}^{m}pdt=[\frac{1+p_{init}}{2}+\frac{1-p_{init}}{2}(\alpha_{2}- \alpha_{1})]m.$ (2) $$

It is worth noting that each layer is optimized fewer times than the total iteration count, according to the mathematical expectation. Through dynamically adjusting the replacement rate $p$, we eliminate the requirement of finetuning and accomplish an end-to-end model compression.

### 3.5 Training and Inference

Our CompressTracker is a general framework applicable to a wide array of student model architectures. For the optimization of student model, our CompressTracker solely requires an end-to-end and easy-to-hand training process instead of multi-stage training methodologies. Furthermore, our approach simplifies the loss function design, eliminating the need for complex formulations. During training, teacher model is frozen and we only optimize student tracker. The total loss for CompressTracker is:

$$ $L=\lambda_{track}L_{track}+\lambda_{pred}L_{pred}+\lambda_{feat}L_{feat}.$ (3) $$

After training, the various stages of the student model are combined to create a unified model for the inference phase. Consistent with previous methods [, ], a Hanning window penalty is adopted.

**Table 3: Compress OSTrack for SMAT. We compress OSTrack into CompressTracker-SAMT with 4 SMAT layers, which is the same as SMAT. We report the performance on 5 benchmarks and calculate the performance gap in comparison to the original OSTrack. Our CompressTracker-SAMT outperforms SMAT under the same setting.**
| Method | LaSOT | LaSOText | TNL2K | TrackingNet | UAV123 | FPS |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AUC | PNorm | P | AUC | P | AUC | P | AUC | PNorm | P | AUC | P |  |  |
| OSTrack-256 [] | 69.1 | 78.7 | 75.2 | 47.4 | 53.3 | 54.3 | - | 83.1 | 87.8 | 82.0 | 68.3 | - | 105 |
| SMAT [] | 61.7 | 71.1 | 64.6 | - | - | - | - | 78.6 | 84.2 | 75.6 | 64.3 | 83.9 | 158 |
| CompressTracker-SMAT | 62.8 91% | 72.2 | 64.0 | 43.4 92% | 46.0 | 49.6 91% | 46.9 | 79.7 96% | 85.0 | 75.4 | 65.9 96% | 86.4 | 138 1.31× |

## 4 Experiments

### 4.1 Implement Details

Our framework CompressTracker is general and not dependent on a specific transformer structure, hence we select OSTrack [] as baseline, which is a simple and effective transformer-based tracker. The training datasets consist of LaSOT [], TrackingNet [], GOT-10K [], and COCO [], following OSTrack [] and MixFormerV2 []. We set $\lambda_{track}$ as $1$, $\lambda_{pred}$ as $1$, and $\lambda_{feat}$ as $0.2$. The $p_{init}$ is set as $0.5$. We train the CompressTracker with AdamW optimizer [], with the weight decay as $10^{-4}$ and the initial learning rate of $4\times 10^{-5}$. The batch size is $128$. The total training epochs is 500 with 60K image pairs per epoch andthe learning rate is reduced by a factor of 10 after 400 epochs. $\alpha_{1}$ and $\alpha_{2}$ are set as 0.1. The search and template images are resized to resolutions of $288\times 288$ and $128\times 128$. We initialize the CompressTracker with the pretrained parameters of OSTrack. We report the inference speed on a NVIDIA RTX 2080Ti GPU.

**Table 4: State-of-the-art comparison. We compare our CompressTracker which is compressed from OSTrack with previous light-weight tracking models. Our CompressTracker demonstrates superior performance over previous models.**
| Method | LaSOT | LaSOText | TNL2K | TrackingNet | UAV123 | FPS |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AUC | PNorm | P | AUC | P | AUC | P | AUC | PNorm | P | AUC | P |  |  |
| CompressTracker-2 | 60.4 | 68.5 | 61.5 | 40.4 | 43.8 | 48.5 | 45.0 | 78.2 | 83.3 | 74.8 | 62.5 | 82.5 | 346 |
| CompressTracker-3 | 64.9 | 74.0 | 68.4 | 44.6 | 49.6 | 52.6 | 50.9 | 81.6 | 86.7 | 79.4 | 65.4 | 88.3 | 267 |
| CompressTracker-4 | 66.1 | 75.2 | 70.6 | 45.7 | 50.8 | 53.6 | 52.5 | 82.1 | 87.6 | 80.1 | 67.4 | 88.0 | 228 |
| CompressTracker-6 | 67.5 | 77.5 | 72.4 | 46.7 | 52.5 | 54.7 | 54.3 | 82.9 | 87.8 | 81.5 | 67.9 | 88.7 | 162 |
| CompressTracker-8 | 68.4 | 78.0 | 73.1 | 47.2 | 53.1 | 55.2 | 54.8 | 83.3 | 88.0 | 81.9 | 68.2 | 89.0 | 127 |
| HiT-Base [] | 64.6 | 73.3 | 68.1 | 44.1 | - | - | - | 80.0 | 84.4 | 77.3 | 65.6 | - | 175 |
| HiT-Samll [] | 60.5 | 68.3 | 61.5 | 40.4 | - | - | - | 77.7 | 81.9 | 73.1 | 63.3 | - | 192 |
| HiT-Tiny [] | 54.8 | 60.5 | 52.9 | 35.8 | - | - | - | 74.6 | 78.1 | 68.8 | 53.2 | - | 204 |
| SMAT [] | 61.7 | 71.1 | 64.6 | - | - | - | - | 78.6 | 84.2 | 75.6 | 64.3 | 83.9 | 158 |
| MixFormerV2-S [] | 60.6 | 69.9 | 60.4 | 43.6 | 46.2 | 48.3 | 43.0 | 75.8 | 81.1 | 70.4 | 65.8 | 86.8 | 325 |
| FEAR-L [] | 57.9 | 68.6 | 60.9 | - | - | - | - | - | - | - | - | - | - |
| FEAR-XS [] | 53.5 | 64.1 | 54.5 | - | - | - | - | - | - | - | - | - | 80 |
| HCAT [] | 59.0 | 68.3 | 60.5 | - | - | - | - | 76.6 | 82.6 | 72.9 | 63.6 | - | 195 |
| E.T.Track [] | 59.1 | - | - | - | - | - | - | 74.5 | 80.3 | 70.6 | 62.3 | - | 150 |
| LightTrack-LargeA [] | 55.5 | - | 56.1 | - | - | - | - | 73.6 | 78.8 | 70.0 | - | - | - |
| LightTrack-Mobile [] | 53.8 | - | 53.7 | - | - | - | - | 72.5 | 77.9 | 69.5 | - | - | 120 |
| STARK-Lightning [] | 58.6 | 69.0 | 57.9 | - | - | - | - | - | - | - | - | - | 200 |
| DiMP [] | 56.9 | 65.0 | 56.7 | - | - | - | - | 74.0 | 80.1 | 68.7 | 65.4 | - | 77 |
| SiamFC++ [] | 54.4 | 62.3 | 54.7 | - | - | - | - | 75.4 | 80.0 | 70.5 | - | - | 90 |

### 4.2 Compress Object Tracker

In this section, we compress the pretrained OSTrack into different layer configurations. We report the performance of our CompressTracker across these configurations in Table 1. CompressTracker-4 compress OSTrack from 12 layers into 4 layers, and maintain $\mathbf{96\%}$ and $\mathbf{99\%}$ performance on LaSOT and TrackingNet while achieving $\mathbf{2.17\times}$ speed up. Furthermore, as shown in Figure 1, the training process of CompressTracker-4 is notably efficient, requiring only approximately 20 hours using 8 NVIDIA RTX 3090 GPUs. For CompressTracker-6 and CompressTracker-8, as we increase the number of layers, the performance gap between our compresstracker and OSTrack diminishes. It is worth noting that our CompressTracker even outperforms the origin OSTrack on some benchmarks. Specifically, CompressTracker-6 reaches $54.7\%$ AUC on TNL2K, and CompressTracker-8 achieves $55.2\%$ AUC on TNL2K and $83.3\%$ AUC on TrackingNet, while the origin OSTrack only achieves $54.3\%$ AUC on TNL2K and $83.1\%$ AUC on TrackingNet. Our framework CompressTracker demonstrates near lossless compression with the added benefit of increased processing speed.

Moreover, to affirm the generalization ability of our approach, we conduct experiments on MixFormerV2 [] and SMAT []. MixFormerV2-S is a fully transformer tracking model consisting of 4 transformer layers, trained via a complex multi-stages model reduction paradigm. Following MixFormerV2-S, we adopt MixFormerV2-B as teacher and compress it to a student model with 4 layers. The results are shown in Table 2. Our CompressTracker-M-S share the same structure and channel dimension of MLP layers with MixFormerV2-S and outperforms MixFormerV2-S by about $1.4\%$ AUC on LaSOT. SMAT replace the vanilla attention in transformer layer with separated attention. We compress OSTrack into a student model CompressTracker-SMAT, aligning the number and structure of transformer layer with SAMT. We maintain the decoder of OSTrack for CompressTracker-SMAT. CompressTracker-SMAT surpasses SMAT by $1.1\%$ AUC on LaSOT, which demonstrates that our framework is flexible and not limited by the structure of transformer layer. Results in Table 1, 2, 3 verify the generalization ability and effectiveness of our framework.

### 4.3 Comparison with State-of-the-arts

To demonstrate the effectiveness of our CompressTracker, we compare our CompressTracker with state-of-the-art efficient trackers in 5 benchmarks. As shown in Table 4, our CompressTracker outperforms previous efficient trackers. Both HiT [] and SMAT [] are solely trained on the groundtruth and reduce computation through specialized network architectures. MixFormerV2-S [] achieves model compression via a model reduction paradigm. Our CompressTracker-4 achieves $66.1\%$ AUC on LaSOT while maintaining 228 FPS. CompressTracker-4 outperforms HiT-Base by $1.5\%$ AUC on LaSOT without any specialized model structure design. CompressTracker-4 achieves the balance between speed and accuracy. Meanwhile, our CompressTracker-2, with just two transformer layers, maintains the highest speed at 346 FPS and also obtains competitive performance. CompressTracker-2 surpasses HiT-Tiny by $5.6\%$ AUC on LaSOT, and achieves about the same performance as MixFormerV2-S with only two transformer layers. As we add more transformer layers with CompressTracker-6 and CompressTracker-8, we see further improvements in performance. These outcomes demonstrate the effectiveness of our CompressTracker framework.

**Table 5: Ablation studies on LaSOT. The default choice for our model is colored in gray.**
| # | Init. method | AUC |
| --- | --- | --- |
| 1 | MAE-first4 | 59.9% |
| 2 | OSTrack-first4 | 62.0% |
| 3 | OSTrack-skip4 | 62.3% |

### 4.4 Ablation Study

In this section, we conduct a series of ablation studies on LaSOT to explore the factors contributing to the effectiveness of our CompressTracker. Unless otherwise specified, the teacher model is OSTrack, and the student model has 4 encoder layers. The student model is trained for 300 epochs. Please see Appendix A.2 for more analysis.

| # | Prediction<br>Guidance | Feature<br>Mimicking | Replacement<br>Traininig | AUC |
| --- | --- | --- | --- | --- |
| 1 |  |  |  | 62.8 |
| 2 | ✓ |  |  | 63.5 |
| 3 |  | ✓ |  | 63.3 |
| 4 |  |  | ✓ | 63.7 |
| 5 | ✓ |  | ✓ | 64.1 |
| 6 |  | ✓ | ✓ | 64.5 |
| 7 | ✓ | ✓ |  | 64.3 |
| 8 | ✓ | ✓ | ✓ | 65.2 |

tableAblation studies on LaSOT to analyze the supervision of student model. The default choice for our model is colored in gray.

figureAblation study on training strategy.

Backbone Initialization.
We initialize the backbone of student model with different parameters and only train the student model with groundtruth supervision. The results are shown in Table 5. It can be observed that utilizing the knowledge from teacher model is crucial. Moreover, initializing with skipped layers (#3) yields slightly better performance than continuous layers. This suggests that initialization with skipped layers leads to improved representation similarity.

Decoder Initialization and Optimization.
We investigate the influence of decoder’s initialization and optimization on the accuracy of student tracker in Table 5. Initializing the decoder with parameters from the teacher model (#2) results in an improvement of approximately $0.3\%$ compared to a decoder initialized randomly (#1), which underscores the benefits of transferring knowledge from the teacher model to enhance the accuracy of the student model’s decoder. Furthermore, making the decoder trainable leads to an additional improvement of 0.2%.

Stage Division.
Our stage division strategy divides the teacher model into the several stages, and we explore the stage division strategy in Table 5. We design two kinds of division strategy: even and uneven, For the even division, we evenly split the teacher model’s 12 layers into 4 stages, with each stage comprising 3 layers. For uneven division, we follow the design manner in [, ] and divide the 12 layers at a ratio of 1:1:3:1. Consequently, the number of layers in each stage of the teacher model is 2, 2, 6, and 2, respectively. The performance of the two approaches is comparable, leading us to select the equal division strategy for simplicity.

Analysis on Supervision.
We conduct a series of experiments to comprehensively analyze the supervision effects on the student model and to verify the effectiveness of our proposed training strategy. Results are presented in Table 4.4. Our proposed replacement training approach (#4) improves by 0.9 % AUC compared to singly training student model on groundtruth (#1), which demonstrates that the replacement training enhances the similarity between teacher and student models. Besides, prediction guidance (#5) and feature mimicking (#8) further boost the performance, indicating the effectiveness of the two strategies. Compared to only training on groundtruth (#1), our proposed replacement training, prediction guidance and feature mimicking collectively assist student model in more closely mimicking the teacher model, resulting in a total increase of $2.4\%$ AUC.

To further explore the generalization ability of our proposed training strategy, we compare the performance of models with different layer numbers and training settings, as illustrated in Figure 4.4. ’Naive Training’ denotes that the student model is trained without teacher supervision and replacement training. ’Distill Training’ represents that the student model is trained only with teacher supervision. ’CompressTracker’ refers to the same training setting in Table 4.4 $\#8$. It can be observed that as the number of layers increases, there is a corresponding improvement in accuracy. Our CompressTracker shows a noticeable performance boost due to our proposed training strategy, which verifies the effectiveness and generalization ability of our framework.

Training Epochs.
Based on the analysis in Section 3.4, the optimization steps for each layer are lower than total training steps. Thus, to ensure adequate training of each stage, we increase the training epochs from 300 to 500, and show the result in Table 5. Extending the training epochs ensures that student models receive comprehensive training, leading to improved accuracy.

## 5 Limitation

While our CompressTracker demonstrates promising performance and generalization, its training is somewhat inefficient, requiring about $2\times$ time compared to training a student model on ground truth data (20h vs. 8h on 8 NVIDIA 3090 GPUs, as shown in Figure 1 (a)). Moreover, a performance gap still exists between the teacher and student models suggests room for improvement in lossless compression. Future efforts will focus on developing more efficient training methods to boost student model accuracy and decrease training duration.

## 6 Broader Impacts

Our CompressTracker framework efficiently compresses object tracking models for edge device deployment but poses potential misuse risks, such as unauthorized surveillance. We recommend users to carefully consider the real-world implications and adopt risk mitigation strategies.

## 7 Conclusion

In this paper, we propose a general compression framework, CompressTracker, for visual object tracking. We propose a novel stage division strategy to separate the structural dependencies between the student and teacher models. We propose the replacement training to enhance student’s ability to emulate the teacher model. We further introduce the prediction guidance and stage-wise feature mimicking to improve performance. Extensive experiments verify the effectiveness and generalization ability of our CompressTracker. Our CompressTracker is capable of accelerating tracking models while preserving performance to the greatest extent possible.

## Appendix A Appendix / supplemental material

Figure: Algorithm 1 Pseudocode of OSTrack in a PyTorch-like style

Figure: Algorithm 2 Pseudocode of CompressTracker for Training in a PyTorch-like style

Figure: Algorithm 3 Pseudocode of CompressTracker for Testing in a PyTorch-like style

### A.1 Replacement Training

We present the pseudocode for the training and testing phases of CompressTracker in Algorithm 2 and Algorithm 3, respectively.
Additionally, the pseudocode of OSTrack [] is also shown in Algorithm 1. During training process, we employ Bernoulli sampling to implement a replacement training strategy, while in the test phase, we integrate the student layers and discard the teacher layer.

**Table 6: Ablation study on replacement training.**
| # | Replacement | AUC | Training<br>Time |
| --- | --- | --- | --- |
| 1 | Random | 65.2% | 12 h |
| 2 | Decouple-300 | 64.6% | 16 h |

Figure: Figure 3: Ablation study on different replacement probability.
Refer to caption: x4.png

### A.2 More Ablation Study

We represent more ablation studies on LaSOT to explore the factors contributing to effectiveness of our CompressTracker. Unless otherwise specified, teacher model is OSTrack,and student model has 4 encoder layers. The student model is trained for 300 epochs, and the $p_{init}$ is set as $0.5$.

Replacement Training.
To evaluate the efficiency and effectiveness of our replacement training strategy, we conduct a series of experiments and results are presented in Table 8. ’Random’ denotes our replacement training, and ’Decouple-300’ represents decoupling the training of each stage. Result of # 1 aligns with our replacement training with 300 training epochs, while in # 2, we employ a decoupled training approach for each stage. Initially, we substitute the first stage of the teacher model with its counterpart in the student model, training the first stage for 75 epochs. Subsequently, the first trained stage of the student model is frozen, and the second stage undergoes training for an additional 75 epochs. Following this iterative process, we train the four stages cumulatively over 300 epochs, with an additional 30 epochs for fine-tuning. The ’Decouple-300’ (# 2) approach achieves $64.6\%$ AUC on LaSOT with the same training epochs, marginally lower by $0.6\%$ AUC than our replacement training strategy (# 1). The ’Decouple-300’ approach (# 2) requires a complex, multi-stage trainingalong with supplementary fine-tuning, while our CompressTracker operates on an end-to-end, single-step basis. Besides, the ’Decouple-300’ approach may suffer from suboptimal outcomes at a specific training process, but our CompressTracker can avoid this problem through its unified training manner,
which validates the superiority of our replacement training strategy.

Replacement Probability.
We investigate the impact of replacement probability on the accuracy of student model in Figure 4. We maintain a constant replacement probability instead of implementing the progressive replacement strategy and train the student model with 300 epochs and 30 extra finetuning epochs. It can be observed from Figure 4 that performance is adversely affected when the replacement probability is set either too high or too low. Optimal results are achieved when the replacement probability is within the range of 0.5 to 0.7. Specifically, a too low probability leads to inadequate training, whereas a too high probability may result in the insufficient interaction between teacher model and student tracker. Thus, we set the $p_{init}$ as $0.5$ based on the experiment result.

Progressive Replacement. In Table 8, we illustrate the impact of progressive replacement strategy. The first row (# 1) corresponds to the same setting of CompressTracker, while in the second row (# 2) we fix the sampling probability as $0.5$ and the student model is trained with 300 epochs followed by 30 finetuning epochs. The absence of progressive replacement leads to a performance degradation of $0.4\%$ AUC, thereby highlighting the efficacy of our progressive replacement approach.

Training Time.
We compare the training time of CompressTracker with 500 training epochs across different layers in Figure 4. ’Naive Training’ denotes solely training on groundtruth data with 300 epochs, and ’CompressTracker’ represents our proposed training strategy with 500 epochs. The training time is recorded on 8 NVIDIA RTX 3090 GPUs. Besides, the training times of our CompressTracker-4, OSTrack, and MixFormerV2-S are presented in Table 8. Although our CompressTracker requires a longer training time compared to the ’Naive Training’, the increased computational overhead remains within acceptable limits. Moreover, MixFormerV2-S is trained on 8 Nvidia RTX8000 GPUs, and we estimate this will take roughly 80 hours on 8 NVIDIA RTX 3090 GPUs based on the relative computational capabilities of these GPUs. The training time of our CompressTracker-4 is significantly less than that of MixFormerV2-S, which validate the efficiency and effectiveness of our framework.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: The application scenarios of efficient tracking models are mostly devices with insufficient examples. The author should provide the speed of the model on the CPU or edge device to verify the word "efficient", rather than just testing the speed on the GPU. For other issues, see weakness.
soundness: 3
strengths: The author has clear ideas and the article is easy to understand. He proposes a general compression framework for single object tracking. This method can efficiently compress large object tracking models into small models. The author has conducted a large number of experiments to prove the effectiveness of this method.
summary: In this paper, the authors proposed a general model compression framework for efficient Transformer object tracking, named CompressTracker. The method adopts a novel stage partitioning strategy to divide the Transformer layers of the teacher model into different stages, enabling the student model to more effectively simulate each corresponding teacher stage. The authors also designed a unique replacement training technique, which involves randomly replacing specific stages in the student model with specific stages in the teacher model. Replacement training enhances the student model's ability to replicate the behavior of the teacher model. To further force the student model to simulate the teacher model, we combine predictive guidance and staged feature imitation to provide additional supervision during the compression process of the teacher model. The authors conducted a series of experiments to verify the effectiveness and generality of CompressTracker.
weaknesses: 1. The innovation is slightly insufficient. The author’s innovation focuses on replacement training, prediction guidance, and feature mimicking. The latter two are common methods of distillation and are not enough to be the innovation of this article. Therefore, the innovation of this article is more focused on the replacement training strategy, but the author’s explanation of the intuitive reasons why this strategy is useful is poor.
2. The author claims that the method is general compression framework, but the paper only experiments on OSTrack and MixFormerV2. However, there are many trackers based on transforemr, such as SimTrack, ODtrack, LoraT, etc. The reviewer thinks that it cannot be called general compression framework after only verifying two trackers.
3. The author only conducted experiments on GPUs with sufficient computing power. However, efficient trackers are more targeted at devices with insufficient computing power, such as CPUs and edge devices. The author did not conduct experiments on such devices to verify the term efficient.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: In the article, the framework proposed is added to OSTrack, where the search area pixels are 256× 256and the template pixels are 128× 128. What is the effect when the search area pixels are 384 × 384 and the template pixels are 192 × 192? Can tracking accuracy still be maintained?

What are the limitations of the stage division strategy, and how does it affect the generalization capabilities of the student model?

According to the experimental section, it can be seen that speed and accuracy cannot be achieved simultaneously. The faster the speed, the lower the accuracy. Taking OSTrack as an example, the encoder layer of OSTrack is 12 layers. Would it be possible to achieve a similar effect by reducing the number of layers appropriately? Will the compression method proposed in the article have more advantages in accuracy and speed?
soundness: 3
strengths: The paper proposes an innovative approach to segmenting transformer layers into stages, allowing for more granular knowledge transfer from a teacher model to a student model.

The experiments show that CompressTracker can achieve a substantial speedup while maintaining a high level of accuracy, which is crucial for real-world applications on resource-constrained devices.

The framework's compatibility with any transformer architecture is a significant advantage, as it increases its applicability and flexibility.
summary: The paper presents CompressTracker, a novel general model compression framework aimed at enhancing the efficiency of transformer-based object tracking models for deployment on resource-constrained devices. The framework employs a stage division strategy to segment the teacher model into distinct stages, which are then emulated by a lighter student model. CompressTracker introduces a replacement training technique, where student model stages are dynamically replaced with teacher model stages during training, enhancing the student's ability to replicate the teacher's behavior. Additionally, prediction guidance and stage-wise feature mimicking are incorporated to refine the learning process. The framework is structurally agnostic and compatible with various transformer architectures.
weaknesses: There is still a performance gap between the teacher and student models, indicating that there might be room for further improvement in the compression strategy to achieve lossless compression. 

While the framework simplifies the compression process, the introduction of multiple training strategies might increase the complexity of the training regimen, which could be a barrier for some users.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. OSTrack and MixFormer are only used to verify the effectiveness of the proposed CompressTracker. As a general model compression framework, this is far from enough.
2. This paper proposes a novel stage division strategy. To demonstrate its effectiveness, the corresponding ablation experimental results are shown in Table 8. It can be seen from Table 8 that the even dividing strategy is better than the uneven dividing strategy, but it cannot indicate that the performance of model with the stage division strategy is better than that without the stage division strategy.
3. What is the difference of the ablation experiments in Table 9 and 10?
4. Does CompressTracker only apply to the model with several same feature extractor modules? If the structures of different stages are different, does it work?
5. The format of Table 5 is non-standard. A table should not contain other tables.
6. There is spelling mistake, such as "Stucture Limitation" in Page 1.
soundness: 2
strengths: The motivation of this paper is clear and it has certain innovation.
summary: This paper introduces a general model compression framework based on the teacher-student knowledge distillation method for efficient transformer object tracking, named CompressTracker, which designs a stage division strategy and a replacement training technique.
weaknesses: Two classical models are only used to verify the effectiveness of the proposed CompressTracker, which are not enough to demonstrate its applicability and scalability.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness.  Moreover, the paper does not compare with other model compression techniques, such as knowledge distillation, model quantization, and pruning. It helps if you provide some comparisons or analysis.
soundness: 3
strengths: 1. The author clearly presents a framework for compressing single object tracking models, effectively reducing larger models to smaller, efficient versions. Extensive experiments demonstrate the method's effectiveness.

2. One of the main benefits of the proposed framework is its structural agnosticism, meaning it can work with any transformer architecture. This adaptability allows CompressTracker to fit different student model configurations, making it suitable for various deployment environments and computational limits.

3. The paper shows through extensive experiments that CompressTracker strikes an impressive balance between inference speed and tracking accuracy. It significantly speeds up the tracking process while preserving high performance, achieving nearly 96% of the original accuracy with a 2.17× increase in speed.
summary: The paper presents CompressTracker, a framework for general model compression designed to enhance the efficiency of transformer-based object-tracking models. This approach utilizes a unique stage partitioning strategy that divides the transformer layers of the teacher model into distinct stages, allowing the student model to simulate each corresponding stage better.

Furthermore, the authors introduced a replacement training technique, where specific stages in the student model are randomly replaced with those from the teacher model. This strategy, combined with predictive guidance and staged feature imitation, provides additional supervision to help the student model mimic the teacher model more effectively during the compression process.

Extensive experiments on vit-based trackers were conducted, which show the proposed method can lightweight the trackers while main a comparable performance.
weaknesses: 1. The primary drawback of this method lies in its dependence on various distillation techniques, such as different training strategies, feature mimicking, and loss guidance. This lack of a clear, consistent framework among these techniques may undermine the generalization ability and transferability of the proposed approach, despite the author's assertions to the contrary.

2. Moreover, the overall complexity of the method raises concerns about its usability for other researchers. For instance, when applied to Mixformer V2, which has only two layers, the improvement in performance is minimal, while the processing speed remains unchanged. Such results indicate possible limitations of the method, as the intricate techniques lead to only marginal benefits.

3. The proposed techniques (stage division, progressive replacement, Replacement Training, Prediction Guidance, and Stage-wise Feature Mimicking) appear to be independent. The title "General Framework" raises my expectations significantly.

Meta-review: This paper presents a model compression framework aimed at enhancing the efficiency of transformer-based object tracking. It received mixed reviews, with scores of 5, 5, 6, and 6, resulting in an average score of 5.5.

Reviewers acknowledged the paper’s strengths, particularly its innovative approach to compressing transformer-based tracking models to improve efficiency, as highlighted by reviewers 3q3t and ejyx.

However, several weaknesses were noted, including limited technical contributions (pointed out by Reviewers ejyx and hzRe), a performance gap (identified by Reviewer hzRe), and insufficient experimental validation (mentioned by Reviewers AUCs and ejyx).

The Area Chair (AC) also assessed the paper and noted that the three technical contributions—stage division, replacement training, and prediction guidance/feature mimicking—are not closely related or unified. This makes the claim of a "general compression framework" somewhat overclaimed. The results show that while the proposed method achieves approximately a 2x speed increase, it results in a performance drop of 3% to 8%. Furthermore, the effectiveness of the method is inconsistent across different backbone variants and scales, indicating that more experiments are needed to validate its generalization ability.

After discussions among the authors, reviewers, and the AC, some concerns were addressed. Nevertheless, issues concerning technical novelty remain unresolved. The AC discussed this paper with the Senior AC in depth. They concluded that they could not recommend the acceptance of this paper and encouraged the authors to take the reviewers' comments into account to improve the paper for the next venue.

Decision: Reject