General Compression Framework for Efficient Transformer Object Tracking

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 09:09:32

Abstract: Transformer-based trackers have established a dominant role in the field of visual object tracking. While these trackers exhibit promising performance, their deployment on resource-constrained devices remains challenging due to inefficiencies. To improve the inference efficiency and reduce the computation cost, prior approaches have aimed to either design lightweight trackers or distill knowledge from larger teacher models into more compact student trackers. However, these solutions often sacrifice accuracy for speed. Thus, we propose a general model compression framework for efficient transformer object tracking, named CompressTracker, to reduce the size of a pre-trained tracking model into a lightweight tracker with minimal performance degradation. Our approach features a novel stage division strategy that segments the transformer layers of the teacher model into distinct stages, enabling the student model to emulate each corresponding teacher stage more effectively. Additionally, we also design a unique replacement training technique that involves randomly substituting specific stages in the student model with those from the teacher model, as opposed to training the student model in isolation. Replacement training enhances the student model's ability to replicate the teacher model's behavior. To further forcing student model to emulate teacher model, we incorporate prediction guidance and stage-wise feature mimicking to provide additional supervision during the teacher model's compression process. Our framework CompressTracker is structurally agnostic, making it compatible with any transformer architecture. We conduct a series of experiment to verify the effectiveness and generalizability of CompressTracker. Our CompressTracker-4 with 4 transformer layers, which is compressed from OSTrack, retains about $\mathbf{96\%}$ performance on LaSOT ($\mathbf{66.1\%}$ AUC) while achieves $\mathbf{2.17\times}$ speed up.

Main paper: 
 introduction: 
 Visual object tracking is tasked with continuously localizing a target object across video frames based on the initial bounding box in the first frame. Transformer-based trackers have achieved promising performance on well-established benchmarks, their deployment on resource-restricted device remains a significant challenge. Developing a strong tracker with high efficiency is of great significance.To reduce the inference cost of models, previous works attempt to design lightweight trackers or transfer the knowledge from teacher models to student trackers. Despite achieving increased speed, these existing methods still exhibit notable limitations. (1) Inferior Accuracy. Certain works propose lightweight tracking models [6, 10, 4, 21, 26] or employ neural architecture search (NAS) to search better architecture [42] . Due to the limited number of parameters, these models often suffer from underfitting and inferior performance. (2) Complex Training. Some works [15] aim to enhance the accuracy of fast trackers through transferring the knowledge from a teacher tracker to a student model. Additionally, the model reduction paradigm in [15] severely restricts the structure of student models to be consistent only with the teacher's model.Thus, we introduce CompressTracker, a novel and general model compression framework to enhance the efficiency of transformer tracking models. The current dominant trackers are one-stream models [44, 15, 4, 10] characterized by a series of sequential transformer encoder layers, each designed to refine the temporal matching features across frames. The output of each layer is a critical temporal matching result that is refined as the layers get deeper. Given this layer-wise refinement, it becomes a natural progression to consider the model not as a single entity but as a series of interconnected stages and encourage student tracker to align teacher model at each stage. We propose the stage division strategy, which involves partitioning the teacher model, a complex pretrained transformer-based tracking model, into distinct stages that correspond to the layers of a simpler student model. This is achieved by dividing the teacher model into a number of stages equivalent to the student model's layers. Each stage in the student model is then tasked with learning and replicating the functional behavior of its corresponding stage in the teacher model. This division is not merely a structural alteration but a strategic educational approach. By focusing each stage of the student model on mimicking a specific stage of the teacher, we enable a targeted and efficient transfer of knowledge.The student model learns not just the 'what' of tracking-i.e., the raw matching of features-but also the 'how'-i.e., the strategies developed by the teacher model at each layer of processing.Contrary to conventional practices that isolate the training of student models, we employ a replacement training methodology that strategically intertwines the teacher and student models. The core of this methodology is the dynamic substitution of stages during training. we randomly select stages from the student model and replace them with the corresponding stages from the teacher model. By doing so, we situate the teacher model and the student model within a collaborative environment. This arrangement permits the unaltered stages of the teacher model to collaboratively inform and enhance the learning of the substituted stages in the student model rather than supervising the entire student model as a single entity. The student model is not merely learning in parallel but is directly engaging with the teacher's learned behaviors. After training, we can just combine each stage of student model for inference. The replacement training leads to a more authentic replication of the teacher's tracking strategies and helps to prevent the student model from overfitting to specific stages of the teacher model, promoting a more stable training.To augment the learning process, we introduce prediction guidance, which serves as a supervisory signal for the student model by leveraging the teacher model's predictions. By using the predictions of the teacher model as a reference, the student model can converge more quickly. Furthermore, to enhance the similarity of the temporal matching features across corresponding stages, we have developed a stage-wise feature mimicking strategy. This approach systematically aligns the feature representations learned at each stage of the student model with those of the teacher model, thereby promoting a more accurate and consistent learning. In Figure 1 (a), we show the procedure and the results we are able to achieve with each step toward an efficient transformer tracker.  
related work: 
 Visual Object Tracking. Visual object tracking aims to localize the target object of each frame based on its initial appearance. Previous tracking methods [2, 28, 46, 3, 16, 27, 5, 23, 12, 41] utilize a two-stream pipeline to decouple the feature extraction and relation modeling. Recently, the one-stream pipeline hold the dominant role. [44, 14, 15, 1, 37, 8, 11, 19] combine feature extraction and relation modeling into a unified process. These models are built upon vision transformer, which consists of a series of transformer encoder layers. Thanks to a more adequate relationship modeling between template and search frame, one-stream models achieve impressive performance. However, these models suffer from low inference efficiency, which is the main obstacle to practical deployment.Efficient Tracking. Some works have attempted to speed up tracking models. [42] utilizes neural architecture search (NAS) to search a light Siamese network, and the searching process is complex. [6, 10, 4, 26] design a lightweight tracking model, but the small number of parameters restricts the accuracy to a large degree. MixFormerV2 [15] propose a complex multi-stage model reduction strategy. Although MixFormerV2-S achieves real-time speed on CPU, the multi-stage training strategy is time consuming, which requires about 120 hours (5 days) on 8 Nvidia RTX8000 GPUs, even several times the original training time of MixFormer [14] . Any suboptimal performance during these stages impact the final model's performance negatively. Besides, the reduction paradigm imposes constraints on the design of student models. To address these shortcuts, we propose the general model compression framework, CompressTracker, to explore the roadmap toward an endto-end and traininig-efficient model compression for lightweight transformer-based tracker. OurCompressTracker break the structure restriction and achieves balance between speed and accuracy.Transformer Compression. Model compression aims to reduce the size and computational cost of a large model while retaining as much performance as possible, and recently many attempts have been made to speed up a large pretrained tranformer model. [18] reduced the number of parameters through pruning technique, and [35] accomplished the quantization of BERT to 2-bits utilizing Hessian information. [34, 36, 25, 38] leverage the knowledge distillation to transfer the knowledge from teacher to student model and exploit pretrained model. Beyond language models, considerable focus has also been placed on compressing vision transformer models. [33, 40, 9, 20, 7, 43, 45] utilize multiple model compression techniques to compress vision transformer models. MixFormerV2 [15] Prediction Guidance Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of our work in our maniscript.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
compresstracker: 
 In this section, we will introduce our proposed general model compression framework, Com-pressTracker. The workflow of our CompressTracker in illustrated in Figure 2 .Recently, transformer-based one-stream tracking models [8, 14, 44, 15] surpass conventional Siamese trackers [2, 13, 12] , becoming the dominant manner in the field of visual object tracking. These models consist of several transformer encoder layers, each generating and progressively refining temporal matching features. Building upon this layer-wise refinement, we introduce the stage division strategy, which segments the model into a series of sequential stages. This approach encourages the student model to emulate the teacher model's behavior at each individual stage. Specifically, we denote the pretrained tracker and the compressed model as teacher and student model, with N t and N s layers, respectively. Both teacher and student models are then divided into N s stages, where each stage in the student model encompasses a single layer, and each corresponding stage in the teacher model may aggregate multiple layers. For a specific stage i, we establish a correspondence between the stages of the teacher and student models. The objective of stage division is to enforce each stage of the student model to replicate its counterpart in the teacher model. This stage division strategy breaks the traditional approach that treats the model as an indivisible whole [6, 10, 4, 15] . Instead, it enables a fine-grained learning process where the student model transfers knowledge from the teacher in a more detailed, stage-specific manner.Unlike the reduction paradigm adopted in [15] , which confines itself to pruning within identical structures, our CompressTracker framework facilitates support for arbitrary transformer structures of the student tracker, thanks to our innovative stage-wise division design. To align the size and channel dimensions of the student model's temporal matching features with those of the teacher model, we implement input and output projection layers before and after the student layers, respectively. These projection layers serve as an adjustment mechanism to ensure compatibility between the teacher and student models and allow for a broader range of architectural possibilities for the student model.During the inference process, these input and output injection layers are omitted. Replacement training enables the student model to learn the behavior of each individual stage, resulting in enhanced performance. However, merely forcing student model to emulate teacher model may be overly challenging for a smaller-sized student. Thus, we employ the teacher's predictions to further guide the learning of compressed tracker. We apply the same loss as L track for prediction guidance, which is denoted as L pred . With the aid of prediction guidance, student benefits from a quicker and stable learning process, assimilating knowledge from teacher model more effectively.While prediction guidance accelerates the convergence, the student tracker might not entirely match the complex behavior of the teacher model. We introduce the stage-wise feature mimicking to further synchronize the temporal matching features between corresponding stages of the teacher and student models. This alignment is quantified by calculating the L 2 distance between the outputs of these stages, which is referred as L f eat . It is worth noting that any metric assessing the discrepancy in feature distributions can serve as the loss function. However, we choose a simple L 2 distance rather than a complex loss to highlight the effectiveness of our stage division and replacement training strategies. The stage-wise feature mimicking not only promotes a closer similarity in the feature representations of corresponding stages but also enhances the overall coherence between the teacher and student models.In Section 3.2, we describe the replacement training strategy. Although setting the Bernoulli sampling probability p as a constant value can realize the compression, these stages have not been trained together at the same time and there may be some dissonance. A further finetuning step is necessary to achieve better harmony among the stages. Thus, we introduce a progressive replacement strategy to bridges the gap between the two initially separate training phases, fostering an end-to-end easy-tohard learning process. By adjusting the value of p, we can control the number of stages to be replaced.The value of p gradually increases from p init to 1.0, allowing for a more incremental and coherent training progression:EQUATIONwhere mEQUATIONIt is worth noting that each layer is optimized fewer times than the total iteration count, according to the mathematical expectation. Through dynamically adjusting the replacement rate p, we eliminate the requirement of finetuning and accomplish an end-to-end model compression.Our CompressTracker is a general framework applicable to a wide array of student model architectures.For the optimization of student model, our CompressTracker solely requires an end-to-end and easyto-hand training process instead of multi-stage training methodologies. Furthermore, our approach simplifies the loss function design, eliminating the need for complex formulations. During training, teacher model is frozen and we only optimize student tracker. The total loss for CompressTracker is:EQUATIONAfter training, the various stages of the student model are combined to create a unified model for the inference phase. Consistent with previous methods [44, 14] , a Hanning window penalty is adopted.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We provide the core pseudocode in appendix, and we will release our code after acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
implement details: 
 Our framework CompressTracker is general and not dependent on a specific transformer structure, hence we select OSTrack [44] as baseline, which is a simple and effective transformer-based tracker.The training datasets consist of LaSOT [17] , TrackingNet [32] , GOT-10K [24] , and COCO [29] ,following OSTrack [44] and MixFormerV2 [15] . We set λ track as 1, λ pred as 1, and λ f eat as 0.2.The p init is set as 0.5. We train the CompressTracker with AdamW optimizer [31] , with the weight decay as 10 -4 and the initial learning rate of 4 × 10 In this section, we conduct a series of ablation studies on LaSOT to explore the factors contributing to the effectiveness of our CompressTracker. Unless otherwise specified, the teacher model is OSTrack, and the student model has 4 encoder layers. The student model is trained for 300 epochs. Please see Appendix A.2 for more analysis.Backbone Initialization. We initialize the backbone of student model with different parameters and only train the student model with groundtruth supervision. The results are shown in Table 5 . It can be observed that utilizing the knowledge from teacher model is crucial. Moreover, initializing with skipped layers (#3) yields slightly better performance than continuous layers. This suggests that initialization with skipped layers leads to improved representation similarity.Decoder Initialization and Optimization. We investigate the influence of decoder's initialization and optimization on the accuracy of student tracker in Table 6 . Initializing the decoder with parameters from the teacher model (#2) results in an improvement of approximately 0.3% compared to a decoder initialized randomly (#1), which underscores the benefits of transferring knowledge from the teacher model to enhance the accuracy of the student model's decoder. Furthermore, making the decoder trainable leads to an additional improvement of 0.2%.Stage Division. Our stage division strategy divides the teacher model into the several stages, and we explore the stage division strategy in Table 7 . We design two kinds of division strategy: even and uneven, For the even division, we evenly split the teacher model's 12 layers into 4 stages, with each stage comprising 3 layers. For uneven division, we follow the design manner in [22, 30] and divide the 12 layers at a ratio of 1:1:3:1. Consequently, the number of layers in each stage of the teacher model is 2, 2, 6, and 2, respectively. The performance of the two approaches is comparable, leading us to select the equal division strategy for simplicity.  
broader impacts: 
 Our CompressTracker framework efficiently compresses object tracking models for edge device deployment but poses potential misuse risks, such as unauthorized surveillance. We recommend users to carefully consider the real-world implications and adopt risk mitigation strategies.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We include all the details in our paper, including datasets, model, and training details. Other researchers can reproduce our result easily. We also provide the core pseudocode in appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In an additional 75 epochs. Following this iterative process, we train the four stages cumulatively over 300 epochs, with an additional 30 epochs for fine-tuning. The 'Decouple-300' (# 2) approach achieves 64.6% AUC on LaSOT with the same training epochs, marginally lower by 0.6% AUC than our replacement training strategy (# 1). The 'Decouple-300' approach (# 2) requires a complex, multi-stage trainingalong with supplementary fine-tuning, while our CompressTracker operates on an end-to-end, single-step basis. Besides, the 'Decouple-300' approach may suffer from suboptimal outcomes at a specific training process, but our CompressTracker can avoid this problem through its unified training manner, which validates the superiority of our replacement training strategy.Replacement Probability. We investigate the impact of replacement probability on the accuracy of student model in Figure 4 . We maintain a constant replacement probability instead of implementing the progressive replacement strategy and train the student model with 300 epochs and 30 extra finetuning epochs. It can be observed from Figure 4 that performance is adversely affected when the replacement probability is set either too high or too low. Optimal results are achieved when the replacement probability is within the range of 0.5 to 0.7. Specifically, a too low probability leads to inadequate training, whereas a too high probability may result in the insufficient interaction between teacher model and student tracker. Thus, we set the p init as 0.5 based on the experiment result.Progressive Replacement. In Table 12 , we illustrate the impact of progressive replacement strategy.The first row (# 1) corresponds to the same setting of CompressTracker, while in the second row (# 2)we fix the sampling probability as 0.5 and the student model is trained with 300 epochs followed by 30 finetuning epochs. The absence of progressive replacement leads to a performance degradation of 0.4% AUC, thereby highlighting the efficacy of our progressive replacement approach.Training Time. Justification: We summarize our contribution in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Our paper does not report error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide the type of GPU we used and time of execution Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the potential negative societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: We describe the safeguards in our paper.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All the code, data and models we used are credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Justification: Our research conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper introduces a model compression framework designed for efficient transformer-based object tracking. It received scores of 3, 5, 5, 6, and 9, with an average score of 5.6.

Reviewers recognized the paper’s strengths, particularly its introduction of a new approach for compressing transformer-based tracking models to improve efficiency (highlighted by reviewers M8LK, 8Frr, pwR6). However, they noted several weaknesses, including limited technical contributions (pointed out by reviewers H7HB, 8Frr, pwR6) and insufficient experiments (cited by reviewers H7HB, 8Frr).

The Associate Chair (AC) also reviewed the paper and identified the main contributions as a stage division strategy, which compresses the original model in a stage-wise manner, and a stage replacement training scheme, which randomly substitutes specific stages in the student model with those from the teacher model. The results indicate that while the proposed method achieves approximately a 2x speed acceleration, it incurs a performance drop of about 3% and 8% in AUC score on the LaSOT dataset using the OSTrack and MixFormer tracking models, respectively. The improvements over using a smaller backbone model are not significant, and the claim of a ‘general compression framework’ does not hold, particularly as performance drops significantly when applied to the MixFormer tracker.

After discussions among the authors, reviewers, and the AC, some concerns have been addressed. However, the issues regarding insufficient experiments (noted by reviewers pwR6 and H7HB) remain unresolved. Therefore, the AC cannot recommend accepting this paper and encourages the authors to consider the reviewers' comments for future submissions.

Reviews: review: paper_summary: paper_summary: This paper introduces CompressTracker, a general model compression framework for efficient transformer-based object tracking. CompressTracker divides the teacher model into stages corresponding to student model layers and randomly replaces student stages with teacher stages during training. It also aligns the teacher and student models using prediction guidance and feature mimicking. The framework gradually increases the probability of using student stages throughout training. CompressTracker achieves significant speed improvements while maintaining high accuracy. For example, CompressTracker-4 accelerates OSTrack by 2.17x while preserving 96% of its accuracy on LaSOT.
strength_weakness: strength_weakness: - Limited theoretical analysis: The paper focuses on empirical results without providing much theoretical justification for the proposed methods.
- Lack of ablation on some components: Some components of the framework are not thoroughly explored. For instance, the impact of different feature mimicking strategies is not extensively analyzed.
- Performance and Efficiency Trade-off: While CompressTracker maintains high accuracy, there's a slight performance drop compared to the original model. Training time for CompressTracker-4 (with only 4 blocks) exceeds that of the original OSTrack. This trade-off between training efficiency, inference speed, and model performance requires further optimization.
- The core idea of reducing the number of Transformer blocks is not new. Similar approaches have been used in other models like TinyViT[1] and MiniViT[2].


[1] Wu K, Zhang J, Peng H, et al. Tinyvit: Fast pretraining distillation for small vision transformers[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 68-85.
[2] Zhang J, Peng H, Wu K, et al. Minivit: Compressing vision transformers with weight multiplexing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 12145-12154.
questions: questions: Please refer to the weakness.
limitations: limitations: No
paper_summary: paper_summary: In this paper, the authors proposed a general model compression framework for efficient Transformer object tracking, named CompressTracker. The method adopts a novel stage partitioning strategy to divide the Transformer layers of the teacher model into different stages, enabling the student model to more effectively simulate each corresponding teacher stage. The authors also designed a unique replacement training technique, which involves randomly replacing specific stages in the student model with specific stages in the teacher model. Replacement training enhances the student model's ability to replicate the behavior of the teacher model. To further force the student model to simulate the teacher model, we combine predictive guidance and staged feature imitation to provide additional supervision during the compression process of the teacher model. The authors conducted a series of experiments to verify the effectiveness and generality of CompressTracker.
strength_weakness: strength_weakness: The font size of the pictures in the article is too small. The author can adjust the font size appropriately to facilitate reading. The training time line in Figure 1a is blocked, resulting in incomplete display. The font size of the tables is inconsistent, for example, the font size of Tables 5, 6, 7, and 8 is too large. The abstract is redundant and can be appropriately deleted.
questions: questions: 1.The font size of the pictures in the article is too small. The author can adjust the font size appropriately to facilitate reading.

2.The training time line in Figure 1a is blocked, resulting in incomplete display.

3.The font size of the tables is inconsistent, for example, the font size of Tables 5, 6, 7, and 8 is too large. 

4.The abstract is redundant and can be appropriately deleted.

5.Did the author test the speed on other devices, such as CPU?

6.Will the codebe open source?
limitations: limitations: For lightweight tracking models, the training time is too long. The author can try to find new ways to reduce the time spent on training.
paper_summary: paper_summary: The paper introduces CompressTracker, a novel general model compression framework that enhances the efficiency of transformer-based object tracking models. It innovatively segments transformer layers into stages, enabling a more effective emulation of complex teacher models by lightweight student models. The framework incorporates a unique replacement training technique, prediction guidance, and feature mimicking to refine the student model's performance. Extensive experiments demonstrate CompressTracker's effectiveness in significantly speeding up tracking models with minimal loss of accuracy, showcasing its potential for real-time applications on resource-constrained devices.
strength_weakness: strength_weakness: 1）The concept of "prediction guidance and stage-wise feature mimicking" and the idea of BEVDistill [1] seem somewhat similar.

2）Despite the model's efficiency in inference, the training process for CompressTracker is relatively inefficient.

3）While the paper shows promising results on certain benchmarks, there may be concerns about how well these findings generalize across different types of tracking tasks and real-world scenarios.

4）The paper does not compare with other model compression techniques, such as knowledge distillation, model quantization, and pruning.

5）According to the results in Table 3, I observed that the outcomes of CompressTracker-2 are inferior to those of MixFormerV2-S. What could be the reason for this?

6）It is necessary to apply compression to other tracking models in order to further validate the efficacy of the CompressTracker presented in this paper.

7）The authors lack a sufficiently comprehensive review of the related work. The authors should give more reasonable related work by carefully introducing the recent approaches to tracking with compression, such as [2].

[1] BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detection, ICLR 2023.

[2] Distilled Siamese Networks for Visual Tracking, TPAMI 2021.
questions: questions: Please refer to weaknesses.
limitations: limitations: Please refer to weaknesses.
paper_summary: paper_summary: This paper aims to  distill knowledge from larger teacher models into more compact student trackers. Three techniques are proposed: A stage division strategy that segments the transformer layers of the teacher model. Replacement training technique. Prediction guidance and stage-wise feature mimicking. Experiment verifys the effectiveness of the method.
strength_weakness: strength_weakness: 1.	The most obvious weakness is that the whole method consists of many distilling techniques, including training strategies, feature mimicking, and loss guidance. It is hard to see the inherent consistency between those techniques. This may harm the generalization ability and transferability of the proposed framework, as the author claims the framework is general. 
2.	The overall method is complex. I am worried about its application to other researchers.
3.	When applied to the Mixformer v2, which has only 2 layers, performance can be improved marginally while speed is unchanged. This may indicate the method's shortcomings. Complex techniques only bring a little improvement.
questions: questions: 1.	Prove those techniques can be unified instead of looking like a bunch of tricks. 
2.	The methods are restricted when the transformer tracker has fewer layers. 
3.	See weakness.
limitations: limitations: Yes
paper_summary: paper_summary: The authors proposed a novel compression strategy of transformer based trackers. Unlike previous works, it divides the teacher network into multiple segments, each segment corresponds a single transformer layer of student network, then train each student layer separately. It also introduced some training strategies to enhance performance including (progressive) replacement training, prediction guidance and feature mimicking. Such compression framework is insensitive to the change of architecture of teacher network.
strength_weakness: strength_weakness: The detailed strategy of dividing the teacher network is not stated clearly in the paper. Base on the pseudo code provided in page 13, it seems that the segmentation strategy is simply mapping the list of transformer blocks of student network to that of the teacher network base on the lengths of the two lists. This could be too simple.

For example, assume teacher network has 8 transformer blocks in module 1 and 2 blocks in module 2, while student network consists of 2 blocks, then the second student block would have to emulate the last 3 blocks of module 1 and the 2 blocks of module 2, while module 1 and  module 2 might have been trained separately and possess different knowledge. Empirically, this would result in sub-optical performance.

A brief discuss on the divide strategy could help this paper become more informative.
questions: questions: I don't quite understand the concept of prediction guidance mentioned in the first paragraph of section 3.3: does it mean using the prediction of teacher model as pseudo label to supervise student's learning? If so, then how exactly does it help learning process, since the original ground truth bounding box already provides similar supervision, I don't really understand why using the less precise, noisy prediction of teacher tracker as additional pseudo label could actually be beneficial.
limitations: limitations: The paper clearly addressed its limitations including inefficient training process and the performance gap between teacher and student network.

Comments: title: Thanks for your acknowledgement and support.
Thanks for your acknowledgement and support.
Theoretical analysis of the superiority of our replacement training
comment: We are grateful for your recognition of the novelty and efficiency of our work. We are glad that our response has addressed your concerns. Thanks a lot for you support for our work.
Thanks for the authors. All concerns have been addressed. I will keep my rating.
We are grateful for your recognition of our contribution. Thanks a lot for your reconsidering the rating. We will work further to make our work better. Thanks a lot for you support for our work.
Thanks for your responses. Though reducing the layer numbers of the vit-based tracking is not new, the contribution is non-trivial. I acknowledge the contribution of this work. Thus, I am willing to raise my initial rating to 5 (bordline accept).
We are grateful for your recognition of the novelty and efficiency of our work. We are also very glad that our response can resolve your concerns and it's our honour! Thanks a lot for you support for our work.
Thank you for clarifying my inquiries. Your explanations on prediction guidance have been enlightening to me, expanding my understanding of knowledge distillation, a concept I have not delved deeply into before. I appreciate your valuable contributions.
Please allow me to explain the superiority of our replacement training strategy from an information theoretic perspective. Dropout increases entropy during training by randomly dropping neurons to reduce model overfitting. Our replacement training strategy is similar to dropout technique in that it introduces the training uncertainty and increase the entropy to reduce the overfitting of the student model to specific stages of the teacher. Next, we will give theoretical derivations to show why our replacement training, similar to dropout, can increase the entropy of training.

Firstly, we provide a uniform formal definition of both dropout and our replacement training. Dropout can be expressed as: if $b(p) = 1$, $h^{'} = h / p$, and if $b(p) = 0$, $h^{'} = 0$, where $h$ is the output of a specific neuron, $h^{'}$ is the output after applying dropout, and $b(p)$ is the Bernoulli sampling with probability $p$. And for our replacement training, $f_{i}$ can be defined as: if $b(p) = 1$, $f_{i} = SS_{i}(f_{i-1})$, and if $b(p) = 0$, $f_{i} = TS_{i}(f_{i-1})$, where $f_{i-1}$ and $f_{i}$ are the input and output feature of the $i$-th stage, and $SS_{i}$ and $TS_{i}$ denote the $i$-th student stage and teacher stage. Without the replacement training, $f_{i} = SS_{i}(f_{i-1})$.

After providing a unified representation of dropout and our replacement training, we can give a derivation of our replacement training based on the formula of dropout. Entropy $H(X)$ can be used to measure the uncertainty of a random variable $X$, which is defined as: $H(X)= - \sum_{x} P(x)\mathrm{log}P(x),$ where $P(x)$ is the probability distribution of the random variable $X$ taking values $x$. Thus, the origin entropy of $f$ can be written as: $H(f_{i})= -\sum_{f_{i}} P(f_{i})\mathrm{log}P(f_{i}).$

Because $P(f_{i}) = P(SS_{i}(f_{i-1}))$, entropy $H_{SS}(f_{i})$ of student stage is: $H_{SS}(f_{i})= -\sum_{f_{i-1}} P(SS_{i}(f_{i-1}))\mathrm{log}P(SS_{i}(f_{i-1})),$ and entropy $H_{TS}(f_{i})$ of teacher stage can be written as: $H_{TS}(f_{i})= -\sum_{f_{i-1}} P(TS_{i}(f_{i-1}))\mathrm{log}P(TS_{i}(f_{i-1})).$

With our replacement training, the $P(f_{i})$ is: if $b(p) = 1$, $P(f_{i}) = pP(SS_{i}(f_{i-1}))$, and if $b(p) = 0$, $P(f_{i}) = (1-p)P(TS_{i}(f_{i-1}))$.

Thus, the entropy of $P(f_{i})$ can be written as:

$H(f_{i}) = -\sum_{f_{i}} P(f_{i})\mathrm{log}P(f_{i})$

$= -[\sum_{f_{i-1}}(1-p)P(TS_{i}(f_{i-1})) \mathrm{log}((1-p)P(TS_{i}(f_{i-1}))) + \sum_{f_{i-1}} pP(SS_{i}(f_{i-1})) \mathrm{log}(pP(SS_{i}(f_{i-1})))]$

$= -[(1-p)\mathrm{log}(1-p)\sum_{f_{i-1}}P(TS_{i}(f_{i-1}))] + (1-p)\sum_{f_{i-1}}P(TS_{i}(f_{i-1}))\mathrm{log}P(TS_{i}(f_{i-1})) + p\mathrm{log}p\sum_{f_{i-1}}P(SS_{i}(f_{i-1})] + p\sum_{f_{i-1}}P(SS_{i}(f_{i-1}))\mathrm{log}P(SS_{i}(f_{i-1}))$

$= -[(1-p)\mathrm{log}(1-p)+p\mathrm{log}p] + (p-1)H_{TS}(f_{i}) + pH_{SS}(f_{i}).$

Thus, the difference in entropy $\Delta H(f_{i})$ is:

$\Delta H(f_{i}) = -[(1-p)\mathrm{log}(1-p)+p\mathrm{log}p]+ (p-1)H_{TS}(f_{i}) + pH_{SS}(f_{i}) - H_{SS}(f_{i})$

$= -[(1-p)\mathrm{log}(1-p)+p\mathrm{log}p] + (1-p)(H_{SS}(f_{i}) - H_{TS}(f_{i})).$

When $p$ is in range $[0,1]$, the term $-[(1-p)\mathrm{log}(1-p)+p\mathrm{log}p]$ is always positive. Since the student model generally performs worse than the teacher model, $H_{SS}(f_{i}) - H_{TS}(f_{i})$ is typically positive as well. Thus, $\Delta H(f_{i})$ is consistently positive. This demonstrates that our replacement training achieves a similar effect to dropout: increases training entropy.

Based on the above theoretical analysis, we demonstrate that our replacement training method is similar to dropout in its ability to increase training entropy. Consequently, our replacement training helps reduce the overfitting of the student model to specific stages of the teacher.