General Compression Framework for Efficient Transformer Object Tracking

Authors: Lingyi Hong, Jinglun Li, Xinyu Zhou, Shilin Yan, Pinxue Guo, Kaixun Jiang, Zhaoyu Chen, Shuyong Gao, Wei Zhang, Hong Lu, Wenqiang Zhang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Object Tracking, Efficient Tracking, Lightweight Model, Knowledge Distillation

Abstract:
Transformer-based trackers have established a dominant role in the field of visual object tracking. While these trackers exhibit promising performance, their deployment on resource-constrained devices remains challenging due to inefficiencies. To improve the inference efficiency and reduce the computation cost, prior approaches have aimed to either design lightweight trackers or distill knowledge from larger teacher models into more compact student trackers. However, these solutions often sacrifice accuracy for speed. Thus, we propose a general model compression framework for efficient transformer object tracking, named CompressTracker, to reduce the size of a pre-trained tracking model into a lightweight tracker with minimal performance degradation. Our approach features a novel stage division strategy that segments the transformer layers of the teacher model into distinct stages, enabling the student model to emulate each corresponding teacher stage more effectively. Additionally, we also design a unique replacement training technique that involves randomly substituting specific stages in the student model with those from the teacher model, as opposed to training the student model in isolation. Replacement training enhances the student model's ability to replicate the teacher model's behavior. To further forcing student model to emulate teacher model, we incorporate prediction guidance and stage-wise feature mimicking to provide additional supervision during the teacher model's compression process. Our framework CompressTracker is structurally agnostic, making it compatible with any transformer architecture. We conduct a series of experiment to verify the effectiveness and generalizability of CompressTracker. Our CompressTracker-4 with 4 transformer layers, which is compressed from OSTrack, retains about $\mathbf{96\%}$ performance on LaSOT ($\mathbf{66.1\%}$ AUC) while achieves $\mathbf{2.17\times}$ speed up.

Decision: reject
Reviewer scores: 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1729234453064

thread_type:
official_review

<thread_object>
creator: reviewer_3q3t

<content>
summary:
The paper presents CompressTracker, a framework for general model compression designed to enhance the efficiency of transformer-based object-tracking models. This approach utilizes a unique stage partitioning strategy that divides the transformer layers of the teacher model into distinct stages, allowing the student model to simulate each corresponding stage better.

Furthermore, the authors introduced a replacement training technique, where specific stages in the student model are randomly replaced with those from the teacher model. This strategy, combined with predictive guidance and staged feature imitation, provides additional supervision to help the student model mimic the teacher model more effectively during the compression process.

Extensive experiments on vit-based trackers were conducted, which show the proposed method can lightweight the trackers while main a comparable performance.

strengths:
1. The author clearly presents a framework for compressing single object tracking models, effectively reducing larger models to smaller, efficient versions. Extensive experiments demonstrate the method's effectiveness.

2. One of the main benefits of the proposed framework is its structural agnosticism, meaning it can work with any transformer architecture. This adaptability allows CompressTracker to fit different student model configurations, making it suitable for various deployment environments and computational limits.

3. The paper shows through extensive experiments that CompressTracker strikes an impressive balance between inference speed and tracking accuracy. It significantly speeds up the tracking process while preserving high performance, achieving nearly 96% of the original accuracy with a 2.17× increase in speed.

weaknesses:
1. The primary drawback of this method lies in its dependence on various distillation techniques, such as different training strategies, feature mimicking, and loss guidance. This lack of a clear, consistent framework among these techniques may undermine the generalization ability and transferability of the proposed approach, despite the author's assertions to the contrary.

2. Moreover, the overall complexity of the method raises concerns about its usability for other researchers. For instance, when applied to Mixformer V2, which has only two layers, the improvement in performance is minimal, while the processing speed remains unchanged. Such results indicate possible limitations of the method, as the intricate techniques lead to only marginal benefits.

3. The proposed techniques (stage division, progressive replacement, Replacement Training, Prediction Guidance, and Stage-wise Feature Mimicking) appear to be independent. The title "General Framework" raises my expectations significantly.

questions:
See weakness.  Moreover, the paper does not compare with other model compression techniques, such as knowledge distillation, model quantization, and pruning. It helps if you provide some comparisons or analysis.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730608758011

thread_type:
official_review

<thread_object>
creator: reviewer_aucs

<content>
summary:
This paper introduces a general model compression framework based on the teacher-student knowledge distillation method for efficient transformer object tracking, named CompressTracker, which designs a stage division strategy and a replacement training technique.

strengths:
The motivation of this paper is clear and it has certain innovation.

weaknesses:
Two classical models are only used to verify the effectiveness of the proposed CompressTracker, which are not enough to demonstrate its applicability and scalability.

questions:
1. OSTrack and MixFormer are only used to verify the effectiveness of the proposed CompressTracker. As a general model compression framework, this is far from enough.
2. This paper proposes a novel stage division strategy. To demonstrate its effectiveness, the corresponding ablation experimental results are shown in Table 8. It can be seen from Table 8 that the even dividing strategy is better than the uneven dividing strategy, but it cannot indicate that the performance of model with the stage division strategy is better than that without the stage division strategy.
3. What is the difference of the ablation experiments in Table 9 and 10?
4. Does CompressTracker only apply to the model with several same feature extractor modules? If the structures of different stages are different, does it work?
5. The format of Table 5 is non-standard. A table should not contain other tables.
6. There is spelling mistake, such as "Stucture Limitation" in Page 1.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730654694600

thread_type:
official_review

<thread_object>
creator: reviewer_hzre

<content>
summary:
The paper presents CompressTracker, a novel general model compression framework aimed at enhancing the efficiency of transformer-based object tracking models for deployment on resource-constrained devices. The framework employs a stage division strategy to segment the teacher model into distinct stages, which are then emulated by a lighter student model. CompressTracker introduces a replacement training technique, where student model stages are dynamically replaced with teacher model stages during training, enhancing the student's ability to replicate the teacher's behavior. Additionally, prediction guidance and stage-wise feature mimicking are incorporated to refine the learning process. The framework is structurally agnostic and compatible with various transformer architectures.

strengths:
The paper proposes an innovative approach to segmenting transformer layers into stages, allowing for more granular knowledge transfer from a teacher model to a student model.

The experiments show that CompressTracker can achieve a substantial speedup while maintaining a high level of accuracy, which is crucial for real-world applications on resource-constrained devices.

The framework's compatibility with any transformer architecture is a significant advantage, as it increases its applicability and flexibility.

weaknesses:
There is still a performance gap between the teacher and student models, indicating that there might be room for further improvement in the compression strategy to achieve lossless compression. 

While the framework simplifies the compression process, the introduction of multiple training strategies might increase the complexity of the training regimen, which could be a barrier for some users.

questions:
In the article, the framework proposed is added to OSTrack, where the search area pixels are 256× 256and the template pixels are 128× 128. What is the effect when the search area pixels are 384 × 384 and the template pixels are 192 × 192? Can tracking accuracy still be maintained?

What are the limitations of the stage division strategy, and how does it affect the generalization capabilities of the student model?

According to the experimental section, it can be seen that speed and accuracy cannot be achieved simultaneously. The faster the speed, the lower the accuracy. Taking OSTrack as an example, the encoder layer of OSTrack is 12 layers. Would it be possible to achieve a similar effect by reducing the number of layers appropriately? Will the compression method proposed in the article have more advantages in accuracy and speed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707402337

thread_type:
official_review

<thread_object>
creator: reviewer_ejyx

<content>
summary:
In this paper, the authors proposed a general model compression framework for efficient Transformer object tracking, named CompressTracker. The method adopts a novel stage partitioning strategy to divide the Transformer layers of the teacher model into different stages, enabling the student model to more effectively simulate each corresponding teacher stage. The authors also designed a unique replacement training technique, which involves randomly replacing specific stages in the student model with specific stages in the teacher model. Replacement training enhances the student model's ability to replicate the behavior of the teacher model. To further force the student model to simulate the teacher model, we combine predictive guidance and staged feature imitation to provide additional supervision during the compression process of the teacher model. The authors conducted a series of experiments to verify the effectiveness and generality of CompressTracker.

strengths:
The author has clear ideas and the article is easy to understand. He proposes a general compression framework for single object tracking. This method can efficiently compress large object tracking models into small models. The author has conducted a large number of experiments to prove the effectiveness of this method.

weaknesses:
1. The innovation is slightly insufficient. The author’s innovation focuses on replacement training, prediction guidance, and feature mimicking. The latter two are common methods of distillation and are not enough to be the innovation of this article. Therefore, the innovation of this article is more focused on the replacement training strategy, but the author’s explanation of the intuitive reasons why this strategy is useful is poor.
2. The author claims that the method is general compression framework, but the paper only experiments on OSTrack and MixFormerV2. However, there are many trackers based on transforemr, such as SimTrack, ODtrack, LoraT, etc. The reviewer thinks that it cannot be called general compression framework after only verifying two trackers.
3. The author only conducted experiments on GPUs with sufficient computing power. However, efficient trackers are more targeted at devices with insufficient computing power, such as CPUs and edge devices. The author did not conduct experiments on such devices to verify the term efficient.

questions:
The application scenarios of efficient tracking models are mostly devices with insufficient examples. The author should provide the speed of the model on the CPU or edge device to verify the word "efficient", rather than just testing the speed on the GPU. For other issues, see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731836426209

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely appreciate the thorough review provided by all the reviewers. The valuable feedback from the reviewers has significantly contributed to enhancing the quality of our manuscript. We extend our gratitude to Reviewer hzRe and Reviewer AUCs for acknowledging the novelty of our work. Their positive recognition of the innovation in our research is greatly appreciated. All reviewers affirm the effectiveness of our work. Furthermore, we kindly request Reviewer ejyx, Reviewer hzRe and Reviewer AUCs to reconsider our work after reviewing our response. Your reconsideration will be highly valued.

Based on the comments from the reviewers, I have summarized the strengths of our paper as follows:

1. **Clear motivation and certain innovation.** (ejyx, hzRe, AUCs)
2. **Compatibility** with any transformer architecture to fit various deployment environments and computational limits. (hzRe, 3q3t)
3. High efficiency and effectiveness, and **impressive** balance between inference speed and tracking accuracy. (ejyx, hzRe, 3q3t)
4. Extensive experiments to verify the **effectiveness** of our CompressTracker. (ejyx, 3q3t)

We have summarized our novelty as follows:

1. We introduce CompressTracker, a novel and general model compression framework for efficient transformer-based object tracking.
2. We propose a stage division strategy, allowing for fine-grained imitation of teacher at the stage level, which enhances both the precision and efficiency of knowledge transfer.
3. We introduce replacement training to enhance the student model's ability to effectively replicate the teacher model's behaviour.
4. We further integrate prediction guidance and feature mimicking to accelerate the learning process.
5. CompressTracker overcomes traditional structural limitations, adapting flexibly to various transformer architectures for student. It outperforms existing methods, accelerating OSTrack by **2.17×** while maintaining approximately **96%** of the original accuracy (66.1% AUC on LaSOT).

In response to the reviewers' general concerns regarding the generalization of our CompressTracker, we provide a unified response. We have conducted additional experiments and summarize the experiments in our paper with the results shown below. 

| # |  Model |  LaSOT | LaSOT_ext  | TNL2K | TrackingNet | UAV123 | FPS |
| -- | -- | -- | -- | -- | -- | -- | -- |
| |  | Model | Generalization |  |  | | | 
| 1 | CompressTracker-4 | 66.1 (96%) | 45.7 (96%) | 53.6 (99%) | 82.1 (99%) | 67.4 (99%) | 228 (2.17×) |
| 2 | CompressTracker-4-ODTrack | 70.5 (96%) | 50.9 (97%) | 60.4 (99%) | 82.8 (97%) | 69.2 (98%) | 86.5 (1.74×)|
| 3 | CompressTracker-4-SeqTrack | 68.1 (95%) | 47.9 (96%) | 54.5 (99%) | 83.1 (98%) | 68.4 (98%) | 62.1 (1.36×) | 
| | | Stage | Scabality |  | | |
| 4 | CompressTracker-2 | 60.4 (87%) | 40.4 (85%) | 48.5 (89%) | 78.2 (94%) | 62.5 (92%) | 346 (3.30) |
| 5 | CompressTracker-3 | 64.9 (94%) | 44.6 (94%) | 52.6 (97%) | 81.6 (98%) | 65.4 (96%) | 267 (2.54×) |
| 6 | CompressTracker-4 | 66.1 (96%) | 45.7 (96%) | 53.6 (99%) | 82.1 (99%) | 67.4 (99%) | 228 (2.17×) |
| 7 | CompressTracker-6 | 67.5 (98%) | 46.7 (99%) | 54.7 (101%) | 82.9 (99%) | 67.9 (99%) | 162 (1.54×) | 
| 8 | CompressTracker-8 | 68.4 (99%) | 47.2 (99%) | 55.2 (102%) | 83.3 (101%) | 68.2 (99%) | 127 (1.21×) |
| | | Larger | Transformer | Scabality | | | |
| 9 | CompressTracker-4-L | 67.5 (96%) | 45.9 (98%) | 58.3 (98%) | 83.2 (99%) | 67.4 (99%) | 228 (2.84×) |
| | | Higher | Resolution | Scabality | | | |
| 10 | CompressTracker-4-384 | 67.7 (96%) | 48.1 (96%) | 54.3 (99%) | 82.7 (99%) | 68.2 (98%) | 228 (3.90×) |
| | | Heterogeneous | Structure | Robustness | | | |
| 11 | CompressTracker-M-S | 62.0 (88%) | 44.5 (88%) | 50.2 (87%) | 77.7 (93%) | 66.9 (96%) | 325 (1.97×) |
| 12 | CompressTracker-SMAT | 62.8 (91%) | 43.4 (92%) | 49.6 (91%) | 79.7 (96%) | 65.9 (96%) | 138 (1.31×) |

Our experiments involve 4 teacher models (OSTrack, OSTrack-384, OSTrack-L, ODTrack, MixFormerV2, SeqTrack) and 11 student models. We would like to emphasize that our CompressTracker is designed as a scalable framework, capable of adapting to variations in image resolution (e.g., #4-8, 10), teacher model size (#4-8, 9), and student model size (#4-8), different teacher models (#1-3, 11), and various student model architectures (#11, 12). Extensive experiments have demonstrated the strong generalization, scalability, and robustness of our CompressTracker. Our CompressTracker supports any kind of transformer structure and layer number in student model, any input resolution, and any kind of teacher model, achieving the general compresstion. 

We believe that the innovative contributions of our work significantly enhance its value in visual object tracking. We have addressed each reviewer's comments in detail and will incorporate reviewers' insightful suggestions, adding essential experiments. We kindly request reviewers to reconsider our research with these aspects in mind and extend their support.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a model compression framework aimed at enhancing the efficiency of transformer-based object tracking. It received mixed reviews, with scores of 5, 5, 6, and 6, resulting in an average score of 5.5.

Reviewers acknowledged the paper’s strengths, particularly its innovative approach to compressing transformer-based tracking models to improve efficiency, as highlighted by reviewers 3q3t and ejyx.

However, several weaknesses were noted, including limited technical contributions (pointed out by Reviewers ejyx and hzRe), a performance gap (identified by Reviewer hzRe), and insufficient experimental validation (mentioned by Reviewers AUCs and ejyx).

The Area Chair (AC) also assessed the paper and noted that the three technical contributions—stage division, replacement training, and prediction guidance/feature mimicking—are not closely related or unified. This makes the claim of a "general compression framework" somewhat overclaimed. The results show that while the proposed method achieves approximately a 2x speed increase, it results in a performance drop of 3% to 8%. Furthermore, the effectiveness of the method is inconsistent across different backbone variants and scales, indicating that more experiments are needed to validate its generalization ability.

After discussions among the authors, reviewers, and the AC, some concerns were addressed. Nevertheless, issues concerning technical novelty remain unresolved. The AC discussed this paper with the Senior AC in depth. They concluded that they could not recommend the acceptance of this paper and encouraged the authors to take the reviewers' comments into account to improve the paper for the next venue.