Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillation

Authors: andong  lu
jiacong  zhao
chenglong  li
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-21 02:22:36

Abstract: Modality gap between RGB and thermal infrared (TIR) images is a crucial issue but often overlooked in existing RGBT tracking methods. It can be observed that modality gap mainly lies in the image style difference. In this work, we propose a novel Coupled Knowledge Distillation framework called CKD, which pursues common styles of different modalities to break modality gap, for high performance RGBT tracking. In particular, we introduce two student networks and employ the style distillation loss to make their style features consistent as much as possible. Through alleviating the style difference of two student networks, we can break modality gap of different modalities well.
However, the distillation of style features might harm to the content representations of two modalities in student networks. To handle this issue, we take original RGB and TIR networks as the teachers, and distill their content knowledge into two student networks respectively by the style-content orthogonal feature decoupling scheme. We couple the above two distillation processes in an online optimization framework to form new feature representations of RGB and thermal modalities without modality gap. In addition, we design a masked modeling strategy and a multi-modal candidate token elimination strategy into CKD to improve tracking robustness and efficiency respectively. Extensive experiments on five standard RGBT tracking datasets validate the effectiveness of the proposed method against state-of-the-art methods while achieving the fastest tracking speed of 96.4 FPS.

Main paper: 
 introduction: 
 In recent years, the field of RGBT tracking attracts great attention due to its wide range of applications in surveillance, object recognition and other fields [1, 6, 17, 27, 28, 40, 42, 53, 56, 57] . RGBT tracking aims to take advantage of the complementary advantages of RGB and thermal modalities to achieve robust object tracking. However, visible spectrum and thermal infrared data are collected by cameras in different imaging bands, reflecting different properties of the target object. As a result, they differ significantly in appearance style, which inevitably leads to the issue of modality gap.Current research on RGBT tracking focuses on three categories, including multi-modal fusion design, multi-modal representation learning, and prompt learning. The first category of studies [26, 27] are usually devoted to designing a reliable late fusion module for 35 7UDFNLQJ6SHHG)36 7KH3HUIRUPDQFHvs. 6SHHGRQ/DV+H5 3UR7UDFN>$&000@ 9L37>&935@ 7%6,>&935@ &0'>&935@ 4$7>$&000@ 7$7UDFN>$$$,@ 6'67UDFN>&935@ &.'>2XUV@ Figure 1: Comparison of performance and speed for state-ofthe-art tracking methods on LasHeR [23] . We visualize the Precision Rate (PR) to the Frames Per Second (FPS). CKD is able to rank the 1st in PR while running at 96.4 FPS.the collaboration between RGB and thermal infrared (TIR) modality features. For instance, Liu et al. [26] proposes a quality-aware fusion module that carefully designs two independently weighted prediction branches to guide the fusion of multi-modal features. The second category of research [5, 17, 28, 45] focus on integrating multi-layer feature interaction modules into backbone network, thus can leverage modality complementary information to enhance the representation of each modality. For example, Hui et al. [17] introduce a Transformer-based multi-layer feature interaction module, which effectively enhances the exchange of information between modalities. The third category of methods [1, 47, 58] explore the concept of prompt learning in multi-modal feature interaction. For example, Cao et al. [1] design a bi-directional adapter for mutual prompting of modality information. However, existing studies have long overlooked the influence of modality gap, thus limiting the performance and efficiency of current RGBT tracker.To address this challenge, we propose a novel Coupled Knowledge Distillation framework (CKD), which pursues common styles of different modalities to break modality gap, for high performance RGBT tracking. Figure 1 presents the advantages of CKD compared to existing state-of-the-art methods in two different dimensional metrics, which suggest a powerful potential in breaking modality gap. Specifically, we analyse the impact of modality style on the modality gap in Figure 2 , which can be seen that the modality gap is significantly reduced after removing the modality style. Therefore, the modality style plays an important factor in breaking modality gap. However, the structure of the modal feature distribution is also affected, which may harm the modal content representation.To this end, we introduce two student networks and design a style distillation scheme between the style features of the two students to make their style features as consistent as possible, aiming to eliminate the modality gap. Here, we utilize the feature mean and standard deviation to represent the style features of a modality, which has been proven to be effective in many studies [15, 16] . However, style feature distillation may harm modality content representations of both student branches. We further introduce the original RGB and TIR networks as teachers, and design a content distillation scheme to pursue the consistency between the content features of the teacher branch and the corresponding student branch. To alleviate the constraints on student style features, we employ a style-content orthogonal feature decoupling strategy that obtains content features by performing instance normalization on the original features. Consequently, we couple the above two distillation processes in an online optimization framework to pursue the common style of the two modalities while avoiding the damage to the modality content representation.In addition, we design a masked modeling strategy to further enhance the robustness of modality features in challenging scenarios. It involves the random mask to create data pairs of contentdegraded and non-degraded of one modality, and then feed into the teacher and student branches. We can employ content distillation loss in CKD to effectively learn representations for content reconstruction. We also design a simple and effective multi-modal candidate token elimination strategy, which collaboratively considers the information of the two modalities to jointly decide the candidate elimination tokens in the search region. By drop these tokens during the inference phase, we can achieve a balance between tracking performance and tracking efficiency.In summary, our major contributions are as follows.• We propose a novel coupled knowledge distillation framework CKD to handle the modality gap issue by eliminating the style difference between RGB and thermal images for high performance RGBT tracking. To the best of our knowledge, this research is the first effort to break the modality gap in RGBT tracking. • We design a style-content coupled distillation scheme based on style-content orthogonal feature decoupling, which effectively eliminates modality gap without harming the modality content representation. • We present a masked modeling scheme that is seamlessly integrated into CKD, which effectively enhances the learning of modality content representation. In addition, a multimodal candidate token elimination strategy is designed, which further improves the tracking efficiency. [1, 12, 14, 47, 58] focuses on modal information interaction in a prompting manner, and is also a recent hot research topic. For example, Yang et al. [47] and Zhu et al. [58] introduce the concept of prompt learning at the pixel level and the feature level of RGBT tracking, respectively, for efficient multi-modal tracking. However, these methods ignore the impact of modality gap, which limits the RGBT tracking performance. 
knowledge distillation: 
 Knowledge distillation [11] aims to transfer the knowledge from a pre-trained large-scale teacher model to a small-scale student model. According to the type of knowledge transferred, the current research is mainly divided into three categories. Probability-based methods [11, 36] leverage the teacher model's predicted probabilities, guiding the student model to emulate the teacher's log-class distribution through minimizing KL divergence. Conversely, Featurebased methods [3, 9, 49] harness the mid-layer output feature map of the network for supervising the training of the student model. Relation-based methods [32, 59] emphasize inter-sample relationships over single instances. While simple to implement, it requires complex teacher modeling and significant training time. To address these issues, some studies [34, 35] utilize a mutual learning strategy for knowledge transfer. In addition, cross-modality knowledge distillation aims to transfer knowledge between different modalities. Some studies [33, 54] use dominant modality to guide weaker ones, while others [7, 18] investigate inter-modal complementarity. In contrast to existing methods, the coupled knowledge distillation method proposed in this paper distills coupled modality content and modality style knowledge from different modalities, thus facilitating modality style transfer for modality content preservation achievable by student models. 
methodology: 
 In this section, we first introduce the overall architecture of Coupled Knowledge Distillation (CKD) and then describe the coupled knowledge distillation approach. Subsequently, mask modeling and multi-modal candidate token elimination are introduced. Finally, the training process and implementation details are given.We provide a detailed description of CKD framework, and its overall structure is shown in Figure 3 . During the training phase, our CKD framework comprises four branches (including a RGB teacher branch, a TIR teacher branch, a RGB student branch, and a TIR student branch) as well as three tracking heads (including a multimodal tracking head and two single-modal tracking heads). In the testing phase, CKD consists of two student branches and one multimodal tracking head. In specific, for the given RGB and TIR modal frames, the search and template frames are first partitioned into patches with the size of 𝑝 × 𝑝 using four independent learnable patch embedding layers, and flattened to obtain four search token sequences (𝑆 𝑡 𝑟𝑔𝑏 , 𝑆 𝑡 𝑡𝑖𝑟 , 𝑆 𝑠 𝑟𝑔𝑏 , 𝑆 𝑠 𝑡𝑖𝑟 ), and four template token sequences (𝑇 𝑡 𝑟𝑔𝑏 , 𝑇 𝑡 𝑡𝑖𝑟 , 𝑇 𝑠 𝑟𝑔𝑏 , 𝑇 𝑠 𝑡𝑖𝑟 ). Following [48] , we also add learnable position embeddings to the above tokens to provide positional prior information. Note that we add random mask to the search frame token sequences (𝑆 𝑠 𝑟𝑔𝑏 , 𝑆 𝑠 𝑡𝑖𝑟 ) in the two student branches during training. Then, we concatenate search and template frame token sequences for the four groups, denoted as𝐼 𝑡 𝑟𝑔𝑏 = [𝑆 𝑡 𝑟𝑔𝑏 ,𝑇 𝑡 𝑟𝑔𝑏 ], 𝐼 𝑡 𝑡𝑖𝑟 = [𝑆 𝑡 𝑡𝑖𝑟 ,𝑇 𝑡 𝑡𝑖𝑟 ], 𝐼 𝑠 𝑟𝑔𝑏 = [𝑆 𝑠 𝑟𝑔𝑏 ,𝑇 𝑠 𝑟𝑔𝑏 ],and𝐼 𝑠 𝑡𝑖𝑟 = [𝑆 𝑠 𝑡𝑖𝑟 ,𝑇 𝑠 𝑡𝑖𝑟 ].We feed 𝐼 𝑡 𝑟𝑔𝑏 and 𝐼 𝑡 𝑡𝑖𝑟 into the RGB and TIR teacher branches, respectively, while 𝐼 𝑠 𝑟𝑔𝑏 and 𝐼 𝑠 𝑡𝑖𝑟 into the RGB and TIR student branches, respectively. These branches share an identical structure, consisting of standard Transformer blocks [8] , but they have independent parameters. Finally, we concatenate the last layer features of the RGB and TIR student branches in the channel dimension and input them into the tracking head [48] for object localization and regression. The last layer features of the RGB and TIR teacher branches are separately fed into two independent tracking head [48] networks for task learning.The proposed CKD is the first effort in the field of RGBT tracking to address modality gap. In particular, CKD employs a style distillation to make the style features of the two modalities as consistent as possible, thus breaking modality gap. It also introduces a content distillation to ensure that the modality content representation is stable. Next, we describe two distillation methods, namely style distillation and content distillation, in detail.Style distillation aims at mutual distillation between the style features of the two student branches, thus pursuing a common style for both modalities to break modality gap. Feature style usually involves the statistical attributes of features, and existing studies [16] usually use the mean and standard deviation of features to represent feature style. Therefore, these two statistical attributes are adopted as feature styles in this study. where 𝑙 denotes the feature from the 𝑙-th layer, 𝐵, 𝑁 , and 𝐷 represent the batch size, number of tokens, and channel dimension of tokens, respectively. For brevity, the 𝐵 dimension is omitted below. The process of calculating the mean and standard deviation vectors of the two student branch features is as follows:𝜇 𝑠 (𝑙 ) 𝑟𝑔𝑏 = 1 𝑁 𝑁 ∑︁ 𝑛=1 𝑓 𝑙 (𝑛) 𝑟𝑔𝑏 , 𝜎 𝑠 (𝑙 ) 𝑟𝑔𝑏 = 1 𝑁 𝑁 ∑︁ 𝑛=1 (𝑓 𝑙 (𝑛) 𝑟𝑔𝑏 -𝜇 𝑠 (𝑙 ) 𝑟𝑔𝑏 ) 2 , (1) 𝜇 𝑠 (𝑙 ) 𝑡𝑖𝑟 = 1 𝑁 𝑁 ∑︁ 𝑛=1 𝑓 𝑙 (𝑛) 𝑡𝑖𝑟 , 𝜎 𝑠 (𝑙 ) 𝑡𝑖𝑟 = 1 𝑁 𝑁 ∑︁ 𝑛=1 (𝑓 𝑙 (𝑛) 𝑡𝑖𝑟 -𝜇 𝑠 (𝑙 ) 𝑡𝑖𝑟 ) 2 . (2)where 𝜇 𝑠 𝑟𝑔𝑏 and 𝜎 𝑠 𝑟𝑔𝑏 represent the mean and standard deviation vectors of RGB student features, respectively, while 𝜇 𝑠 𝑡𝑖𝑟 and 𝜎 𝑠 𝑡𝑖𝑟 indicate the corresponding vectors for TIR student features. Then, we can compute the style distillation loss, denoted as L 𝑆𝐷 , by quantifying the mean squared error between the style features across all layers of the two student branches.EQUATIONBy minimizing this style distillation loss during training, we can achieve style consistency across different modality features.Although RGB and TIR features of two student branches can eliminate modal gaps through style distillation, this process may harm the modal feature content representation. To alleviate this issue, we perform content distillation between teacher and student branches with the same modality input. To avoid imposing constraints on the style features in student branch, we adopt the classical instance normalization operation to normalize teacher and student features, thus obtaining content features that are orthogonal to the style features. We can then calculate the similarity between the content features of the two groups of teachers and students for content distillation.In particular, we take the TIR modality as an example and can describe above process as follows. Firstly, we perform feature instance normalization along the channel dimension as follows:EQUATION)where 𝐹 𝑙 𝑡𝑖𝑟 denotes the intermediate feature from TIR teacher branch, 𝑑 represents the 𝑑-th channel dimension of each token. F𝑙 𝑡𝑖𝑟 represents the 𝑙-th layer normalized feature (i.e., content feature) of TIR teacher branch, and f 𝑙 𝑡𝑖𝑟 denotes the 𝑙-th layer content feature of student branch. 𝜇 𝑡 𝑡𝑖𝑟 , 𝜎 𝑡 𝑡𝑖𝑟 , 𝜇 𝑠 𝑡𝑖𝑟 , and 𝜎 𝑠 𝑡𝑖𝑟 correspond to the mean and standard deviation, respectively. Subsequently, we calculate the TIR feature content distillation loss, denoted as L 𝑡𝑖𝑟 𝐶𝐷 , by measuring the mean squared error (MSE) between the content features across all layers of teacher and student branches:EQUATION)Minimizing this loss during training ensures consistency between the content features of TIR student branch and those of TIR teacher branch. Similarly, this approach is applied in RGB student branching learning by incorporating a constraint from RGB teacher branch to maintain stability in content features. Therefore, the total content distillation loss can be defined as follows:EQUATIONwhere L 𝑟𝑔𝑏 𝐶𝐷 denotes the RGB feature content distillation loss. In summary, coupled knowledge distillation is a combination of style distillation and content distillation that can break the modality gap in RGBT tracking. Thus, by incorporating CKD into our task, we can train two student branches that extract style-consistent features from different modalities while preserving their individual semantic content features.Inspired by recent success and scalability of pretraining with masked reconstruction in different domains [4, 10] , we design a novel masked modeling in the proposed framework to enhance modality content representation in challenging scenarios. Since there is natural feature content supervision between teacher and student branches of the same modality input in CKD, we can implement mask modeling without introducing any additional loss and design, simply by applying random mask to the input of student branches.In particular, we randomly mask some of the input tokens of a student branch with 𝑚 ∈ {0, 1} 𝑁 , where 𝑁 is the number of tokens. Hence the masked tokens are represented as {𝐼 𝑠 𝑟𝑔𝑏 |𝑚 𝑖 = 1} and {𝐼 𝑠 𝑡𝑖𝑟 |𝑚 𝑖 = 1} while the remain tokens are denoted as {𝐼 𝑠 𝑟𝑔𝑏 |𝑚 𝑖 = 0} and {𝐼 𝑠 𝑡𝑖𝑟 |𝑚 𝑖 = 0}. Subsequently, we feed these tokens into the corresponding student branches, respectively. It is noteworthy that the input tokens for corresponding teacher branch remain unchanged. Consequently, we can utilize the token features from two teacher branches to guide the feature learning of the masked tokens in two student branches. In fact, the aforementioned process can be realized by minimizing the content distillation loss, which seeks feature content consistency between the masked student tokens and the teacher tokens. Therefore, the masked modeling can be seamlessly integrated into CKD. In addition, we set the mask ratio as 25% empirically.The effectiveness of using candidate elimination strategies to provide inference efficiency has been demonstrated in [48] . Specifically, existing RGB trackers determine which tokens to eliminate by leveraging the attention weights established by the tokens in target and search regions. However, this strategy ignores that the elimination results are unreliable when the input data are of poor quality. To address this issue, we propose a multi-modal candidate token elimination strategy, which aims to improve elimination quality in challenging scenarios by combining attention weights of two modalities through cooperative making. Given the query vector 𝑞 𝑧𝑖 𝑟 from RGB template queries and the query vector 𝑞 𝑧𝑖 𝑡 from TIR template queries, a scalar ℎ 𝑖 is assigned to each token in the search region, calculated as follows:EQUATIONwhere 𝑘 𝑥 𝑟 and 𝑘 𝑥 𝑡 represent the key vectors of tokens in the search region. We use ℎ to sort the tokens in the search region and keep the top-k tokens. Therefore, the proposed method not only enhances the robustness of token elimination in challenging scenarios, but also improves the inference speed of the model.In CKD, we can define the final loss function as a combination of content distillation, style distillation, and task losses, as follows:EQUATIONHere, L 𝑡𝑎𝑠𝑘 denotes the tracking task loss [48] . 𝜆 𝑐𝑑 and 𝜆 𝑠𝑑 are the loss coefficients corresponding to the two loss terms, respectively.In our study, we set the ratio of 𝜆 𝑠𝑑 and 𝜆 𝑐𝑑 to 2:1, respectively. 
experiments 4.1 implementation details: 
 We choose OSTrack [48] as our foundational tracker, employing ViT [8] as its feature extractor. For parameter initialization, we utilize the original pre-trained model of OSTrack-base-256 [48] .For each sequence in a given training set, we collect the training samples and subject them to standard data augmentation operations, including rotation, translation, and gray-scale, aligning with the data processing scheme of the base tracker [48] . During training, the entire model utilizes AdamW to minimize the classification and regression loss functions. We use the LasHeR training set to train the entire tracking network in an end-to-end manner, which is used to evaluate GTOT [19] , RGBT210 [24] , RGBT234 [20] , and LasHeR [23] . For the evaluation of VTUAV [31] , we utilize the training set from VTUAV as the training data. In addition, we set the learning rate of the backbone network to 5e-6, and the tracking head to 5e-5. The CKD implementation is conducted on the PyTorch platform using two Nvidia A100 GPUs with 40G memory, and a global batch size of 40. The model fine-tuning takes 30 epochs that each epoch contains 60000 sample pairs. For VTUAV, we fine-tuned 5 epochs. In our study, we utilize precision rate (PR) and success rate (SR) as the main evaluation metrics for one-pass evaluation (OPE), which are commonly employed in current RGBT tracking tasks. These metrics enable a quantitative analysis of tracking performance. PR evaluates the fraction of frames where the distance between the tracker's output position and the true bounding box value falls below a predetermined threshold. Note that, we set the threshold to 5 pixels for the GTOT dataset and 20 pixels for other datasets, thereby calculating a representative PR score. SR the percentage of successfully tracked frames whose Intersection over Union (IoU) is greater than a specified threshold, and then SR metric is calculated by varying the threshold and computing the Area Under the Curve (AUC) of the resultant curve. To eliminate these effects, the PR is normalized using the scale of the ground truth box to calculate the NPR. The normalized accuracy curve can be obtained by changing the normalization threshold, and the region under the normalized accuracy curve with the normalization threshold in the range of [0, 0.5] is calculated as the representative NPR score.We evaluate our algorithm on five popular RGBT tracking benchmarks and compare its performance with current state-of-the-art trackers. The effectiveness of our proposed method is demonstrated in Table 1 , which provides a summary of the comparison results.GTOT dataset are shown in Table 1 . Compared to state-of-theart trackers, our method exhibits superior performance in GTOT dataset, achieving gains over QAT [26] by 1.0% in PR. We further compare our method with CMPP [38] , the best-performing tracker on the PR metric for this dataset. Although the PR of our method is 0.1% below that of CMPP [38] , our CKD outperforms CMPP by 2.0% in SR. As for our low PR, we attribute this to the prevalence of small objects in the GTOT dataset, for which CMPP's feature pyramid strategy aggregates features across all layers to enhance the feature representation capabilities. Additionally, CMPP builds a historical information pool using external storage, improving the representation of the current frame. However, these strategies significantly affect the efficiency of CMPP, making our CKD approximately 90 times faster than CMPP.As shown in Table 1 , CKD exceeds almost state-of-the-art trackers in RGBT210 dataset. Compared to mfDiMP [31] , which is the winner of VOT2019-RGBT, CKD achieves significant improvements in PR/SR with a gain of 9.8%/9.7%. Moreover, compared with TBSI [17] , which is the second best-performing algorithms in terms of SR, CKD outperforms it by 2.9%/2.7% on the PR/SR metrics. We further compare CKD with QAT [26] , which are the second best-performing algorithms in terms of PR, and CKD outperforms it by 1.6%/3.3% on PR/SR.To further evaluate the effectiveness of CKD, we conduct a series of experiments on RGBT234 dataset, including overall and attribute-based comparison. Overall Comparison. RGBT234 dataset is one of the most important datasets in the field of RGBT tracking, and also the dataset with the most evaluation results of existing algorithms. Therefore, we evaluate CKD against 24 state-of-the-art RGBT trackers on RGBT234 dataset. The evaluation results are presented in Table 1. CKD outperforms all state-of-the-art RGBT methods in PR/SR metrics, and obtains new SOTA results of 90.0%/67.4% in PR/SR. Compared with QAT [26] and TATrack [39] algorithms, which are the second best-performing algorithms in terms of PR and SR, respectively, CKD outperforms them by 1.6% and 3.0% in both PR and SR metrics.Challenge-based Comparison. We also present the results of CKD against the most advanced RGBT trackers available, including BAT [1] , SDSTrack [14] , TBSI [17] , and ViPT [58] , on different challenge subsets. The evaluation results are shown in 4, where the marks of each corner represent the attributes of the challenge subset and the highest and lowest performance under that attribute, respectively. These attributes include no occlusion (NO), partial occlusion (PO), heavy occlusion (HO), low illumination (LI), low resolution (LR), thermal crossover (TC), deformation (DEF), fast motion (FM), scale variation (SV), motion blur (MB), camera moving (CM) and background clutter (BC). The results show that our method performs best in all challenge subsets, and it proves that CKD has great potential in various complex tracking scenarios. 1 . CKD again outperforms all existing trackers by a clear margin. For instance, compared to BAT [1] , the second best performing algorithm in this dataset, CKD exhibits a 3.0%/2.0% performance advantage on PR/SR metrics. The experiment further verifies the effectiveness of our approach in more complex scenarios. 4.3.5 Evaluation on VTUAV dataset. We evaluate the proposed method CKD on VTUAV dataset, a recently proposed drone perspective RGBT tracking dataset. From Table 1 , it can be seen that CKD obtains 90.2%/77.8% on PR and SR metrics, which again confirm its effectiveness. Moreover, CKD surpasses the state-of-the-art MACFT [29] by 10.1% and 11.0% in PR and SR scores, respectively. The experiment further verifies the effectiveness of our approach in drone tracking scenarios.To verify the effectiveness of the proposed method, several ablation studies are performed on RGBT234 and LasHeR datasets. 2 , we conduct ablation studies on RGBT234 and LasHeR datasets to verify the effectiveness of different designed modules in CKD. Our baseline structure is the same as CKD, along with consistent training data and task losses, to fairly verify the effectiveness of the proposed components. w/ SD denotes the baseline equipped with style distillation, which achieves a certain improvement. The experiment shows that aligning modality styles is effective, but there are limitations.w/ SD CD indicates that adding content distillation to w/ SD results in significant performance improvements. The experiment shows that it is crucial to preserve the stability of modality content representation, as unconstrained style distillation could harm modality content representation, which could explain the limitations of w/ SD. w/ SD CD MM represents that adding masked modeling to w/ SD CD. The experiment demonstrates the effectiveness of the masked modeling strategy. 4.4.2 Impact of Pretrained model. We also explore the DropMAE [43] pretrained model trained on the Kinetics700 dataset [2] as our pretrained model, which further achieves significant performance gains. Compared to the "SOT" pretrained model usually exploited by existing RGBT tracking methods [1, 12, 17, 44] , DropMAE can bring superior performance to RGBT tracking. The experiment provides insights to further improve RGBT performance.To verify the effectiveness of the proposed multi-modal candidate token elimination strategy, we evaluate different token elimination methods in Table 3 . Here, CKD 𝑠𝑙𝑜𝑤 represents the CKD method without a token elimination strategy, but it is still faster than existing RGBT trackers.w/ CE [48] indicates that the two student branches individually apply the candidate token elimination strategy, following [48] . However, although CE brings an improvement in tracking efficiency, it also causes a significant performance drop.w/ MCE indicates that the two student branches follow the proposed multi-modal candidate token elimination strategy for collaborative token elimination. It can be seen that MCE achieves a balance between tracking efficiency and accuracy. Impact of loss weights. We explore the influence of different loss weights between style and distillation losses on CKD performance in Figure 5 . From Figure 5 it can be observed that the two kinds of losses in CKD are robust to the hyperparameters for these weights. Impact of masked ratios. As shown in Table 4 , we analyze the influence of different masked ratios on masked modeling strategy in CKD. It can be observed that the performance of CKD is always improved after the introduction of mask modeling, but the performance decreases slightly with the increase of mask modeling. 4.4.5 Analysis of feature decoupling scheme. In Table 5 , we design several variants to verify the effectiveness of feature decoupling.baseline w/ IN denotes the introduction of instance normalization in both student branches, which performs tracking with only content features. The experiment shows that modality style features certain discriminative information, which can lead to performance loss when directly dropped.baseline w/ FD represents the introduction of non-decoupled feature distillation (FD) only between two student branches. The experiment suggests that the non-decoupled distillation scheme may harm the modality content representation.baseline w/ SD is to perform distillation only in the style features between two student branches. The experiment further verifies that performing distillation for all modal features is unnecessary.baseline w/ CKD is the coupled distillation scheme proposed in this paper. The experiment further demonstrating that the importance of feature decoupling scheme. 4.4.6 Visual analysis. In Figure 6 , we visualize and compare the TIR content features extracted by the models trained with different distillation methods and not trained with distillation methods, and show their similarity relationships. The experiment shows that the proposed CKD method achieves a good balance between modality gap elimination and modality content representation preservation. 
conclusion: 
 In this work, we present a novel Coupled Knowledge Distillation (CKD) for RGBT tracking, which is the first effort to break the modality gap challenge in RGBT tracking. We first analyze the influence of modality style on modality gap, and then the proposed CKD can effectively enhance the consistency of modality style and avoid harming to modality content representation. Moreover, the proposed masked modeling strategy and a multi-modal candidate token elimination strategy effectively improve tracking performance and efficiency. Extensive experiments demonstrate the superiority of the proposed method. In the future, we will explore the benefits of the proposed CKD in other multi-modal visual tasks, such as RGBD/RGBE tracking [60, 63] , image fusion [25, 55] and collaborative learning [46] . 
 
 Appendix:  


Decision: The paper is fine from all reviewers, the responses well addressed the concerns from the reviewers.

Reviews: review: paper_summary: paper_summary: This paper focuses on the modality gap between RGB and thermal infrared images. The authors propose a novel Coupled Knowledge Distillation framework called CKD, which aims to bridge the modality gap by pursuing common styles of different modalities. Since the distillation of style features might harm the content representations of the two modalities in the student networks, the authors use the original RGB and TIR networks as teachers. They distill their content knowledge into two student networks respectively through a style-content orthogonal feature decoupling scheme.
strength_weakness: strength_weakness: In the paper, Coupled Knowledge Distillation is used to reduce cross-modal differences. Style distillation makes the style features of the two modalities as consistent as possible, while content distillation ensures that the modality content representation remains stable. This approach achieves state-of-the-art (SOTA) performance in certain settings on some datasets.
limitations: limitations: 1. The CKD framework comprises four branches: an RGB teacher branch, a TIR teacher branch, an RGB student branch, and a TIR student branch. The teacher branches typically use pre-trained large-scale models, which increases model complexity. Please analyze and compare the complexity of the models.

2. The intermediate features 𝑓_{𝑡𝑖𝑟}^l from the TIR student branch and 𝑓_{rgb}^l from the RGB student branch are essential. Could you explain the specific process of how these intermediate features are obtained within the framework?

3. In Formula (1), when calculating the standard deviation vectors of RGB student features 𝜎^{𝑠(l)}_{𝑟𝑔𝑏}, why is the  𝜇_{𝑟𝑔𝑏}^t used instead of the 𝜇_{𝑟𝑔𝑏}^s.

4. In Figure 3, how do the RGB student features and TIR student features become style features and content features without interfering with each other?
paper_summary: paper_summary: The paper addresses the challenge of the modality gap between RGB and TIR images in RGBT tracking. The proposed solution, a novel CKD framework, aims to harmonize the style features across modalities to enhance tracking performance. This approach leverages dual student networks and a style-content orthogonal feature decoupling scheme, maintaining the integrity of content representations while aligning style features. Extensive evaluations across multiple RGBT tracking datasets demonstrate the effectiveness and efficiency of CKD.
strength_weakness: strength_weakness: 1. The paper is well-organized and articulates complex concepts clearly and concisely. 
2. CKD uniquely addresses the modality gap by harmonizing the style features between RGB and TIR modalities while preserving their content integrity through a style-content orthogonal feature decoupling scheme. 
3. The CKD framework is rigorously evaluated across five standard RGBT tracking datasets, demonstrating its superiority over existing state-of-the-art methods.
limitations: limitations: 1. The paper extensively compares the proposed CKD framework against other state-of-the-art methods primarily based on knowledge distillation. A broader comparative analysis including non-distillation based approaches could provide a more comprehensive understanding of CKD's performance relative to the entire spectrum of RGBT tracking methodologies.
2. Although the paper reports impressive speeds of 96.4 FPS, the computational complexity and resource requirements are not thoroughly discussed. Given the dual-network architecture and the intricate distillation processes, the actual resource demand during training and deployment could be significant, which might limit its applicability in resource-constrained environments.
3. The evaluations focus on standard datasets, which may not fully represent challenging real-world conditions such as extreme weather, varying illumination, or occluded environments. The performance of the CKD framework under such diverse and challenging conditions remains less explored.
4. The paper could enhance its methodological rigor by providing a more detailed analysis of the sensitivity of the model's performance to various hyperparameters, such as the weights of the distillation and task losses.
paper_summary: paper_summary: This paper focuses on RGB-Thermal tracking
strength_weakness: strength_weakness: The authors propose a novel coupled knowledge distillation framework CKD to handle the modality gap issue by eliminating
the style difference between RGB and thermal images for high performance RGBT tracking
limitations: limitations: Why can the style and content information be de-coupled ?
Can the proposed method be applied to other settings (such ad cross-domain image retrieval)?
title: title: Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillation
paper_summary: paper_summary: This paper focuses on the modality gap between RGB and TIR modalities and tries to break this gap through the knowledge distillation.
It uses style and content transfers to achieve this goal. Besides, a token elimnination strategy is employed to discard the unhelpful tokens in challenging scenarios.
main_review: main_review: This paper focuses on the modality gap between RGB and TIR modalities and tries to break this gap through the knowledge distillation.
After reading, my conclusion is displayed below:

Strengths:
1. The paper is well-structured and easy for understanding.
2. Various experiments are conducted.
3. Very good performance and efficiency.

Weaknesses: 
1. The reasons for some of the designs are not clear: In introduction, the issue is the modality gap caused by the image style. It's reasonable to use techniques like style transfer, but the reason for using knowledge distillation is missed; For the content distillation, the authors claim that style distillation 'may' harm the feature embeddings.Although the performance is improved with content distillation, it still lacks further illustrations like visualisation of feature maps.
2.  Although the papes are limited, I believe that the qualitative analysis is not enough.
3. The most important problem is that it is not explained why we need to break the modality gap. In my opinion, we use the RGB and TIR data together because of the modality gap, which provides the complementary information.Furthermore, why breaking the modality gap can improve better performance.

Based on this, 'Marginally below acceptance threshold' is recommended since the most improtant part is not well explained. Also, the grade can be changed by explaining this well.
strength_weakness: strength_weakness: 1. The paper is well-structured and easy for understanding. 
2. Various experiments are conducted. 
3. Very good performance and efficiency.
The whole manuscript will be much better if the following weaknesses are solved.
limitations: limitations: 1. The reasons for some of the designs are not clear: In introduction, the issue is the modality gap caused by the image style. It's reasonable to use techniques like style transfer, but the reason for using knowledge distillation is missed; For the content distillation, the authors claim that style distillation 'may' harm the feature embeddings.Although the performance is improved with content distillation, it still lacks further illustrations like visualisation of feature maps. 
2. Although the papes are limited, I believe that the qualitative analysis is not enough. 
3. The most important problem is that it is not explained why we need to break the modality gap. In my opinion, we use the RGB and TIR data together because of the modality gap, which provides the complementary information.Furthermore, why breaking the modality gap can improve better performance. Based on this, 'Marginally below acceptance threshold' is recommended since the most improtant part is not well explained. Also, the grade can be changed by explaining this well.
4. Honestly, the similar thoughts of the modality gap can also be found in existing papers like MANet, MANet++ (modaliy-specific and modality-shared design), DFAT (data bias). So I don't think it's a new insight, and the authors also don't provide further analysis about this.