MDR: Multi-stage Decoupled Relational Knowledge Distillation with Adaptive Stage Selection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 13:24:45

Abstract: The effectiveness of contrastive-learning-based Knowledge Distillation (KD) has sparked renewed interest in relational distillation, but these methods typically focus on angle-wise information from the penultimate layer. We show that exploiting relational information derived from intermediate layers further improves the effectiveness of distillation. We also find that adding distance-wise relational information to contrastive-learning-based methods negatively impacts distillation quality, revealing an implicit contention between angle-wise and distance-wise attributes. 
Therefore, we propose a ${\bf{M}}$ulti-stage ${\bf{D}}$ecoupled ${\bf{R}}$elational (MDR) KD framework equipped with an adaptive stage selection to identify the stages that maximize the efficacy of transferring the relational knowledge.
MDR framework decouples angle-wise and distance-wise information to resolve their conflicts while still preserving complete relational knowledge, thereby resulting in an elevated transferring efficiency and distillation quality.
To evaluate the proposed method, we conduct extensive experiments on multiple image benchmarks ($\textit{i.e.}$ CIFAR100, ImageNet and Pascal VOC), covering various tasks 
($\textit{i.e.}$ classification, few-shot learning, transfer learning and object detection). 
Our method exhibits superior performance under diverse scenarios, surpassing the state of the art by an average improvement of 1.22\% on CIFAR-100 across extensively utilized teacher-student network pairs.

Main paper: 
 introduction: 
 Over the past decades, unprecedented development in neural networks has created numerous multimedia applications on vision and/or language, ranging from image classification [8, 13, 20] , object detection [25] , and visual question answering [17] . However, these neural networks demand substantial computational and storage resources due to their large model sizes, resulting in expensive and cumbersome model deployment. To address this limitation, various model compression techniques have been systematically explored, such as pruning [10] , quantization [7] , Neural Architecture Search (NAS) [33] , and Knowledge Distillation (KD) [12] . Among them, KD stands out for its compatibility with other compression techniques, superior generalization ability [2, 19, 42] and model structure flexibility, thus making it vital in applications such as object detection [3, 36] and Multiple Object Tracking (MOT) [16, 41] .KD aims to transfer knowledge from a heavy-weight model (teacher) to a light-weight one (student). A straightforward approach is to align the student's output probability distribution with that of the teacher [12] . However, due to the limited scope of information in this distribution, subsequent research has shifted towards matching outputs of intermediate layers [11, 27] , which further bifurcates into feature-based and relation-based methods. ReviewKD [2] , a feature-based method, exploits the residual structure to selectively refine the outputs of multiple intermediate layers. In comparison, prominent relation-based approaches excel by combining the relational matrix from multiple samples with contrastive learning in unsupervised domain. Particularly, SSKD [34] , a relation-based method, transfers knowledge by fitting the angle-wise relational matrix composed of positive and negative sample pairs. Despite these successes, we argue that existing contrastive learning based distillation methods have yet to realize their full potential for two primary reasons. First, these methods only use single-stage output features for relationship extraction, thereby overlooking the utility of multi-stage relational information between samples. Second, they rely only on angle-wise information, neglecting the informative distance component for relational representation.However, leveraging these missed opportunities presents certain challenges. On one hand, as shown in Fig. 1a , the mere incorporation of multi-stage relational information does not necessarily improve the distillation efficacy, even when the volume of transferred information increases. This suggests that raw multi-stage relational data may introduce redundant or even harmful information during the knowledge transfer from the teacher to the student model. On the other hand, Fig. 1b highlights the limitations of solely relying on angle-wise relationships. We calculated the length distribution (denotes the distance from the origin point) of the penultimate layer's output from the student model, in order to eliminate the influence of angle-wise information. Fig. 1b shows the length distribution of penultimate features from models trained by various KD methods, where a larger overlapping area with the teacher's distribution implies greater retention of distance information. This observation reveals that using only the angle-wise relationship between samples for knowledge distillation leads to evident information loss of the length distribution. Moreover, as demonstrated by RKD [22] , directly fitting both angle-wise and distance metrics between samples results in complex, interdependent matrices, and thereby culminates in sub-optimal performance.To address these constraints, we (a) The impact of different stage on Top-1 accuracy of SSKD. MixUp is used in the experiment, and all means using all stages' information. Figure 1 : Experimental results on relational information of samples in CIFAR100.the most suitable stages for each sample based on the relational representation capability of both its angle-wise and distance-wise relational information. In addition, MDR decouples inter-sample relationships into angle-wise and length-wise dimensions, allowing for a simultaneous and conflict-free transfer of both types of information. Moreover, in order to prevent Self-supervised Module (SM) from neglecting length information during feature normalization in contrastive-learning-based methods, we present a novel training methodology for SMs that replaces the training based on contrastive learning with an auxiliary classifier. This modification preserves length attributes while enhancing angle-wise representation capability. Our evaluation demonstrates that MDR framework surpasses the state of the art (SOTA) by an average of 1.22% on CIFAR-100 across extensively utilized network pairs. In summary, this paper makes the following contributions:• We present critical insights into the constraints of the existing contrastive learning based knowledge distillation frameworks. We delineate the avenues to further improve the methods through the optimized selection of multi-stage information and the strategic decoupling of angle-wise and length-wise relational representation.  
related work: 
 Knowledge distillation trains a smaller network using the knowledge from a larger network. Based on the types of the knowledge, existing KD frameworks can be divided into three categories: responsebased, feature-based and relation-based methods [6] .Response-based KD, also known as the classic KD [12] , usually relies on the neural response of the last output layer of the teacher model. The main idea is to directly mimic the final prediction (logits) of the teacher model. DKD [42] proposes a decoupled approach using the fundamental concept of KD. Unlike our proposed decoupled approach, DKD only separates the output categories into target and non-target classes, and assigns different importance to them. HSAKD [35] trains separate classifiers for each stage while transferring multi-stage response-based information.Featured-based KD, represented by FitNet [27] , encourages the student models to mimic the intermediate-level features from the hidden layers of teacher models. VID [1] and PKT [23] reformulate knowledge distillation as a procedure of maximizing the mutual information between the teacher and the student networks. There are also other methods using multi-stage information to transfer knowledge. OFD [11] uses a novel distance function to transfer knowledge from teacher to student; ReviewKD [2] proposes a new multi-stage architecture that allows the student to select the appropriate teacher stage for distillation. In contrast, our method employs an adaptive stage selection strategy to extract the most relevant relational information applicable to distillation for different samples.Relation-based KD emphasizes the exploitation of relationships between distinct layers or samples. FSP [37] guides the student model by generating a relation matrix between different layers of the teacher model. SP [32] , CC [24] , and RKD [22] utilize the relationships between samples to guide the student in learning higher-dimensional representations. Leveraging the success of contrastive learning in unsupervised tasks [22, 24] , many methods utilize the representation space of contrastive learning to model the relationships between samples. CRD [31] pioneered the integration of contrastive learning into knowledge distillation, SSKD [34] separately trains the teacher's SM to extract richer knowledge, PACKD [38] uses an optimal transport-based positive pair similarity weighting strategy to better transfer discriminative information from teachers to students. However, all of the existing contrastivelearning-based methods extract relationships at the penultimate feature layer. According to our experiments, we found that effective relational information can also be extracted from intermediate layers. Therefore, we propose a multi-stage distillation framework with adaptive stage selection strategy to comprehensively extract relational knowledge between samples. Moreover, our novel method decouples the relationship between samples into angle and length difference, compensating information loss in length-wise relationships in the existing contrastive-learning-based methods. 
methodology: 
 In this section, we first provide a brief review of KD and the details of contrastive-learning-based knowledge distillation methods. In light of the aforementioned problems and limitations of the existing methods, we present our proposed framework featuring an adaptive stage selection strategy, followed by the concept of relationship decoupling.The response-based methods transfer the dark knowledge from the teacher by approximating the distribution of soft targets, which can be formulated as:EQUATIONwhere 𝒛 𝒔 and 𝒛 𝒕 are the logits from the student and the teacher respectively; 𝜎 (•) is the softmax function that produces the category probabilities from the logits, and 𝜏 is a temperature hyperparameter to scale the smoothness of the distribution; 𝐾𝐿 means Kullback-Leibler divergence, which is the measurement of dissimilarity between two categorical distributions.The main idea of the feature-based KD methods is to mimic the feature representations between student and teacher, which can be formulated as the following loss function:EQUATIONwhere for stage 𝑘, 𝐹 𝑠 𝑘 and 𝐹 𝑡 𝑘 denote the feature maps from the student and the teacher respectively; T 𝑠 , T 𝑡 denote the student and the teacher transformation module respectively; L 𝑓 (•) denotes the function which compute the distance between two feature maps. Using multi-stage information has become the prevailing approach for feature-based methods [2, 35] .In contrast to the methods that distill knowledge from individual samples, the relation-based KD methods exploit the relationship between distinct samples, which can be formulated as:EQUATIONwhere (𝑡 𝑖 , 𝑡 𝑗 ) ∈ 𝐹 𝑡 and (𝑠 𝑖 , 𝑠 𝑗 ) ∈ 𝐹 𝑠 , 𝐹 𝑡 and 𝐹 𝑠 are the sets of feature representations of samples from the teacher and student respectively; 𝜓 (•) denotes the similarity function of (𝑡 𝑖 , 𝑡 𝑗 ) or (𝑠 𝑖 , 𝑠 𝑗 );L 𝑅 2 (•)is the correlation function of the feature representations between teacher and student (e.g., Huber loss). However, the existing relation-based methods focus on the design of the relational matrix and neglect the valuable multi-stage information.As the predominant relation-based method, contrastive-learningbased knowledge distillation captures inter-sample relationships to transfer knowledge by leveraging the cosine similarity within the representation space. Given a mini-batch containing 𝑁 samples {𝑥 𝑖 } 𝑖=1:𝑁 (i.e., anchor set P), we apply strong data augmentation 𝑡 (•), such as Random Rotation [34] or MixUp [38] , to each sample and obtain { 𝑥 𝑖 } 𝑖=1:𝑀 ((i.e., positive set P) where 𝑀 = 3𝑁 . Both 𝑥 𝑖 and 𝑥 𝑖 are fed into the teacher or student networks to extract their representations 𝜙 𝑖 = 𝑓 (𝑥 𝑖 ), 𝜙 𝑖 = 𝑓 ( 𝑥 𝑖 ). The similarities between 𝑥 𝑖 and 𝑥 𝑖 can be represented by the following matrix A:EQUATIONwhere 𝑧 𝑖 and 𝑧 𝑗 are the outputs of SM, which transforms 𝜙 𝑖 and 𝜙 𝑖 into a contrastive learning representation space. A 𝑖,𝑗 represents the similarity between 𝑥 𝑖 and 𝑥 𝑗 . ( 𝑥 𝑖 , 𝑥 𝑖 ) refers to the positive pair and ( 𝑥 𝑖 , 𝑥 𝑗 ) 𝑖≠𝑗 the negative pair. The SM consists of a 2-layer perceptron with a pooling layer, which is trained by maximizing the similarity between positive pairs. A commonly used contrastive objective is defined as:EQUATIONIn addition to the angle-wise relational matrix formation, the distance-wise relational matrix between samples can also be used to transfer knowledge [22] , which can be expressed as:EQUATIONHowever, utilizing both the angle-wise and distance-wise matrices simultaneously leads to a degraded performance due to their strongly coupled relationship.As mentioned in Sec.1, the existing contrastive-learning-based knowledge distillation methods only use single-stage output features for relationship extraction. However, as shown in Fig. 1a , the output of each stage contains valuable angle-wise relational information for the student to learn. To better exploit these information, we adopt a multi-stage framework to transfer knowledge. Specifically, for each distillation stage, both teacher and student networks are equipped with SMs to capture relational information. We augment the data set using MixUp and derive the anchor set P and the positive set P. To fully exploit the representation capability, we incorporate both the angle-wise and distance-wise information instead of solely relying on angle-wise relationships in the loss function. Therefore, unlike Eqn. 3, the loss of multi-stage knowledge transfer is represented as:EQUATIONwhere for 𝑘-th stage, B 𝑠 is a probability matrix, consisting of student's similarity matrix A 𝑠 (Eqn. α ! α " α # 𝑙 ! 𝑙 # [α ! , α " , α # ] [𝑙 ! , 𝑙 " , 𝑙 # ] Relational Decoupled Module (RDM) … … … … Anchor set Positive set 𝒫 " 𝒫 stage1 Teacher backbone 𝑓 $ • stage2 stage3 stage1 Student backbone 𝑓 % • stage2 stage3 ℒ !" ℒ #$% 𝑆𝑀 ! $ 𝑆𝑀 " $ 𝑆𝑀 # $ 𝑆𝑀 ! RDM ℒ &'( ℒ $)'Classifier 𝑐 $ (•) P in the mini-batch. The same procedure is applied to the teacher to obtain B 𝑡 . As shown in Fig. 1a , incorporating multi-stage relational information straightforwardly does not necessarily improve the distillation efficacy. We argue that every stage contains beneficial information, but using all stages introduce redundant or even harmful information during the knowledge transfer. In order to obtain just enough relational information effectively, we propose an adaptive stage selection strategy to select the most appropriate stage for angle-wise knowledge transfer.Classifier 𝑐 % (•) selection transfer ADSS ADSS ADSS ADSS ADSS ADSS ADSS 𝑙 " 𝑙 " α # α " α ! 𝑙 ! 𝑙 #Since both the intra-class (positive pairs) and inter-class (negative pairs) correlation can reflect the representational capability, it is insufficient to use only the cosine similarity of positive pairs as a metric. Moreover, the representation capabilities of each stage are different, and it is not suitable to directly compare the absolute values of similarity between positive and negative pairs. Therefore, we use relative numerical ranking instead of absolute cosine similarity. Specifically, we use Eqn. 4 to calculate the cosine similarity between each positive anchor pair and rank them individually at each stage of the mini-batch extending the dimensions of the positive sample. By comparing the order of the similarity between a positive sample and its corresponding anchor in each stage, we select the highest-ranking stage for knowledge transfer.As illustrated in Fig. 2 , we exploit distance-wise relationship, in order to convey more comprehensive information during distillation. As for the stage selection, we use absolute distance as the criterion and also exploit relative numerical ranking by using the following formula:EQUATIONwhere 𝐾 is the number of stages in a network; 𝑀 𝑘 𝑖 is angle-wise or distance-wise similarity matrix between positive sample 𝑖 and all anchor samples in the mini-batch; 𝑅𝑎𝑛𝑘 is the function that sorts the similarity in descending order.As shown in Fig. 1b , exclusively depending on angle-wise relationship during distillation leads to information loss of the length distribution. Therefore, it is crucial to utilize both angle-wise and distance-wise information in order to comprehensively capture the relational information between samples for distillation. Yet, it is challenging to effectively combine these two types of information. For example, RKD directly used two types of information, but the best distillation results are often obtained by taking one of the two. This is because the distance metric contains both angle and length (the latter indicating distance from the origin) information, which may obstruct the comprehension of angle information while learning distance information. In the feature representation space, the sample distance does not align with the principles of contrastive learning as described in Eqn. 5. Specifically, the distance between the positive pairs does not always need to be close. Therefore, conflicts arise when fitting angle-wise and distance-wise relationships simultaneously.To solve this problem, we propose the concept of relationship decoupling. As illustrated in Fig. 2 , we decouple the relationship between samples into angle and length difference, and the latter can be expressed by the following equation: where 𝜇 is a normalization factor for length difference. Similar to RKD, we set 𝜇 to be the average length difference between pairs from P and P in the mini-batch:EQUATIONEQUATION)Unlike the traditional way of transferring angle-wise knowledge, we directly use MSE loss to fit the length-wise relational matrix. The length-wise loss and angle-wise loss are defined respectively as: ).L 𝑙𝑒𝑛 = ∑︁ 𝑖 ∈ P,(EQUATIONThe final loss for the student network is the combination of aforementioned terms, including the original training loss L 𝑐𝑙𝑠 , the response-based loss L 𝑘𝑑 , and the relation-based loss L 𝑎𝑛𝑔 and L 𝑙𝑒𝑛 :EQUATIONwhere the 𝜆 𝑖 is the balancing weight.Before training the student, we freeze the teacher's backbone and train the SM. SM is typically trained by explicitly improving the representational ability of contrastive learning (Eqn. 5) in existing methods, which neglect length-wise information during feature normalization to prioritize angle-wise relationships. To preserve length-wise information while maintaining the representational ability of contrastive learning, we place a classifier behind each SM and directly use cross-entropy (CE) loss. This approach ensures that the dimensions of the outputs are consistent while retaining the relational information. Compared with the prior training methods, the contrastive learning representational ability of SM is further amplified with CE loss. Moreover, training through a classifier is more effective for SM to obtain the global information of the data set, rather than the relational information between samples within a mini-batch. 
experiments: 
 To demonstrate the effectiveness of our work, we evaluate MDR in various tasks: classification, few-shot learning, transfer learning and object detection. Moreover, we present various ablation study for the proposed method.Datasets and Competitors We conduct evaluations on standard CIFAR-100 [15] and ImageNet [28] benchmarks across the widely applied network families including ResNet [9] , WRN [40] , VGG [30] , MobileNet [29] , ShuffleNet [21] . CIFAR-100 [15] contains 50K images for training and 10K images for testing, labeled into 100 finegrained categories. The size of each image is 32×32. ImageNet [28] consists of 1.2M images for training and 50K images for validation, covering 1,000 categories. All images are resized to 224 × 224 during training and testing. We report the top-1 and top-5 accuracy on this dataset for image recognition.Moreover, we employ the SIL-10 [4] and TinyImagenet [28] datasets to assess the transferability of learned representations generated by distillation method. STL-10 [4] is composed of 5K labeled training images and 8K test images in 10 classes. TinyImageNet [28] is composed of 100K training images and 10k test images in 200 classes. We evaluate the proposed MDR on this dataset with image recognition and report the top-1 accuracy.Following the consistent protocol, we use Pascal VOC [5] train-val07 + 12 for training and test07 for evaluation. The result set consists of 16551 training images and 4952 test images in 20 classes. The image scale is 1000 × 600 pixels during training and inference.We compare MDR with a wide range of representative KD methods, including KD [12] , FitNets [27] , AT [39] , SP [32] , CC [24] , RKD [22] , PKT [23] , OFD [11] , CRD [31] , SSKD [34] , CRCD [43] , ReviewKD [2] , DKD [42] , CTKD [18] , ML-LD [14] .Implementation details We attach one SM after each convolutional stage. The SM is composed of global average pooling(GAP) and two fully-connected(FC) layer. For training teacher SMs, we attach one FC layer for CE loss, where the input dimension is same as the dimension of SM's output feature (e.g., 128 on CIFAR-100, 1280 on ImageNet) and the output dimension is same as the number of categories. During the training stage of teacher's SMs, we connect a classifier after each SM (composed of a layer of fully connected), and directly uses category information for supervised learning. In this process, except for SM and classifier, the backbone part of the network remains frozen.On CIFAR-100, the batch size and initial learning rate are set to 64 and 0.05. We train the models for 240 epochs in total with SGD optimizer, and decay the learning rate by 0.1 at 150, 180, and 210 epochs. The weight decay and the momentum are set to 5e-4 and 0.9. On ImageNet, we adopt the SGD optimizer to train the student networks for 100 epochs with a batch size of 512. The initial learning rate is 0.2 and decayed by 10 when the epoch is 30, 60 and 90. Weight decay and momentum are the same as above. We set 𝜏 in L 𝑘𝑑 for P to be 1, P to be 1, 𝜏 in L 𝑎𝑛𝑔 and L 𝑙𝑒𝑛 to be 0.5. We set 𝜆 1 = 1.0, 𝜆 2 = 2.0, 𝜆 3 = 300, 𝜆 4 = 1.0 in Eqn. 13.Due to the page limit, we provide more training details in the supplementary materials.Results on CIFAR-100. We compare our MDR with representative distillation methods using a variety of teacher-student pairs, with both identical and different architectural styles on the CIFAR-100 dataset. As shown in Table 1 , our MDR consistently outperforms other methods by a significant margin. Specifically, our method achieves an average of 0.88% accuracy improvement when compared with the existing optimal method for each network pair configuration. The amount of accuracy improvement is larger than that of many previous methods. Collectively, there's an average improvement of 1.22% in accuracy over the best-performing SSKD (more experiments in the supplementary materials). These results indicate that our proposed MDR effectively exploits the decoupled relationship across multiple stages between samples for knowledge distillation. Note that the student's accuracy surpasses the teacher's in certain identical architecture pairs, such as WRN40-2→WRN40-1. This underscores our method's capability to comprehensively extract more valuable information from teachers. Results on ImageNet. We further evaluated a teacher-student pair on the large-scale ImageNet and its downstream task, using ResNet34 as a teacher and ResNet18 as a student. As shown in Table 2 , our MDR delivers the best accuracy in both Top-1 and Top-5 categories. Specifically, MDR improves the accuracy by 0.62% over SSKD for Top-1 accuracy. The accuracy improvement on ImageNet is less pronounced than on CIFAR100. This can be attributed to higher class similarity within categories on ImageNet, which is challenging for the SM training method. Despite this, MDR still achieves one of the highest incremental gains over existing methods, reducing the student-teacher accuracy gap to 1.28% (about 20% relative improvement) compared to 1.61% for the previous best. These results highlight MDR's remarkable effectiveness on largescale datasets even in the presence of challenging class structures.Transferability of Learned Representations. Beyond achieving superior accuracy on the object dataset, it is imperative for the student network to produce generalized feature representations that can exhibit robust transferability to novel semantic recognition datasets. To this end, we adopt the strategy of freezing the backbone 𝑓 𝑆 (•) that has been pre-trained on the upstream CIFAR-100. We then train two linear classifiers based on the fixed penultimate features for downstream classification on the STL-10 and Tiny-ImageNet, respectively [31] . Table 3 shows the ability of transfer learning using different KD methods. Specifically, our MDR method outperforms the best-competing DKD by an accuracy gain of 1.41% on STL-10 and an accuracy gain of 1.14% on TinyImageNet, demonstrating its superior transferability to various recognition tasks.Efficiency under Few-shot Scenario. We evaluate our method against conventional KD, CRD, SSKD and CTKD in a few-shot learning environment, using retention rates of 25%, 50%, and 75% of the original training samples. To ensure a fair comparison, we maintain a consistent data split strategy for each few-shot scenario, while keeping the original test set intact. Our evaluation utilizes the Accuracy (%)ℒ !"# ℒ !"# + ℒ $%&' ℒ !"# + ℒ ()" + ℒ $%&' ℒ !"# + ℒ ()"(b) Effect of RDM. ResNet56-ResNet20 pair. As depicted in Table 4 , our method consistently outperforms the other techniques by large margins across various few-shot scenarios. Notably, compared with the baseline trained on the complete set, our method achieves higher accuracy with only 25% of the training data. This outcome is attributed to our method's ability to effectively learn general relational information from limited data. In comparison, the previous methods typically focus on mimicking inductive biases from intermediate feature maps or incomplete relationships, which may overfit on the limited dataset and reduce generalization on the test set.Transferability for Object Detection. We further evaluate the student network ResNet-18, which is pre-trained with the teacher ResNet-34 on ImageNet, as a backbone for downstream object detection on Pascal VOC. For this evaluation, we adopt the Faster-RCNN [26] framework, adhering to the standard data pre-processing and fine-tuning protocols. Table 5 shows our method's superior detection performance, surpassing the original baseline by 2.24% mAP and the best-competing DKD method by 0.61% mAP. These results underscore our method's efficacy in guiding a network to achieve superior feature representations for diverse semantic tasks.In this section, we provide ablation studies to analyze the effects of each component of MDR. The experiments are conducted on CIFAR-100 for classification task.Effect of Adaptive Stage Selection. As shown in Fig. 3a , MS means using multi-stage decoupled relational information to transfer, which contains angle-wise and length-wise information in each stage. Applying angle-wise adaptive stage selection strategy (MS + ADSS_A) substantially boosts the accuracy upon the original multistage information, indicating that we extract a larger amount of beneficial angle-wise relationship. As we further add distance-wise adaptive stage selection strategy (MS + ADSS_A + ADSS_D), an even higher accuracy is achieved thanks to the positive contribution of valuable distance-wise information.Effect of Relational Decoupled Module. To explore the effectiveness of proposed RDM, we conduct the evaluation in three variants: only using angle-wise information (L 𝑎𝑛𝑔 ), angle-wise and distance-wise information (L 𝑎𝑛𝑔 + L 𝑑𝑖𝑠𝑡 ), both angle-wise and length-wise information (L 𝑎𝑛𝑔 + L 𝑙𝑒𝑛 ) and all three information (L 𝑎𝑛𝑔 + L 𝑙𝑒𝑛 + L 𝑑𝑖𝑠𝑡 ). The results are shown in Fig. 3b , where coupled information (L 𝑑𝑖𝑠𝑡 ) often have a negative impact on distillation, and RDM boosts the accuracy compared to the others. To assess the impact of CE loss for SM training, we compare the teacher's SMP accuracy and student's accuracy under three cases: no SM, with contrastive loss, and with CE loss. SMP accuracy shows the faction of positive samples correctly assigned to the corresponding anchor. As shown in Fig. 3c , compared with the no-SM case, training with contrastive loss improves SMP accuracy, which is further improved by CE loss in both types of accuracy. Among them, the improvement of SMP accuracy in the early stage is more obvious with the modification of SM training strategy.The number of adaptive stages. We validate various number of adaptive stages based on angle and distance respectively: 1/2/3 with two teacher-student pairs, including identical and distinct architectures. For the fairness of the comparison, when verifying the number of adaptive stages based on angle, length-wise one is fixed to be 1, and vice versa. As shown in Table 6 , regardless of angle-wise or length-wise information, the best result is achieved when the number of adaptive stages is 1. Combined with stage selection, the accuracy improves steadily.Due to the page limitation, more ablation studies and experiment analysis can be found in the supplementary materials.In this part, we present the visualization to show that our MDR does bridge the teacher-student gap in the relation-level. The experiments are conducted on the sampled CIFAR-100 validation set (10,000 samples). We compute relational matrix with a batch size of 25 for the penultimate stage, so these are 400 values for each experiment. We visualize the MSE loss of relational matrix between ResNet32×4 and ResNet8×4 in Fig. 4 , which is formed by the addition of normalized angle-wise and length-wise matrix. For better presentation, we rank these values and organize them as the heatmap representation. The smaller the value, the more similar the matrix are. We can find that our MDR significantly improves the similarity of angle-wise and length-wise relational matrix between the student and the teacher.Due to the page limitation, more related visualizations can be found in the supplementary materials.Compared with other relational distillation methods that only use the penultimate layer information for distillation, our method needs to use the middle layer information, so the teacher and the student need to have the same number of stages. Therefore, there is a constraint on the selection of distillable networks.In addition, similar to other knowledge distillation methods based on contrastive learning, our method needs to first train the teacher's SM module and obtain the relationship matrix of angle and length, so it takes a longer time than the traditional KD method (under the same hardware conditions , the training time is 5.2 times that of traditional KD and 1.1 times that of SSKD). 
conclusion: 
 In this paper, we propose a novel framework equipped with an adaptive stage selection strategy for relation-based knowledge distillation, which enables efficient extraction of relational information across multiple stages. By decoupling the relationship into angle and length difference and introducing a novel training method for the self-supervised module, our approach enables the student to acquire knowledge more effectively. Experiment results show that our method significantly surpasses SOTA performance on the standard image classification benchmarks in the field of KD. It also opens the door for further improvements of knowledge transfer methods based on relationship.  
 
 Appendix: & 𝑆𝑀 # & 


Decision: This paper proposes a Multi-stage Decoupled Relational (MDR) KD framework with adaptive stage selection, which allows for effective multi-stage information extraction to enhance relational knowledge transfer. The MDR framework separates angle-wise and distance-wise information to avoid conflicts, ensuring the preservation of complete relational knowledge and improving both transfer efficiency and distillation quality. Extensive experiments across various benchmarks and tasks validate the effectiveness of the method. Although the applicability of MDR to network architectures with different stages remains unverified (pointed out by Reviewer 4Wr4), most of the concerns are well-addressed during the rebuttal and most reviewers acknowledge the novelty and the effectiveness of the proposed method. Therefore, the AC recommends to accept this submission.

Reviews: review: paper_summary: paper_summary: This paper propose a Multi-stage Decoupled Relational (MDR) KD framework equipped with an adaptive stage selection which enables multi-stageinformation extraction to maximize the efficacy of transferring the relational knowledge. MDR framework decouples angle-wise and distance-wise information to resolve their conflicts while still preserving complete relational knowledge by resulting in an elevated transferring efficiency and distillation quality.This paper also formulate a new SM trainingparadigm to compensate for the loss of length-wise information and augment its contrastive learning representation capability.
strength_weakness: strength_weakness: 1. The sentence structure of the paper is fluent and there are no obvious grammar or spelling errors.
2. The paper has conducted rich comparative experiments and ablation experiments, as well as supplementary materials for ablation experiments and experimental analysis, which provide strong support for the experimental results.
3. The paper provides a novel multi-stage distillation framework with adaptive stage selection strategy, which extracts effective relationship information from the intermediate layer and is innovative.
The method proposed in the paper has good performance in tasks such as classification, few shot learning, transfer learning, and object detection, and has good generalization ability.
limitations: limitations: 1. The process in Figure 2 needs to be explained more clearly, for example, what standards are used for information filtering in areas with red crosses in the figure? I hope it can be reflected in the picture.
2. In the experimental setup of section 4.1, there was no detailed introduction to the data augmentation methods used in the experiment. Only line 299 mentioned the use of methods such as Random Rotation and MixUp. Please provide a specific list of the data augmentation methods used in the experiment.
3. The author validated and compared the performance of the method across multiple datasets, but the comparison on ImageNet is relatively weak, which should ideally be the most crucial comparison. It is recommended that the author strengthens this aspect of the content to increase the reliability of the paper.
4. This study appears to be quite interesting and shows some degree of innovation. It is suggested that the author better elucidate the motivation behind this paper to enhance its readability, aid in understanding the author's work, and assist readers in evaluating the performance of this study.
paper_summary: paper_summary: This paper introduces a novel KD framework that enhances distillation by using multi-stage relational information. It decouples angle-wise and distance-wise information to improve the transfer of relational knowledge and resolves conflicts between these attributes. The framework features an Adaptive Stage Selection (ADSS) strategy to identify the most effective stages for KD. Extensive experiments on CIFAR-100, ImageNet, and Pascal VOC demonstrate MDR's superior performance, surpassing state-of-the-art methods with an average improvement of 1.22% on CIFAR-100.
strength_weakness: strength_weakness: 1. This paper proposed an adaptive stage selection strategy to decouple the relational information.  This innovative perspective addresses previously unrecognized conflicts in relational information for knowledge distillation.
2. The structure of this paper is logical and coherent, beginning with an introduction to the problem, followed by a detailed explanation of the proposed methodology, and concluding with experimental results and discussions.
limitations: limitations: 1. Some sections explaining complex concepts like angle-wise and length-wise relational information are difficult to understand and would benefit from more diagrams and simplified explanations.
2. The method requires the teacher and student networks to have the same number of stages, limiting its applicability to certain network architectures.
3. It is recommended that the illustrations in the introduction should not display the method results, but rather showcase novel findings that explain the proposed problems and phenomena.
paper_summary: paper_summary: The article discusses a novel framework for relation-based knowledge distillation (KD) that incorporates an adaptive stage selection strategy to efficiently extract relational information across multiple stages. This method decouples the relationship between samples into angle and length differences, enhancing the transfer of relational knowledge from teacher to student models. The proposed approach significantly improves performance on standard image classification benchmarks and addresses limitations in existing contrastive-learning-based methods by extracting effective relational information from intermediate layers.
strength_weakness: strength_weakness: The article introduces a novel framework for relation-based knowledge distillation that incorporates an adaptive stage selection strategy and relationship decoupling into angle-wise and length-wise components. This approach enhances the efficiency of relational information extraction across multiple stages, leading to more effective knowledge transfer. The proposed method significantly outperforms state-of-the-art techniques in image classification benchmarks, demonstrating superior accuracy and potential for further improvements in knowledge distillation methods.
limitations: limitations: 1.Limited Comparison: The paper lacks a comprehensive comparison with a wider range of existing knowledge distillation methods, potentially restricting the assessment of MDR's performance against a broader set of benchmarks.
2.Scalability Concerns: While MDR demonstrates effectiveness in certain tasks, the scalability of the method to larger and more complex models or datasets has not been thoroughly explored, leading to questions about its applicability in diverse scenarios.
3.Generalization: The discussion on the generalizability of MDR across different domains or applications is limited, creating uncertainty regarding its effectiveness in varied contexts. 
4.Theoretical Depth: The paper could benefit from more in-depth theoretical analysis or comparisons with existing theoretical frameworks in knowledge distillation to enhance the academic rigor of the proposed MDR approach.
paper_summary: paper_summary: This paper introduces a multi-stage knowledge distillation method based on contrastive learning.  The proposed method contains two main folds, using the intermediate feature for feature distillation, and decoupling feature relationships to length and angle distances. Experiments demonstrate the efficacy of the proposed method.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow.
2. The proposed method is easy to apply and straightforward.
3. The authors conducted abundant experiments not only on classification but detection as well.
limitations: limitations: 1. Did the authors reimplement the SOTA methods? They look different from the original papers.
2. Why do the authors use top-K to choose the layer rather than bottom-K or other methods? What is the rationale behind the design?
3. In Fig.3 (b), the result of L_len is missing.