Beyond Direct Relationships: Exploring Multi-Order Label Pair Dependencies for Knowledge Distillation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 09:45:55

Abstract: Multi-label image classification is crucial for a wide range of multimedia applications. To address the resource limitation issue, various knowledge distillation (KD) methods have been developed to transfer knowledge from a large network (referred to as the "teacher") to a small network (referred to as the "student"). However, existing KD methods do not explicitly distill the dependencies between labels, which limits the model ability to capture multi-label correlation. Furthermore, although existing methods for multi-label image classification have utilized the second-order label pair dependency (direct dependency between two labels), the high-order label pair dependency, which captures the indirect dependency between two labels, remains unexplored. In this paper,  we propose a  \textbf{\underline{M}}ulti-Order Label Pair \textbf{\underline{D}}ependencies \textbf{\underline{K}}nowledge \textbf{\underline{D}}istillation (MDKD) framework. MDKD explicitly distills the knowledge to capture multi-order dependencies between labels, including the label pair dependencies from second-order and high-order, thus transferring the insight of label correlations from different perspectives. Extensive experiments on  Pascal VOC2007, MSCOCO2014, and NUS-WIDE demonstrate the superior performances of MDKD.

Main paper: 
 introduction: 
 Natural scenes in the real world often cover multiple objects. For example, as shown in Figure 1 , an image from a street scene may include annotations for labels such as person, dog, car, and parking meter. Multi-label image classification aims to recognize each of these labels concurrently, which is crucial for a wide range of multimedia applications, including recommendation systems [27] , image retrieval [23] , and scene understanding [21] . To capture complex label features, there is always a necessity to train large models to obtain desirable performance in multi-label image classification.Figure 1 : An illustration of fourth-order label pair dependency formed by parking meter-car-person-dog, and the corresponding attention maps by different methods: 1) Vanilla: the backbone trained without distillation; 2) L2D: a logitsbased method that achieves the state-of-the-art performance in KD for multi-label image classification; 3) Teacher: the pretrained backbone used for distillation; 4) Ours: the MDKD framework that explicitly transfers the knowledge to recognize multi-order label pair dependencies during distillation.Despite the remarkable achievements in multi-label image classification facilitated by deep neural networks, deploying these substantial models on resource-constrained devices, such as mobile phones, poses a significant challenge due to limited computational capacity and the necessity for swift inference times. To address this issue, knowledge distillation (KD) [10] have emerged as a powerful strategy, aiming to enhance the performance of small networks (referred to as "students") through the knowledge from large networks (known as "teachers"). Recently, KD methods have shown promising results to boost the capabilities of multi-label image classification [13, 22, 25, 26] . However, a common shortfall among these methods is their lack of explicit focus on modeling the multi-label dependencies, such as second-order label pair dependencies, which restricts their ability to fully understand the intricate relationships among multi-label.The second-order label pair dependency, which considers the direct label co-occurrence, has played a crucial role in reasoning the occurrence of multi-label. For instance, MLGCN [3] proposed modeling second-order label pair dependencies, achieved through a learnable static correlation matrix. Additionally, Ye et al. [28] introduced a dynamic graph model to handle occasional label correlations in different images, while CADM [3] proposed incorporating the class-aware graph to explore label correlations. However, despite these advancements in second-order label pair dependencies, the exploration into the high-order label pair dependencies remains unexplored. High-order label pair dependencies extend beyond direct co-occurrences to consider indirect relations. For example, in a street scene scenario like Figure 1 , the association between car and person and the co-occurrence of person and dog imply an indirect dependency between car and dog. This scenario reflects a thirdorder label pair dependency, considering the indirect dependencies with person as the mediate label. Moreover, given that a parking meter is often associated with a car in urban street scenes, the above third-order label pair dependency could be further extended to a fourth-order one which implies the indirect relations between parking meter and dog, composing of the mediate second-order label pair dependencies of parking meter-car, car-person, and person-dog. Additionally, it is worth noting that the label sequences with the same endpoint labels but different mediate labels give rise to multiple high-order label pair dependencies. That is, a fourth-order label pair dependency formed by label sequence parking metercar-person-dog does not consider the same thing as that in parking meter-person-car-dog, as the former considers the mediate secondorder label pair dependencies of parking meter-car, car-person, and person-dog, while the latter considers the mediate second-order label pair dependencies of parking meter-person, person-car, and car-dog.In this paper, we propose a Multi-Order Label Pair Dependencies Knowledge Distillation (MDKD) framework for multi-label image classification, which explicitly distills the knowledge to discern label pair dependencies at multiple orders, including the second-order and high-order ones. Specifically, MDKD measures the multi-order label pair dependencies directly from the label predicted probabilities, which requires no extra architecture and training strategy within the general framework of multi-label image classification. Then, the measured label dependencies from teacher and student are aligned to encourage the knowledge transfer of multi-order label pair dependencies. As shown in Figure 1 , by learning to capture the multi-order label pair dependencies, our MDKD correctly recognizes the parking meter, car, person, and dog, matching the attention from the teacher, while the vanilla and L2D fail to achieve this.The main contributions are three-fold as follows:• We first explore the indirect label pair dependencies, denoted by high-order label pair dependencies, in the field of multi-label image classification.• We propose an MDKD framework for multi-label image classification that transfers knowledge to identify label pair dependencies at multiple orders, including direct (second-order) and indirect (high-order) ones.• Extensive experiments on Pascal VOC2007 [8] , MSCOCO2014 [12] , and NUS-WIDE [4] demonstrate the capability of our method. 
related work 2.1 multi-label dependency: 
 Multi-label image classification has recently yielded significant attention due to its importance in multimedia applications. Existing methods for multi-label image classification focus on modeling the label correlations, which provides prior knowledge for multi-label classification: MLGCN [3] proposed modeling the label dependencies by a learnable static correlation matrix. Considering the occasional label co-occurrence, Ye et al. [28] proposed a dynamic graph model, capable of capturing the occasional label correlations at various images. Moreover, Zhu et al. [33] proposed refining the dynamic label co-occurrence by specific scenes. Beyond secondorder label dependencies (between two labels), Wu et al. proposed an adaptive hypergraph for modeling these intricate many-to-one high-order interactions. However, these methods still can not explicitly model the indirect label dependencies. In this paper, we harness the multi-order label indirect correlation into KD. It is also worth noting that while the existing method [24] proposed utilizing the high-order dependency in multi-label learning, it simply considers how a single label will be impacted by a group of labels. In our MDKD, the combination of multi-order label pair dependencies provides a more granular perspective about how a single label impacts another via different orders of dependencies (direct and indirect) under a multi-label context.KD [10] aims to transfer knowledge from a large teacher network to a simpler and smaller student network. Existing KD methods can be primarily divided into two categories: logits-based methods and feature-based methods. Logits-based methods concentrate on designing distillation loss to transfer knowledge in the logits: For instance, the mutual learning strategy [32] encourages concurrent training of the teacher and student networks, enhancing the learning process through a cooperative framework. On the other hand, TAKD [16] proposed the "teacher assistant"-an intermediate-sized network that promotes the knowledge transfer between the more complex teacher and the simpler student networks. Feature-based methods focus on distilling knowledge from intermediate feature layers: For instance, FitNet [1] proposed minimizing the similarity between the feature maps of teacher and student. Attention transfer [30] focuses on attention maps rather than direct feature map comparisons. ReviewKD [2] proposed using multiple layers in teachers to supervise one layer in students.While KD has been demonstrated to be a powerful strategy for transferring knowledge to students, the direct transplantation of typical KD to multi-label image classification presents a significant challenge. Specifically, logits-based methods often obtain the predicted probabilities based on the softmax function. However, softmax function is not suitable for multi-label, because the sum of predicted probabilities may not equal one in multi-label image classification. Feature-based methods proposed aligning the feature maps between teacher and student. This approach, while effective in harnessing contextual information, inadvertently biases the model towards the predominant objects. Therefore, it leads to the marginalization of less prominent objects within the scene.Previous efforts have explored performing KD in multi-label image classification: Xu et al. [25] proposed a complementary parallel self-distillation to learn the joint patterns and the category-specific patterns of labels. Song et al. [22] proposed estimating the uncertainty of prediction to handle difficult labels. Liu et al. [13] proposed to enhance the label prediction by a weakly-supervised detection. Yang et al. [26] proposed using the feature compactness in labelwise embeddings of the same classes, as well as the dispersion across different classes. Moreover, Yang et al. proposed a Multi-Label Logits Distillation (MLD) loss, which divides multi-label tasks into several binary classification problems. This stems from the understanding that the probabilistic of labels do not sum to one in a multi-label image. However, existing methods ignore explicitly modeling the distillation of label dependencies. To address this problem, we explore the distillation of label dependencies in this paper. 
methodology: 
 Given a data sample {(x, 𝑦)}, we have x ∈ X with its corresponding label vector 𝑦 ∈ Y, where X ⊂ R 𝑑 is the 𝑑-dimensional input space and Y ⊂ {0, 1} 𝐶 denotes the target space with 𝐶 classes. For each instance x, the 𝑗-th component 𝑦 𝑗 of 𝑦 is used to indicate whether 𝑦 𝑗 is relevant to x or not. Specifically, 𝑦 𝑗 is relevant to x when 𝑦 𝑗 = 1, and conversely 𝑦 𝑗 is irrelevant to x if 𝑦 𝑗 = 0. Figure 2 illustrates the MDKD framework. Given a batch of input instances (images), and the label indices {1, 2, • • • , 𝐶}, the backbone network 𝑓 outputs a batch of predicted probabilities from teacher model {( ŷ𝑖1 , ŷ𝑖2 , . . . , ŷ𝑖𝐶 )} 𝑏 𝑖=1 , where 𝐶 represents the total number of labels and 𝑏 denotes the batch size. ŷ𝑖𝑘 denotes the predicted probability for 𝑘-th label within 𝑖-th instance. Note that the notation used in this paper takes superscripts T and S to denote the teacher and the student models. For example, ŷ T 𝑖𝑘 and ŷS 𝑖𝑘 represent two label predicted probabilities computed from the teacher network 𝑓 T and the student network 𝑓 S , respectively.Following previous KD methods [10, 16, 26] , MDKD integrates supervision loss between student output and the ground truth, and the soft label distillation loss, which encourages the knowledge transfer from teacher to student. For a batch of samples {(x 𝑖 , 𝑦 𝑖 )} 𝑏 𝑖=1 , the ground truth loss is implemented by Binary Cross Entropy (BCE) loss, encouraging the label predicted probabilities of the student network to be near to the ground truth:EQUATIONThe soft label loss is implemented by multi-label logits distillation loss L MLD , which considers the fact that in multi-label learning, the label predicted probabilities of all classes should sum to one. It minimizes the divergence between the binary predicted probabilities of teacher and student, which is computed as:EQUATIONwhere[•, •] is the concatenation operation. D (•||•) represents the KL divergence D (𝑃 ||𝑄) = 𝑥 ∈ X 𝑃 (𝒙) log 𝑃 (𝒙 )𝑄 (𝒙 ) . However, with the BCE loss and multi-label logit distillation loss, the label dependencies still can not be explicitly distilled to the student. MDKD aims at distilling the multi-order label pair dependencies discerned by the teacher from input instances. Specifically, MDKD includes a batch-level multi-order label pair dependency distillation loss L BD and an instance-level multi-order label pair dependency distillation loss L ID . The motivation behind incorporating these two levels of dependency distillation comes from the observation that some labels exhibit consistent co-occurrence across all scenarios, such as tennis racket and ball, whereas others predominantly co-occur in specific contexts, for instance, person and car in street scenes. While the batch-level multi-order label , (1) (1, 2)EQUATION, (1) (-1, ),() (-1, ), (1) (, ),() (, ) pair dependency distillation aims to encourage knowledge transfer of common multi-order label dependencies within a batch, the instance-level multi-order label pair dependency distillation transfers the knowledge in recognition of occasional multi-order label dependencies within an instance. These two levels of distillation loss collaboratively promote the knowledge transfer of label pair dependencies.Overall, the total objective function of the proposed MDKD framework can be represented as:EQUATIONwhere 𝜆 MLD , 𝜆 BD , and 𝜆 ID are trade-off hyperparameters.Figure 3 (left) illustrates the process of batch-level multi-order label pair dependency distillation. Given the sets { ŷ1𝑘 , ŷ2𝑘 , . . . , ŷ𝑏𝑘 } and { ŷ1𝑙 , ŷ2𝑙 , . . . , ŷ𝑏𝑙 }, where each represents the label predicted probabilities by the teacher model for 𝑘-th label and 𝑙-th label across various instances within a batch, respectively. The batchlevel second-order label pair dependencies are measured through the mean square difference between two sets:EQUATIONwhere ŷ𝑖𝑘 and ŷ𝑖𝑙 denote the label predicted probabilities for 𝑘-th label and 𝑙-th label, respectively, for the 𝑖-th instance in a batch. Based on this measurement, the frequently co-occurring label pair exhibits similar label predicted probabilities across various instances, resulting in a lower mean square difference. We employ the Huber loss to measure the second-order label pair dependencies disparity between teacher and student, denoted as ℧ Ψ T (2) (𝑘, 𝑙), Ψ S (2) (𝑘, 𝑙) , where ℧(𝑎, 𝑏) is defined as follows:EQUATIONIt's worth noting that the dependencies Ψ (2) (𝑘, 𝑙) and Ψ (2) (𝑙, 𝑘) are considered the same for a lightweight complexity. This stems from the understanding that the label dependency is mutual, where the order slightly impacts the recognition of their dependency. Consequently, it can be easily proved that the total number of label pairs within second-order label pair dependency equals the combinations (i.e., half of the permutations). Suppose P b a = a! (a-b)! as the permutations, the batch-level second-order label pair dependency loss, L BD 2 , which integrates the disparity of common second-order label pair dependencies within a batch between teacher and student, is defined as:EQUATIONwhere [𝐶] represents the sequence of label indices {1, 2, • • • , 𝐶}.denotes the reciprocal of the total number of second-order label pair dependencies, serving as a normalization for loss.For the batch-level 𝑛-order label dependencies (𝑛 ≥ 3), each batch-level 𝑛-order label pair dependency is measured by averaging the batch-level second-order label pair dependencies involving all intermediate label pairs. For example, given a fourth-order label sequence parking meter-car-person-dog, the batch-level fourth-order label pair dependency between parking meter and dog, articulated through this sequence, is derived by sequentially averaging the batch-level second-order label pair dependencies of parking metercar, car-person, and person-dog. This stems from the understanding that a sequence of 𝑛-order label pair dependency is achieved through a series of mediate second-order label pair dependencies, creating a chain of dependencies that bridge the two endpoint labels. Given this, to present the formula for a sequence of batch-level 𝑛-order label pair dependency between the 𝐿 1 -th and 𝐿 𝑛 -th labels, we define the label sequence as {𝐿 1 , 𝐿 2 , . . . , 𝐿 𝑛 }. Here, each element 𝐿 𝑘 signifies a label index among the complete set of label indices {1, 2, . . . , 𝐶}. The batch-level 𝑛-th order label pair dependency, denoted by Ψ (𝑛) (𝐿 1 , 𝐿 2 , . . . , 𝐿 𝑛 ), is measured by:EQUATIONThe batch-level 𝑛-order label pair dependency distillation considers the disparity of label pair dependencies captured by teacher and student for all unique label sequences, where the Huber loss ℧ Ψ T (𝑛) (𝐿 1 , 𝐿 2 , . . . , 𝐿 𝑛 ), Ψ S (𝑛) (𝐿 1 , 𝐿 2 , . . . , 𝐿 𝑛 ) (as defined in Eq. 5) is employed to measure the disparity between teacher and student. Given 𝐶 labels, it is easy to prove that the total number of label pairs within the 𝑛-order is half of the permutation, i.e., P 𝑛 𝐶 2 . The batch-level 𝑛-order label pair dependency distillation loss, denoted by L BD 𝑛 , is defined as: P 𝑛 𝐶 2 denotes the total number of 𝑛-order label pair dependencies, and its reciprocal serves as a normalization for loss. It is notable that when 𝑛 = 2, this equation is with the same formulation as Eq. 6.EQUATIONIn the objective function of MDKD (Eq. 3), the part of batchlevel multi-order label pair dependency loss is obtained by the weighted sum of global label dependency distillation loss from different orders:EQUATIONwhere the series of 𝜆 BD 𝑖 denotes the trade-off parameters.Different from the batch-level multi-order label pair dependency distillation, which transfers the knowledge to recognize multi-order label pair dependencies within a batch, the instance-level multiorder label pair dependency distillation loss considers the occasional label pair dependencies within different images. Figure 3 (right) illustrates the process of instance-level multi-order label pair dependency distillation. Given a batch of output probabilities from a teacher model, denoted as {( ŷ𝑖1 , ŷ𝑖2 , . . . , ŷ𝑖𝐶 )} 𝑏 𝑖=1 . The instancelevel second-order label pair dependency between 𝑘-th label and 𝑙-th label within 𝑖-th instance is measured by the square difference of their label predicted probabilities:EQUATION)This approach stems from the understanding that while the occasional label pair dependency is high, the square difference of label predicted probabilities within each instance is low.The instance-level label dependency loss aggregates the difference of instance-level second-order label pair dependencies between teacher and student across the entire batch 𝑏, where the disparity of recognized instance-level second-order label pair dependencies are also measured by Huber loss (Eq. 5):EQUATION)For the instance-level 𝑛-order label dependencies (𝑛 ≥ 3), each instance-level 𝑛-th order label pair dependency is measured by averaging the instance-level second-order label pair dependencies involving all intermediate label pairs. Consider a label sequence {𝐿 1 , 𝐿 2 , . . . , 𝐿 𝑛 }, which forms an 𝑛-order label pair dependency between the 𝐿 1 -th and 𝐿 𝑛 -th labels. For the 𝑖-th instance, we define the instance-level 𝑛-order label dependency, denoted as Φ (𝑖 ) (𝑛) (𝑘, 𝑙), in the following formula:EQUATION)The instance-level 𝑛-order label dependency distillation loss encourages the teacher model and student model to be consistent for the instance-level 𝑛-order label dependencies, formula as:EQUATIONwhere𝑏 𝑖=1 P 𝑛 𝐶 2𝑗=1 ensures that all possible label sequences within each instance in a batch are considered in the loss function to measure and minimize the difference in dependency predictions between the teacher model and the student model. P 𝑛 𝐶 2 is the total number of 𝑛-order label pair dependencies, and its reciprocal serves as the normalization for loss. It is notable that when 𝑛 = 2, this equation is with the same formulation as Eq. 11. In the objective function of MDKD (Eq. 3), the part of instance-level multi-order label pair dependency distillation loss is obtained by the weighted sum of instance-level label dependency distillation loss from different orders:EQUATION)where the series of 𝜆 ID 𝑖 represents the trade-off parameters.Given the detailed formulas in Eq. 9 and Eq. 14, the total objective function of MDKD in Eq. 3 can be re-written as:EQUATIONwhere the series of 𝜆 ID 𝑖 and 𝜆 BD 𝑖 are renewed trade-off hyperparameters for each specific label pair dependency distillation loss.  
experiment 4.1 datasets: 
 We perform experiments on MS-COCO2014 (MS-COCO for short) [12] , Pascal VOC 2007 (VOC for short) [8] , and NUS-WIDE [4] , to validate the effectiveness of our proposed method. VOC includes To evaluate our proposed method, we perform a comparison study with state-of-the-art KD methods, including the logits-based (RKD [17] , MSE [25] , PS [22] , and L2D [26] ) and feature-based (PKT [18] and ReviewKD [2] ) methods. Table 1 and Table 2 demonstrate the comparison results on MS-COCO, evaluating the effectiveness of teacher-student model pairings with both the same and different architectures. Table 1 reveals that the MDKD method consistently outperforms others across three metrics. For instance, a notable improvement is observed when ResNet56 serves as the teacher and ResNet20 as the student, with MDKD achieving mAP improvements of 3.89% over RKD and 4.12% over the base student model. Moreover, MDKD frequently surpasses even the performance of teachers. This superior performance is evident across various backbone combinations, demonstrating the superior ability of MDKD. Additionally, Table 2 reveals that the disparity in performance between teacher and student models with differing architectures is larger than that when architectures are the same, yet MDKD still significantly outperforms other methods and occasionally exceeds the teacher model. Furthermore, Table 3 shows the performance of different methods on VOC in terms of AP and mAP. It is shown that MDKD achieves the highest scores in most cases than other KD methods, including AP and mAP. Notably, for challenging classes such as horse, where other methods may underperform compared to the vanilla (without performing distillation), MDKD demonstrates a remarkable 1.21% AP improvement over vanilla. Table 6 (Appendix) presents comparison results on NUS-WIDE. It is shown that MDKD also outperforms other state-of-the-art methods in all cases, which demonstrates its superior ability. These experimental results highlight that while the state-of-the-art methods only transfer the knowledge within feature maps and logits, they achieve sub-optimal efficacy due to the lack of knowledge to capture label dependencies, while our method leverages label dependencies to enhance the KD in multi-label image classification tasks and achieves high performances.To further evaluate the ability of multi-order label pair dependency distillation, we conduct ablation studies on MS-COCO, employing ResNet-101 as the teacher and ResNet-34 as the student. The results are shown in Table 4 , where the first row presents the baseline performance of the student model without any distillation applied. The results in rows 2, 3, 5, and 9 demonstrate a step-wise enhancement with the implementation of second-order, third-order, and fourthorder label pair dependency distillation, before the implementation of fifth-order label pair dependency distillation. However, the distillation of fifth-order label pair dependencies and sixth-order label pair dependencies (as seen in the last two rows) does not significantly enhance the ability of the model. We argue that this phenomenon occurs because the correlation between label pairs becomes increasingly marginal when extended to overly high orders.This section performs a sensitivity analysis on the parameters 𝜆 𝐵𝐷 2 , 𝜆 𝐵𝐷 3 , 𝜆 𝐵𝐷 4 , 𝜆 𝐼 𝐷 2 , 𝜆 𝐼 𝐷 3 , and 𝜆 𝐼 𝐷 4 to further evaluate their impact on model effectiveness. As depicted in Figure 4 , an increase in these parameters from 0 to 1 promotes an improvement in model performance. This observation reveals that the distillation of multiorder label pair dependencies can improve the model performance. Additionally, beyond the value of 1, the performance of the model is insensitive to their values. To evaluate whether the student focuses on objects correctly corresponding to the teacher, we visualize the attention maps from different classes using Grad-CAM [20] . Our MDKD method demonstrates superior precision in recognizing objects compared to other methods, as illustrated in Figure 5 and Figure 7 (Appendix). For instance, in Figure 5 , MDKD accurately captures the cup, while other methods exhibit recognition failures. The clear recognition of different classes reveals the superior ability of MDKD compared with other methods.To evaluate whether the knowledge to recognize multi-order label pair dependencies is better transferred in MDKD, we compare the mean square and square differences in output probabilities from different methods. The teacher is ResNet-101 and the student is ResNet-34. We demonstrate the second-order label pair dependencies in Figure 6 and Figure 8 (Appendix). The third-order label pair dependencies and fourth-order label pair dependencies are presented in Table 7 (Appendix) and Table 8 (Appendix). The results show that the label pair dependencies captured by MDKD better match the ones from the teacher compared to other methods, which reveals its capability to transfer knowledge in recognition of multi-order label pair dependencies.Previous study [29] has shown that even a high-performing model can be further improved through guidance from a less complex model. To delve deeper into the efficacy of MDKD within the context of reverse KD, we experiment by assigning ResNet-34 as the teacher and ResNet-101 as the student, resulting in the student surpassing the performance of the teacher. As shown in Table 5 , MDKD still achieves higher scores than all other methods, underscoring its exceptional capability. 
conclusion: 
 In this paper, we propose the MDKD framework, which explicitly transfers the knowledge to capture multi-order label pair dependencies during KD. The multi-order distillation transfers the insight to capture label correlations from different perspectives. Specifically, MDKD includes batch-level multi-order label pair dependency distillation, which considers the multi-order label pair dependencies within a batch, and instance-level multi-order label pair dependency distillation, which addresses occasional label multi-order label pair dependencies based on specific scenes. Extensive experiments on different datasets reveal the superior ability of MDKD. 
 
 Appendix:  


Decision: - The paper proposes the MDKD framework for the task of multi-label image classification by explicitly distilling both second-order (direct) and high-order (indirect) label pair dependencies from a teacher model to a student model and, therefore, enhancing the student's ability to recognize complex label correlations.

- We have two sufficiently positive reviews and two slightly negative reviews.

- In my opinion, the idea is fine and interesting. It is suggested that the authors carefully revise this paper based on each reviewer's comment in the final version.

Reviews: review: paper_summary: paper_summary: This paper introduces the Multi-Order Label Pair Dependencies Knowledge Distillation (MDKD) framework, an innovative approach for multi-label image classification. It aims to address the limitations of existing knowledge distillation methods by explicitly distilling dependencies between labels, including both second-order and high-order dependencies. Therefore, it manages to capture more nuanced label correlations. The paper validates its approach through extensive experiments on datasets such as Pascal VOC2007, MSCOCO2014, and NUS-WIDE, demonstrating superior performances of MDKD.
strength_weakness: strength_weakness: The strength of this paper lies within its focus on an important, yet often-overlooked aspect of multi-label knowledge distillation. While the field of knowledge distillation is well-explored in the context of single-label classification, the area remains significantly under-researched for multi-label settings. The authors have made a commendable effort by exploring and making advancements in this challenging aspect.
limitations: limitations: 1. While the experiments are overall sufficient, I find the performance gap between the teacher and student models to be too minimal. This leads me to doubt whether the baseline for the teacher model is set solid enough. It is my strong suggestion to increase the input size from 224 to 448. My recommendation is based on the observation that the COCO dataset tends to yield less stable results with smaller input sizes. I also suggest adding the results from the study 'Knowledge Distillation from Single to Multi Labels: An Empirical Study' to your paper. It provides solid baselines for multi-label KD with a simple method.
2. From Figure 2, the CAM visualization for the L2D method appears to be inferior to that of the vanilla method, which seems counter-intuitive. How would the authors interpret this? Would it be possible to include more visualization figures in the paper for further clarity?
3. How does this approach deal with the issue of missing labels, especially considering that missing labels are a common occurrence in the field of multi-label classification?
4. How would this method perform in scenarios with a larger number of labels? The paper currently only validates the approach with 80 labels; how can the authors ensure the method's generalizability beyond this?
5. I would suggest that, to improve readability and comprehension, the authors could provide an illustrative example and pseudo code of the core method, akin to what was offered in the "RKD" paper. This enhancement can significantly aid readers in understanding the method better.

I am open to further improving my score if the authors can adequately address the above-mentioned concerns and questions.
paper_summary: paper_summary: The manuscript presents a novel framework for Multi-Order Label Pair Dependencies Knowledge Distillation (MDKD) aimed at improving multi-label image classification by explicitly distilling knowledge, on both second-order and high-order label dependencies. This method is tested on several well-known datasets, showing promising improvements over existing methods.
strength_weakness: strength_weakness: 1. The proposed MDKD framework introduces an innovative way of considering not only direct label pair dependencies but also higher-order indirect relationships.

2. The authors provide extensive experimental results to validate the effectiveness of the proposed method.

3. The paper is well-written and organized.
limitations: limitations: 1. How does the method mine the multi-order co-occurrence? Sequentially averaging the
batch-level second-order label pair dependencies could do this? The authors should give more explanations for that.

2. What's the benefit of the use of Huber loss? No further discussion can be found in this paper.

3. Is it an innovative contribution of this paper for using the Huber loss to measure the second-order label pair dependencies? If not, why not cite relevant literature?

4. The comparison methods seem to be outdated, whether it is possible to add more recent methods?

5. What is the computation complexity and the train/test time of the proposed method?

6. Is the teacher net pre-trained on multi-label datasets, please provide the details. How to get the final prediction results, by the student or fusion of S and T?
paper_summary: paper_summary: The paper proposes a Multi-Order Label Pair Dependencies Knowledge Distillation (MDKD) framework for multi-label image classification.The key idea is to explicitly distill both second-order (direct) and high-order (indirect) label pair dependencies from a teacher model to a student model, enhancing the student's ability to recognize complex label correlations.

To distill label correlations, the paper proposes Batch-level Multi-Order Label Pair Dependency Distillation and Instance-level Multi-Order Label Pair Dependency Distillation to transfer knowledge from both batch and instance, as well as with the consideration of multi-order dependencies (derived from second-order dependencies).

Finally, the method's effectiveness is validated on the Pascal VOC2007, MSCOCO2014, and NUS-WIDE.
strength_weakness: strength_weakness: The paper is well-structured and easily understandable, introducing two loss functions that can capture high-order label correlations from both batch-level and instance-level perspectives.

The experimental results seem impressive, supported by an ablation study that illustrates the efficacy of the two introduced loss functions.
limitations: limitations: Compared to L2D, the novelty seems lacking, as it mainly involves replacing the label-wise distillation and instance-wise distillation loss with batch-level and instance-level dependency loss. More explanation is needed on the advantages of these two proposed loss functions over L2D's Label-wise Embedding Distillation.

The superiority of MDKD in capturing high-order label dependencies is demonstrated with only a single image (Figure 1), which is less convincing. More visualization examples are needed to strengthen this point.

There is a lack of explanation on the calculation method for multi-label probabilities and the activation function. In lines 325-326, the paper states that "the predicted probabilities of all classes should sum to one," suggesting that the activation function is likely to be **softmax**, which differs from L2D. However, $\mathcal{L}_{\mathrm{MLD}}$ is directly referenced from L2D. It is necessary to clarify the activation function and explain the differences in probability calculation compared to L2D.
The phenomenon where the student's performance surpasses the teacher's after distillation (as seen in Table 1) is not reasonably explained. The reviewer would like to see the explanation for such experimental results.

In Figure 3(b), it appears that only the first instance is distilled, but not the b-th instance. In fact, the teacher should distill the b-th instance as well, but the consistency is not depicted, which may lead to misunderstanding.
paper_summary: paper_summary: This paper tackles the multi-label classfication problem by exploreing multi-order label pair dependencies for knowledge distillation.
strength_weakness: strength_weakness: - The paper is clearly written and easy to follow.
limitations: limitations: - There is a general issue in the abstract. The authors claim that the high order label pair dependency remains unexplored. In many survey papers of multi-label leanring, they often divide existing methods into three categories according to the exploration of label correlations: first-order, second-order and high order. Thus, such description in abstact is too arbitrary.

- You should explain why your proposed method is superior to other high-order based method.

- In Figure 2, it is easy to understand the instance-level multi-order label pair, how to understand batch-level? Compared with instance-level, what are the superiorities of batch-level?