Multi-Label Knowledge Distillation
Authors: Peng-Hui Yang
Ming-Kun Xie
Chen-Chen Zong
Lei Feng
Gang Niu
Masashi Sugiyama
Sheng-Jun Huang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Existing knowledge distillation methods typically work by enforcing the consistency of output logits or intermediate feature maps between the teacher network and student network. Unfortunately, these methods can hardly be extended to the multi-label learning scenario. Because each instance is associated with multiple semantic labels, neither the prediction logits nor the feature maps obtained from the whole example can accurately transfer knowledge for each label. In this paper, we propose a novel multi-label knowledge distillation method. On one hand, it exploits the informative semantic knowledge from the logits by label decoupling with the one-versus-all reduction strategy; on the other hand, it enhances the distinctiveness of the learned feature representations by leveraging the structural information of label-wise embeddings. Experimental results on multiple benchmark datasets validate that the proposed method can avoid knowledge counteraction among labels, and achieve superior performance against diverse comparing methods.

Under review as a conference paper at ICLR 2023
MULTI-LABEL KNOWLEDGE DISTILLATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Existing knowledge distillation methods typically work by enforcing the consis- tency of output logits or intermediate feature maps between the teacher network and student network. Unfortunately, these methods can hardly be extended to the multi-label learning scenario. Because each instance is associated with multiple semantic labels, neither the prediction logits nor the feature maps obtained from the whole example can accurately transfer knowledge for each label. In this pa- per, we propose a novel multi-label knowledge distillation method. On one hand, it exploits the informative semantic knowledge from the logits by label decou- pling with the one- mee-veruus-al reduction strategy; on the other hand, it enhances the distinctiveness of the learned feature representations by leveraging the structural information of label-wise embeddings. Experimental results on multiple bench- mark datasets validate that the proposed method can avoid knowledge counterac- tion among labels, and achieve superior performance against diverse comparing methods.
INTRODUCTION
Despite the remarkable success in training deep neural networks (DNNs) (Krizheskyy et al. 2012). is hard to deploy these large neural networks on lightweight terminals, e.g., mobile phones, under the constraint of computational resource or requirement of short inference time. To mitigate this issue, knowledge distillation (Hinton et al., 2015) aims to improve the performance of a small network (also known as the student) by requiring the knowledge from a large network (also known as the teacher) to guide the training of the student network. Existing knowledge distillation methods can be roughly divided into two categories: logits-based methods and feature-based methods. The former minimizes the difference between logits of teacher model and student model (Hinton et al. 2015 Zhao et al. 2022). while the latter distills knowledge from feature maps of intermediate layers (Park)
ct al.
2019 Tian et al. 2019: Chen et al., 2021).
Typical knowledge distillation methods focus on the multi-class classification task, where each in- stance is associated with only one class label. However, in many real-world scenarios, an instance in- herently contains complex semantics and can be simultaneously assigned with multiple class labels. For example, an image of street scene may be annotated with labels building, car and person. To learn the complex object-label mapping, there is always necessity of training large models to obtain desirable performance in multi-label classification. Unfortunately. due to computational resource constraints, it cannot be allowed adopt large neural networks in many practical applications, lead- ing to noticeable decrease in model performance (Gou et al., 2021). To alleviate the performance degradation, it is necessary design specific knowledge distillation methods for multi-label learn- ing. We formalize such problem as a new learning framework called Multi-Label Knowledge Distillation (MLKD).
Although knowledge distillation has been proven to be effective for improving the performance of the student network in single-label classification, is still a challenging problem to directly extend existing KD methods to solve MLKD problems. Specifically, logits-based methods often obtain the predicted probabilities based on the softmax function, which is unavailable for MLKD, since the sum of predicted probabilities may not equal to one in Multi-Label Learning (MLL). Feature- based methods often perform knowledge distillation based on feature maps of the whole image with multiple semantics, which makes the model focus the major objects while neglect the minor objects. This would lead the model to obtain sub-optimal even undesirable distillation performance. Figure provides empirical validations for these observations.

---

Under review as a conference paper at ICLR 2023
82
73.5
73.0
mAp
80
71.5
Vanilla
Softmax
ReviewKD
L2D
Vanilla
Softmax
ReviewKD
L2D
(a) Models with the same architecture
(h) Models with different architectures
Figure 1: Comparison results between our proposed L2D method and conventional KD methods on MS-COCO. We compare our method with the following three baselines: 1) Vanilla: student trained without distillation; 2) Softmax (Hinton et al. 2015): representative logits- logits-based method by measuring KI. divergence on softmax score after logits: 3) ReviewKD (Chen al. 202T): feature-based method that achieves sota performance. The red dashed lines mark the performance of teachers. Conventional KD methods achieve unfavorable performance similar the student, while method significantly outperforms these methods and achieves comparable performance to teacher. Details about the implementation of this experiment can found Appendix A
In this paper, to perform multi-label knowledge distillation, propose new method consisting of multi-label logits distillation and label-wise embedding distillation (L2D for short). Specifically, to exploit informative semantic knowledge compressed in the logits, L.D employs the versussulllll reduction strategy to obtain a series of binary classification problems and perform logits distillation for each one. To enhance the distinctiveness of learned feature representations, L2D encourages the student model to maintain a consistent structure of intra-class and intra-instance (inter-class) label-wise embeddings with the teacher model. By leveraging the structural information of the teacher model, these two structural consistencies respectively enhance the compactness of intra- class embeddings and dispersion of inter-class embeddings for the student model.
Our main contributions can be summarized as follows:
A general learning framework called MLKD is proposed. To best knowledge, the framework is the first study specially designed for knowledge distillation the multi-label learning scenario.
A new approach called L.2D is proposed. It performs multi-label logits distillation and label-wise embedding distillation simultaneously. The former provides informative seman- tic knowledge while the latter encourages the student model learn more distinctive fea- ture representations.
Extensive experimental results on benchmark datasets demonstrate the effectiveness of our proposed method.
RELATED WORK
The concept of knowledge distillation (KD) proposed by Hinton et al. (2015) defines a learning framework that transfers knowledge from large teacher network small student network. Ex- isting works can be roughly divided into two groups, i.e., logits-based methods and feature-based methods. Logits-based methods mainly focus on designing effective distillation losses distill knowledge from logits and softmax scores after logits. DML (Zhang et al.) 2018) introduces a mutual learning method to train both teachers and students simultaneously. TAKD (Mirzadeh al.) 2020) proposes new architecture called "teacher assistant", which is an intermediate- sized network bridging the gap between teachers and students. Besides, a recent study (Zhao et al. 2022) proposes novel logits-based method to reformulate the classical KD loss into two parts and achieves the state-of-ther to-th- performance by adjusting weights for these two parts. Some other methods focus on distilling knowledge from intermediate feature layers. FitNet (Romero al. 2014) is the first approach to distill knowledge from intermediate features by measuring the distance between feature maps. Attention transfer (Zagoruyko & Komodakis 2016a) achieves better performance than FitNet
2

---

Under review as a conference paper at ICLR 2023
by distilling knowledge from the attention maps. PKT PPssslli & Tefas 2018 measures the KL divergence between features by treating them as probability distributions. RKD Park et al. 2019) utilizes the relations among instances guide the training process of the student model. CRD (Tian et al. 2019) incorporates contrastive learning into knowledge distillation. ReviewKD (Chen et al.) 2021 proposes review mechanism which uses multiple layers in the teacher to supervise one layer in the student. ITRD (Miles et al., (02T)) aims to maximize the correlation and mutual information between the students' and teachers' representations.
Multi-label learning has increasingly attracted lot of interest recently. Existing solutions for solv- ing MLL problems can be categorized into three directions. The first type attempts to design novel loss functions for tackling the intrinsic positive-negative imbalance issue in multi-label classifica- tion tasks. For example, ASI. (Ridnik et al., 2021a) uses different weights to re-weight positive and negative examples for the balanced training. The second type focuses modeling the label correla- tions, which provides prior knowledge for multi-label classification. Among them, MI.GCN (Chen) et al. 2019b) is representative method that employs graph convolutional network to model cor- relation matrix. CADM (Chen et al, 2019a) constructs a similar graph based on class-aware maps. To handle the multiple semantic objects contained in an image, the last type of methods aims to lo- cate areas of interest related to semantic labels by using attention techniques. Among them, C-aran (Lanchantin et al. 2 first utilizes the transformer to retrieve embeddings from visual features for each label. Query21 Labell (Liu et al. (021a) uses several stacked transformer encoders to identify in- teresting areas. ML-Decoder (Ridnik et al. 20211) simplifies transformer encoders in Query2Label. ADDS (Xu et al., (022b) introduces encoders from CLIP (Radford et al. 2021 in order to get better textual and visual embedding inputs the classification head. In addition, ADDS adds a multi-head cross-attention layer and a skipping connection from the query input to the query output based on ML-Decoder.
Several previous studies have applied KD techniques to improve the performance of MLL. For example, Liu et al. (2018) and Xu et al. (2022a) simply minimized mean squared error (MSE) loss between teacher logits and student logits. Song et al. (202T) designed a partial softmax function by combining a positive label with all other negative labels. Then, the conventional KD loss can be computed for each positive label. The main difference between that we aim to improve the distillation performance instead of the performance of MLL. This can be reflected in experiments in Section and Appendix [B] where the proposed method mainly compares with KD methods while the previous method mainly compares with MLL methods.
work and previous works is
THE PROPOSED APPROACH
Let E X' be an instance and y E y be its corresponding label vector, where X C Rd is the input space with d dimensions and y C {0, 13 is the target space with q class labels. We further use yj to denote the j-th component of y. For given instance æ. y/ = 1 indicates the j-th label is relevant to the instance; yj = 0, otherwise. In multi-label learning, each instance may be assigned with more than one label, which means Σ I(y, 1) 1, where I[() is the indicator function. We also denote by [q] the integer set (1,2.-.-.
In this paper, we use a classification model consisting of three components, i.e., a visual backbone f. which extracts a feature map_
[(x) for the input æ. a label-wise embedding encoder g (Lanchantin et al. 2021 Liu et al. 2021a). which produces label-wise embedding e 9% (æx) with re- spect to the k-th class based on the feature map fxx) and a multi-labed classifier h. which predicts (hg((eg)) where ar.) denotes the sig- moid function. is noteworthy that the used model is very general, which can be built by equipping commonly used backbones, e.g., ResNet (He et al. 20166. with label-wise embedding encoder g. For the notations mentioned above, we use the superscripts T (or S) to denote the teacher (or student) model. For example, we use ej and e to denote the label-wise embeddings for teacher and student models.
multi-label probabilities ÿ = (a(ha (ei)), a (h2(e))),
In multi-label learning, popular method is to employ the one one-versus-ale all reduction strategy to trans- form the original task into multiple binary problems. Among the various loss functions, the most commonly used one is the binary cross entropy (BCF) loss. Specifically, given batch of examples
3

---

Under review as a conference paper at ICLR 2023
FT
g²
h
Label-wise embedding distillation
Multi-label logits distilattonn
ff
g'S
h
RCE loss
A batch of multi-label examples
Label-wise embeddings
Predicted probabilities
Figure 2: An illustration of the L2D framework. The framework simultaneously performs multi- label logits distillation and label-wise embedding distillation to improve the performance of the student model.
{(ri y-) and the predicted probabilities ÿ. the BCE loss can be defined follows:
E LACR - Σ 1=1 k=1
-' you log(ūa))
yas) log(1 )ss))
Figure illustrates the distillation process of the proposed L2D framework. For batch of training examples, we feed them into the teacher/student model to obtain the label-wise embeddings and predicted probabilities.
order to train the student model, besides the BCE loss, we design the following two distillation losses: 1) multi-label logits distillation loss LMLD exploit informative semantic knowledge compressed in the logits, 2) label-wise embedding distillation loss CLED to leverage structural information for enhancing the distinctiveness of learned feature representations. The overall objective function can be presented as
LLD = LBCE + AM + ÂLEDCLED.
where AMLD and ALED are two balancing parameters.
3.1 MULTI-LABEL LOGITS DISTILLATION
Traditional logits-based based distillation normally minimizes the Kullback-Leibler (KL) divergence be- tween the predicted probabilities, i.e., the logits after the softmax function, of teacher and student model. However, the method cannot be directly applied to the MLL scenario, since it depends on one, which hardly
basic assumption that the predicted probabilities of all classes should holds for MLL examples.
To mitigate this issue, inspired by the idea of one- velsus-all reduction, propose multi-label logits distillation (MLD) loss, which decomposes the original multi-label task into multiple binary classification problems and minimizes the divergence between the binary predicted probabilities of two models. Formally, the MLD loss can be formulated as follows:
CMLD = 1 Σ Σ
(LUTe Iills 0.rl))
(3)
where [ is operator used concatenate two scalars into vector, and D is divergence func- P(x)
tion. The most common choice is the KL divergence DKL (kl(()(Q) = Σ P(x) log
where
[(z)
P and Q are two different probability distributions. The MLD loss aims to improve the performance of student model by sufficiently exploiting informative knowledge from logits.
3.2 LABEL- EMBEDDING DISTILLATION
The MLD loss performs distillation on the predicted probabilities that can regarded as a high- level representation, i.e., the final outputs of model. The knowledge distilled from the teacher model
4

---

Under review as a conference paper at ICLR 2023
Instance-aware label-wise distillation
Class-ware wise
Figure 3: illustration of class/instance-un aware label-wise embedding distillation. Class-aware label-wise embedding distillation (CD) captures structural relations among intra-class label-wise embeddings from different examples, while instance-aware label-wise embedding distillation (ID) explores structural relations among intra-instance (inter-class) label-wise embeddings.
by only using the ML.D loss would insufficient to train a student model with desirable perfor- mance due to the limited information carried by the logits. To further strengthen the effectiveness of distillation, we design the label-wise embedding distillation (LED) loss, which aims to explore the structural knowledge from label-wise embeddings. The main idea is to capture two types of structural relations among label-wise embeddings: 1) class-aware label-wise embedding distillation (CD) loss LLED-CD. which captures the structural relation between any two intra-class label-wiss embeddings from different examples; 2) instance-aware label-wise embedding distillation (ID) loss LLEDAD. which models the structural relation between any inter-class label-wise embeddings from the same example. In the following content, we introduce these two distillation losses in detail.
3.2.1 CLASS-AWARE LABEL-WISE WISE EMBEDDING DISTILLATION
Class-aware lahel-wise embedding distillation aims improve the distillation performance by ploiting the structural relations among intra-class label-wise embeddings. Generally, the same se- mantic objects from different images often differ from each other by their individual character- istics, such as two cars with different colors and diverse styles (see the left side in Figure 3). Since our goal to distinguish between car and other semantic classes instead of identifying different cars, these distinctiveness would be confusing information for the corresponding classification task. Due the powerful learning capacity, the large model able capture the highly abstract semantic representations for each class label by neglecting the useless individual information. From the per- spective of learned feature representations, as shown in the left side of Figure 3 the teacher model tends to obtain more compact structure of intra-class label-wise embedding, which often leads to better classification performance. By transferring the structural knowledge from the teacher model to the student model, CD encourages the student model to enhance the compactness of intra-class label-wise embeddings, which can improve its classification performance.
For batch of examples, let {e fi=1 and { 1i respectively denote the intra-class label-wise embeddings with respect to class E lg] generated by the teacher and student models. Then, we capture the structural relation between any intra-class label-wise embeddings eT and e

---

Under review as a conference paper at ICLR 2023
by measuring their distance in the embedding space:
{ 1 fall2 Vik = 1,k {0 00
(4)
otherwise.
It is worth to note that we only consider the structural relation between any two valid label-wise embeddings, iee. the embeddings with respect to positive labels. Similar to Eq. for any two intra-class label-wise embeddings e,t and e we can obtain the structural relation (CD ejk) for the student model.
By enforeing the teacher and student structural relations to maintain the consistency for each pair of intra-class label-wise embeddings, we can achieve the class-aware structural consistency as follows:
LLED-CD Σ E Lej (( e,,,, ΦCD (e efk)),
(5)
where l is function measure the consistency between the teacher and student structural relations. In experiments, we use the following Huber loss function as a measurement:
1 - b)2 f(a, b) - -
|a - bj < 1, otherwise.
(6)
where a and b are two different structural relations.
3.2.2 INSTANCE-AWARE LABEL-WISS WISE EMBEDDING DISTILLATION
Instance-aware label- wise embedding distillation (ID) aims to improve the distillation performance by exploring the structural relations among inter-class label-wise embeddings from the same image. Generally, one can hardly distinguish between two different semantic objects occurring in an image due to the high similarities they share. For example, see the right side of Figure 3 for an image annotated with sky, sea and beach, one can hardly distinguish between the semantic objects sky and sea, since they share the same color and similar texture. A feasible solution is exploit other useful information, such as their spatial relation, Le., sky is always above sea. Due to the powerful learning capacity, the large model is able to distinguish between the similar semantic objects by exploiting such implicit supervised information. From the perspective of learned feature representations, as shown in the right side 3 the teacher model tends to learn a dispersed structure of inter-class label-wiss wise embedding, which is beneficial for improving its discrimination ability. By distilling the structural knowledge from the teacher model, ID enforces the student model to enhance the dispersion of inter-class label-wise embeddings, which can improve its discrimination ability.
For given instance F,, let {e and {e respectively denote the label-wise embeddings generated by teacher and student models. Then, we can capture the structural relation between any two inter-class label-wise embeddings e/7 and edi by measuring their distance in the embedding space:
( fle% - eelll Vik 1, War
T e Φ② (
otherwise.
Note that in 7 only consider the structural relation between any two valid label-wise embed- dings, Le., the embedding with respect to positive labels. Similar to Eq. for any two inter-class label-wise embeddings eit and eif we can obtain the structural relation @ID ck eg) for the student model.
By encouraging the teacher and student model maintain the consistent structure of intra-instance label-wise embeddings, we can minimize the following LLEDAM loss to achieve the instance-aware structural consistency where E(.) Huber loss as defined in 6).
LLED-I >' 1=1 E
(( eT)) ( e.)).
(8)
Finally, the overall objective function of L2D (Eq. (2) can be re-written follows: LLDD LACR. + AMIDLMED + AL + Aumn.nDumm..
(9)
where AMLD, ADED-CD and ALED in all balancing parameters.

---

Under review as a conference paper at ICLR 2023
Table 1: Results MS-COCO where teacher and student models are in the same architectures.
Teacher
RepVGG-A2
ResNet-101
WRN-101
Swin-S
Student
RepVGG-A0
ResNet-34
WRN-50
Swin-T
Metrics
mAP
OF1
mAP
OF1
mAP
CFI
mAP
CFI
Teacher
72.71
74.11
73.62
73.89
74.70
75.56
70.73
81.70
77.12
Student
70.02
72.49
66.77
70.31
72.49
66.82
74.45
75.43
70.61
79.59
79.18
75.42
RKD
70.08
72.39
66.73
70.13
72.44
66.78
74.70
75.71
70.84
79.63
79.19
75.57
PKT
70.11
72.47
66,80
70.43
72.64
66.68
74.54
75.47
70.58
79.64
79.09
75.39
RD
72.35
70.39
72.62
66.76
74.03
75.29
70.36
79.81
79.18
75.55
MSE
70.26
72.54
66,99
70.54
72.75
66.85
74.53
75.60
70.71
79.67
79.20
75.52
PS
70.65
72.89
67.60
70.86
72.66
67.12
75.12
76.05
71.63
79.96
79.64
76.20
MLD
70.74
72.81
67.46
70.68
72.69
67.19
74,92
75.75
71.21
80.11
79.68
76.44
L2D
72.81
74.59
69.49
72.87
74.45
69.43
76.61
77.08
72.79
81.59
81.03
77.86
Table 2: Results on MS-COCO where teacher and student models are in the different architectures.
Teacher
ResNet-101
Swin-T
ResNet-101
Swin-T
Student
RepVGG-A0
ResNet-34
MobileNet v2
MobileNet
Metrics
mAP
OFI
CFI
mAP
OF1
CFI
mAP
OF1
CFI
mAP
OF1
CFI
Teacher
73.62
73.89
68.61
79.43
78.77
75.07
73.62
73.89
68.61
79.43
78.77
75.07
Student
70.02
72.49
66.77
70.31
72.49
66.82
71.85
73.59
68.26
71.85
73.59
68.26
RKD
70.08
72.35
66.72
70.00
72.34
66.64
71.76
73.68
68.40
71.74
73.68
68.37
PKT
72.35
66.56
70.26
72.39
66.82
71.88
73.60
68.35
71.84
73.76
68.37
ReviewKD
70.00
72.33
66.62
70.29
72.39
66.58
71.92
73.73
68.48
71.73
73.71
68.36
MSE
70.07
72.50
66.85
70.33
72.57
66.72
71.91
73.68
68.28
71.80
73.74
68.38
PS
70.30
72.61
67.10
70.94
72.93
67.57
72.11
73.89
68.42
72.42
74.14
68.94
MILD3
70.48
72.77
71.14
72.99
72.17
68.52
72.35
74.10
68.91
L2D
72.14
74.08
68.78
73.42
74.97
70.20
73.24
74.85
69.72
74.21
75.72
70.87
EXPERIMENTS
Datasets. We perform experiments on three benchmark datasets Pascal VOC2007 (Everingham al., 2015) (VOC for short), MS-COC02014 (Lin et al.] 2014) (MS-COCO for short) and NUS- WIDE (Chua al. 2009). VOC contains 5,011 images in the train-val set, and 4,952 images in the test set. It covers 20 common objects, with an average of 1.6 labels per image. MS-COCO contains 82,081 training images and 40,137 test images. It covers 80 common objects, with average of 2.9 labels per image. NUS- WIDE contains 161,789 training images and 107 107,859 test images. covers 81 visual concepts, with average of 1.9 labels per image.
Metrics. Following existing works (Zhang & Zhou 2013 Liu et al. 2021a la Sun et al. 2022), we adopt the mean average precision (mAP) over all classes, overall FI score (OF1) and average per- class Fl-score (CF1) to evaluate the performance. We choose OF1 and CFI, since they consider both recall and precision and thus more comprehensive.
Methods. To validate the proposed method, we compare it with the following KD methods: RKD (Park al. 2019), which captures the relations among instances guide the training of the student model; PKT (Passalis & Tefas 2018), which measures KL divergence between features by treating
them as probability distributions; ReviewKD (Chen et al., 202T) which transfers knowledge across
different stages instead of just focusing on features in the same levels; MSE (Xu et al., 2022a), which
minimizes the MSE loss between logits of teacher and student model; PS (Song et al. minimizes KI. divergence of logits after partial softmax function.
2021 which
Implementation Details. We use the models pretrained on ImageNet (Deng et al. 2009) as the backbones. We resize the resolution of all images to 224 x 224 and set the batch size as 64. For each training image, we adopt a weak augmentation consisting of random horizontal flipping and a strong augmentation consisting of Cutout (Devries & Taylor 2017) and RandAugment Cubuk et al. 2020). We use the Adam optimization method (Kingma & Ba 2015) to train the model for 80 epochs. The one-cycle policy is used with maximal learning rate of 0.0001 and the weight decay (Loshchilov & Hutter 2018) of 0.0001. For all experiments, we set AMIDD = 10, ALED = 100,
and ÀLEDAD 1000. Parameter sensitivity analysis in Appendix D shows that the performance of
7

---

Under review as a conference paper at ICLR 2023
100
Vanilla
90
RKD
MLD
PKT
L2D
mAp
80
bottle
plan
chair
sofa
table
cow
sheep
bus
mbile
dog
bird
haar
hike
cat
car
horso
person
train
aoro
mdp
Figure 4:
Comparison results of the comparing methods on VOC terms of AP and mAP (%),
where the backbones of teacher and student model are respectively ResNet-50 and ResNet-18.
L2D insensitive all of our balancing parameters. For the comparing methods, we set their parameters as suggested in the original papers. Especially, for all feature-basee based methods, we just deploy them the feature maps which is output from the visual backbone fi All the experiments are conducted on GeForce RTX 2080 GPUs. More details about the used models and implementation of the label-wise embedding encoder are attached in Appendix
4.1 COMPARISON RESULTS
Table 1 and Table 2 report comparison results on MS-COCO with the same and different archi- tectures of student and teacher models. From Table [ can observed that: 1) Conventional feature-based distillation methods only achieve minor improvements in performance when com- pared with the student model (without distillation). This indicates these methods do not work in multi-label scenarios due to their disability to capture multiple semanties occurred in feature maps. 2) MLD can outperform conventional feature- based distillation methods in most cases, which indi- cates by performing one-versus-all reduction, the logits-based distillation can be adapted into multi- label knowledge distillation. 3) The proposed L,D significantly outperforms all other methods and achieves comparable performance with the teacher model, which convineingly validates the effec- tiveness of the proposed label-wise embedding distillation. From Table 2 we can see: 1) Compared with the same architecture, the performance gap between teacher and student model is larger for different architectures, which indicates the corresponding distillation task is harder. 2) Our method significantly outperforms all the comparing methods by significant margin in all cases. 3) L2D achieves the best performance in all cases and significantly outperforms MLD. These results pro- vide strong empirical validation for the effectiveness the proposed method.
Figure 4 fillustrates the performance of the proposed methods and other comparing methods on VOC in terms AP and mAP. noteworthy that the performance is ranked in descending according to the performance of student model. From the figure, it can be observed that: 1) Our proposed L,D achieves the best performance and significantly outperforms the comparing methods in terms of mAP. 2) L2D consistently achieves superior performance the comparing methods in most classes and the performance gap is large especially for classes that the student model achieves poor performance. This observation discloses that L2D improves the performance of hard classes by enhancing the distinctiveness of feature representations. These experimental results demonstrate the practical usefulness of the proposed method. More results on VOC and NUS-WIDD can be found in Appendix B and Appendix
4.2 ABLATION STUDY
In this section, to further analyze how the proposed method improves distillation performance, Table 3 reports results of ablation studies MS-COCO (teacher: ResNet-101, student: ResNet-34) and VOC (teacher: ResNet-50, student: ResNet-18). The first line in Table 3 reports the performance of student model without knowledge distillation. It can be observed that the distillation perfor- mance is improved slightly by only conducting multi-label logits distillation. Compared with MLD, label-wise embeddings distillation achieves the major improvement for the proposed method. It can
8

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation studies MS-COCO and VOC.
MS-COCO
Pascal VOC 2007
MLD
CD
ID
mAP
OFI
CFI
mAP
OF1
CFI
70.31
72.49
66.82
84.01
83.60
79.42
70.68
72.69
67.19
84.48
84.07
80.29
71.91
73.74
68.76
85.65
85.64
81.94
71.79
73.62
68.43
84.59
84.19
80.24
72.87
74.45
85.71
85.70
82.11
(a) Vanilla
(b) PKT
(c) MLD
(d) L2D
Figure 5: The differences between correlation matrices student and teacher predicted probabilities on MS-COCO. The student distilled by method shows significant matching between student's and teacher's correlations compared to others.
be observed that by performing CD and ID, the mAP performance achieves 1.6% and 1.48% incre- ments, respectively. Finally, we also examine the combination of these techniques. By incorporating these components together, fusing method achieves the best performance and significantly out- performs each other method. These results demonstrate that all of three components are of great importance to the performance of the proposed L2D.
4.3 DISTILLING INTER-CLAS: CORRELATIONS
As discussed in the previous works (Hinton et al. 2015), the conventional supervised learning losses, e.g. BCE loss, often neglects the correlations among class predictions. However, the label corre- lation is foundational element for multi-label classification. Distillation losses utilize "soft tar- gets", which can effectively distill such correlation information from the teacher model, leading to desirable distillation performance. To validate whether the correlations can be captured by L2D effectively, Figure 5 illustrates the differences between correlation matrices of student and teacher predicted probabilities MS-COCO. From the figure, it can be observed that: 1) Without knowl- edge distillation, the teacher and student correlations are very different, which indicates teacher model tends capture formance. 2) The representative comparing method PKT shows large difference, which discloses that the conventional KD methods ineffective to capture the correlations in multi-label scenarios. 3) MLD shows reduced difference, which indicates that the rich informativeness of logits is bene- ficial for capturing precise correlations. 4) L2D shows significant matching between the teacher and student correlations. By enhancing the distinctiveness of label-wind -wiee embeddings, L2D can obtain more correct predicted probabilities, leading more precise correlation estimation.
precise correlation than student model and thus achieves better per-
5 CONCLUSION
The paper studies the problem of multi-label knowledge distillation. In the proposed method, the multi-label logits distillation explores the informative semantic knowledge compressed the teacher logits to obtain more semantic supervision. Furthermore, the label-wise embedding distillation ex- ploits the structural knowledge from label-wise embeddings to learn more distinctive feature repre- sentations. Experimental results benchmark datasets validate the effectiveness of the proposed method. In future, we plan to improve the performance of MLKD by exploiting other abundant structural information.

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY
We list the parameters for running the proposed algorithm in Section 4 and implementation details of the label-wise embedding encoder can be found in Appendix [ The source code and detailed description about the code are attached in the supplementary material.
The used neural networks
REFERENCES
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5-55500005111 2021.
Zhao-Min Chen, Xiu-Shen Wei, Xin Jin, and Yanwen Guo. Multi-label image recognition with the IEEE
joint class-aware map disentangling and label correlation embedding. In Proceedings the International Conference on Mullimedia and Expo, pp. 622-627, 2019a.
Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5177-5186, 2019b.
Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yan-Tao Zheng. Nus- wide: A real-world web image database from national university of singapore. In Proceedings of ACM Conference Image and Video Retrieval, 2009.
Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Randaugment: Practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Systems, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2009.
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 04555,, 2017.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma. Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vtg-style convnets great again. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 13733-13742, 2021.
Mark Everingham, SM Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, pp. 98-136, 2015.
Frangois Fernandez. Torchcam: class activation explorer. https://github. com/ Errgfm/torch-cae 2020.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A
survey. International Journal of Computer Vision, pp. 1789-1819, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings the IEEF/CV Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02533 2015.
Peng-Tac -ag Jiang. Chang-Bou Bin Zhang, Qibin Hou, Ming- Cheng, and Yunchao Wei. Layercam: Exploring hierarchical class activation maps for localization. IEEE Transactions on Image Pro- cessing, pp. 5875-5888, 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference Learning Representations, 2015.
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, 2012.
Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi. General multi-label image classi- fication with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16478-16488, 2021.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision, pp. 740-755, 2014.
Shilong Liu, Lei Zhang, Xiao Yang. Hang Su, and Jun Zhu. Query2label: A simple transformer way to multi-label classification. arXiv preprint arXiv:2107.1083/ 2021a.
Yongcheng Liu, Lu Sheng, Jing Shao, Junjie Yan, Shiming Xiang, and Chunhong Pan. Multi-lahel image classification via knowledge distillation from weakly-supervised detection. In Proceedings of the 26th ACM International Conference on Multimedia, pp. 700-708, 2018.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 10012-10022, 2021b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer-
ence on Learning Representations, 2018.
Roy Miles, Adrián López Rodríguez, and Krystian Mikolajezyk. Information theoretic representa- tion distillation. arXiv preprint arXiv:2 (tt::2//2.00455 2021.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 5191-5198, 2020.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision, pp. 268-284, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, pp. 8748-8763 2021.
Tal Ridnik, Emanuel Ben- Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. Asymmetric loss for multi-label classification. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 82-91, 2021a.
Tal Ridnik, Gilad Sharir, Avi Ben-Cohen, Emanuel Ben-Baruhh, and Asaf Noy. MI-decoder: Scal- able and versatile classification head. arXiv preprint arXiv:2111.129.0 2021b.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint axXiv:1412.6550, 2014.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018
Liangchen Song, Jialian Wu, Ming Yang, Qian Zhang, Yuan Li, and Junsong Yuan. Handling difficult labels for multi-label image classification via uncertainty distillation. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 2410-2419, 2021.
Feng Sun, Ming- Kun Xie, and Sheng-Jun Huang. A deep model for partial multi-label image clas- sification with curriculum based disambiguation. arXiv preprint arXiv:2207.00 02410, 2022.
11

---

Under review as a conference paper at ICLR 2023
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference on Learning Representations, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
Jiazh Xu, Sheng Huang, Fengtao Zhou, Luwen Huangfu, Daniel Zeng, and Bo Liu. Boosting multi-labe image classification with complementary parallel self-distillation. In Proceedings of the Thirty-Fite First International Joint Conference Artificial Intelligence, 2022a.
Shichao Xu, Yikang Li, Jenhao Hsiao, Chiuman Ho, and Zhu Qi. A dual modality approach for (zero-shot) multi-label classification. arXiv preprint S2ki::200..05662.. 2022b.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03922 2016a.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference, 2016b.
Min-Ling Zhang and Zhi-Hua Zhou A review on multi-label learning algorithms. IEEE Transac- tions Knowledge and Data Engineering, pp. 1819-183,, 2013.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4320- 4328, 2018
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11953-1 11962, 2022.
12

---

Under review as a conference paper at ICLR 2023
A MORE DETAILS OF IMPLEMENTATION
In order to validate the proposed method with diverse architectures, we employ some commonly used models, including ResNet (He et al. 2016), Wide ResNet (WRN) ZZaoorukko & Komodakis 2016b), RepVGG (Ding et al.) 2021), Swin Transformer (Liu et al.
2021b), and MobileNet v2 (Sandler et al. 2018). For all the backbones, we utilize their pre-trained version on the ImageNet Deng al. 2009) our base model.
For all experiments, similar to the previous work (Ridnik et al., 2021b). we employ the label-wise embedding encoder consisting of cross-attention module and a feed-forward fully-connected layer (Vaswani et al.) 20177. The cross-attention module takes full queries and feature maps as the input. We assign a query per class ensure that each query corresponds to a single semantic. The multi- label classifier h(.) is fully-connected layer for each class, which outputs predicted logit for class label based on the input label-wise embedding.
In experiments shown in Figure [ for distillation cross the same architectures, we use Swin-S (Liu et al., 2021b) as the teacher and Swin-T as the student; for distillation cross different architectures, we use ResNet-101 (He et al. 2016) as the teacher and MobileNet v2 (Sandler et al. 2018) as the student.
B MORE RESULTS ON PASCAL VOC 2007
Table and Table 5 report comparison results on Pascal VOC 2007 with the same and different architectures of student and teacher models. From the tables, it can be observed that the proposed L,D significantly outperforms all comparing methods, which convineingly validates the effective- ness of the proposed label-wise embeddings distillation. Compared with the results on MS- -OCO, the performance gap between L2D and comparing methods seems to become smaller. One possible reason is that VOC only contains about 1.5 labels per image, which leads conventional KD methods to obtain a better performance.
Table 4: Results on Pascal VOC 2007 validation teacher and student models are in the same archi- tectures.
Teacher
RepVGG-A2
ResNet-50
WRN-101
Swin-S
Student
RepVGG-A0
ResNet-18
WRN-50
Swin-T
Teacher
86.20
85.63
82.62
86.73
84.92
81.21
88.00
87.03
83,72
92.75
91.05
88.82
Student
83.79
83.36
79.83
84.01
83.60
79.42
88.52
87.21
84.08
91.31
88.00
RKD
83.75
83.41
79.85
84.48
83.54
79.83
88.21
87.33
84.55
91.52
90.44
88.51
PKT
83.63
83.53
80.04
84,
83.10
79.31
87.69
87.07
84.14
91.28
90.17
88.03
Review KD>
83.87
83,98
80.54
83.71
83.01
79.25
88.23
87.13
84,20
91.45
90.17
88.06
MSE
84.02
83.67
79.94
84.23
83.16
79.29
88, 88.04
86.49
83.57
91.06
89.99
37.66
PS
83.77
83.74
80.28
84.44
83.78
79.95
88,30
86.92
83.91
91.21
90.25
88.12
MLD L2D
83.65
83,66
80.02
84.48
84.07
80.29
88.29
87.16
84,25
91.43
90.72
88.81
84.56
84.37
80.82
85.71
85.70
82.11
89.52
88.25
85.69
91.92
91.34
89.58
Table 5: Results on Pascal VOC 2007 validation where teacher and student models are in the different architectures.
Teacher
ResNet-50
Swin-T
ResNer-50
Swin-T
Student
RepVGG-A0
18sNet-18
MobileNet v2
MobileNet
Teacher
86.73
84.92
81,21
91.43
89.81
87.63
86.73
84.92
81.21
91.43
89.81
87.63
Student
83.79
83.36
79,83
84.01
83.60
79.42
86.12
85.01
81.76
86.12
85.01
81.76
RKD
84.26
84.29
80.70
83.27
83.05
79.55
86.22
84.97
81.76
85.68
85.31
81.57
PKT
83.93
83.79
80.03
83.45
83.25
79.64
86.10
84.84
81.66
85.67
85.22
81.68
Review KD
84.07
83.62
80.34
83.37
83.08
78.93
85.87
85.04
81.73
85.69
85.10
81,56
MSB
84.01
84.05
80,52
83.60
83.06
79.46
86.20
84.94
81 81.84
85.80
85.51
81.98
PS
84.80
84.46
81.13
83.97
83.75
79.86
86.26
85.47
82.06
86.07
85.73
82.39
MLD
85.07
84,91
81.55
84.61
84.26
80.78
86.38
85.67
82.43
86.1
85.98
82.55
L2D
86.26
85.85
82.55
85.87
85.67
82.17
87.32
86.48
83.26
87.37
86.88
83.68
13

---

Under review as a conference paper at ICLR 2023
C MORE RESULTS ON NUS-WIDE
Table reports comparison results on NUS-WIDE with the same and different architectures of student and teacher models. For distillation between the same architectures, we choose a ResNet- 101 (He et al. (016) as the teacher and ResNet-33 34 as the student. For distillation between different architectures,
choose Swin-T (Liu et al. 202 the teacher and MobileNet v2 (Sandler et al. 2018) as the student. From the tables, it can be observed that the proposed L2D significantly outperforms all comparing methods, which convincingly validates the effectiveness of the proposed label-wise embeddings distillation.
Table 6: Results NUS-WIDE validation.
Teacher
ResNet-101
Swin-T
Student
ResNet-34
MobileNet v2
Metries
mAP
OFI
CFI
mAP
OF1
CFI
Teacher
55.32
75,56
61.31
59.73
77.30
65.44
Student
53.41
75.10
60.08
54.49
75.72
61.74
RKD
53.62
75.20
59.91
54.76
75.69
61.74
PKT
53,55
75.08
60.35
54.59
75.69
61.74
ReviewKD
53.52
75.23
60.44
54.85
75.84
61.75
MSE
53.52
75.13
59.94
54.86
75.80
61.69
PS
54.14
75.43
60.79
55.18
75.91
62.35
MLD
54.44
75.36
60.73
55.36
76.00
62.52
L2D
55.31
76.17
62.79
56.91
76.92
63.89
D PARAMETER SENSITIVITY ANALYSIS
In this section, study the influence of balancing parameters AMLD. ÁLED-CD and ÀLED-ID the performance of L2D. A commonly used setting the hyperparameter in vanilla KD that balances KL divergence against CE is 0.9 (Tian al. 2019), which means the balancing parameter for CE is 0.1 and the one for KL divergence is 0.9. So we choose for the balancing parameter for MLD, which is closest to the setting of vanilla KD. We the balancing parameter for LED-ID larger than LED-CD considering that the LED-ID may carry less information because there are only less than 3 labels for instance average, though it seems unnecessary since parameter sensitivity experiments in Figure 6 show that the performance of L2D are not sensitive to all of our balancing parameters.
75
70
70
70
65
mAP
65
mAP
65
mAP
OF1
--A--
OFI
OFL
60
CF1
60
CF1
60
CFI
10
100
10
100
1000
100
1000
10000
(a) AMLD
(b) ALED-CD
(c) ALED-I
Figure 6: Student models' performance comparisons with different values of \MLD. ALED-CD and ALedad respectively on MS-COCO with a ResNet-101 as the teacher and ResNet-34 as the student.
E VISUALIZATION OF ATTENTION MAPS
To further show the effectiveness of our proposed method L2D, we visualize some attention maps of the penultimate layer in the visual backbones using LayerCAM (Jiang al. 2021 implemented
14

---

Under review as a conference paper at ICLR 2023
by Fernandez (2020). We compare attention maps of the student model trained by L2D with some other methods in Figure 7 We compare L2D with: Vanailla: student trained without distillation; 2)PKT. classical feature-basee based method. In each figure, the first column shows the raw picture and the other columns show class activation maps overlaying on the raw picture. Fach row represents certain class. From these figures, we can find that L.2D can locate the specified object more pre- cisely than the other methods, which means it can not only pay attention to target objects, but also resist interference from similar but unrelated objects. All these comparisons show that L2D outper- forms all comparing methods. It validates the effectiveness of our proposed label-wise embeddings distillation and shows great potential in MLKD.
raw picture
Vanilla
PKT
L2D
handbag
fie
person
Figure 7: An example of visualization of attention maps. We can find that on attention head for class handbag, both Vanilla and PKT are interfered by some other objects and do not pay all attention to the handbag, but our L2D resists such interference successfully.
15

---

Under review as a conference paper at ICLR 2023
raw picture
Vanilla
PKT
L2D
airplane
bicycle
person
Figure 8: An example of visualization of attention maps. We can find that on attention head for class airplane, both Vanilla and PKT do not pay all attention to the airplane: Vanilla is interfered by the plant and PKT is interfered by the boy. But our L2D resists such interference successfully. On attention head for class person, both Vanilla and PKT are interfered by the shades the cloud, but L2D is not.
raw picture
Vanilla
PKT
L2D
umbrella
person
Figure 9: An example of visualization of attention maps. We can observe that on attention heads for class car and class person, Vanilla pays some of its attention to unrelated objects. attention head for class umbrella, PKT pays some of its attention to the house. Only L2D can concentrate on these three targets precisely.
16


Reviews:
[{"review_id": "jNTELrS0xX", "reviewer": "Reviewer_fqJx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a multi-label knowledge distillation method. On the one hand, it exploits the logit's informative semantic knowledge by label decoupling with the one-versus-all reduction strategy. On the other hand, it enhances the distinctiveness of the learned feature representations by leveraging the structural information of label-wise embeddings. ", "summary_of_the_review": "Although it would be better to add several more results in the main paper, this paper addresses an interesting problem, and the performance of the proposed model seems favorable.\n\n\n-----after rebuttal-----\n\nAfter reading the comments from the other reviewers, I think the novelty of the proposed method is not strong enough.\nTherefore, I decided to reduce the score to 'below threshold.'", "strength_and_weaknesses": "Strength\n- The authors tackle a relatively new problem, knowledge distillation for multi-label classification, and they propose a method suitable to address the problem of the existing approaches.\n- The proposed method is straightforward and shows favorable performance.\n\n\nWeakness\n- This work is not the first work to apply the knowledge distillation technique to the multi-label classification model. As the authors mentioned, Song et al. and Xu et al. also apply knowledge distillation to multi-label classification models. There are several more works with similar frameworks such as [1]. In the introduction, the authors should not only introduce the “distillation for multi-label classification” but also explain what the problem of the existing frameworks is and how the proposed method addresses the problem. The authors should explain why the proposed method shows favorable performance.\n[1] Multi-label image classification via knowledge distillation from weakly-supervised detection. ACM MM 2018.\n- The proposed method itself is simple and straightforward. Therefore, it would be better to further analyze the design choices of the proposed model to claim the impact of the proposed method. For example, why is the loss function in Eqn (6) used? Is there any better option for the authors to try for the loss function?\n- It would be better to have more experimental results in the main paper. For example, instead of the results for the VOC dataset in Fig. 4, it would be more informative to show the result with different architectures like Tables 1 and 2. Also, the results on the NUS-WIDE dataset, which is a famous multi-label classification dataset, are missing.\n- Finally, it would be better to have an analysis of the CAM before and after applying the proposed method, which is popularly used in knowledge distillation literature. For the analysis of Fig. 5, it is hard to see the correlation due to the large class number. I suggest analyzing the VOC dataset to better visualize the correlation between classes.\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "P-EZjdeJmgf", "reviewer": "Reviewer_gMao", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel method for knowledge distillation in multi-label learning scenarios. Multi-label learning aims to solve the problem where multiple positive labels exists in a given single sample. However, general knowledge distillation methods mainly focus on single-label learning scenario while ignoring the latent label correlations/structural knowledge across the labels or the label embedding.\n\nThis work proposed a specific design framework which preserves the label structural knowledge in knowledge distillation training process. More specifically, a label decoupling with the one-versus all reduction strategy is deployed, and label-wise embedding is further explored to get the label structural knowledge across different label. Experimental results demonstrate the effectiveness of the proposed model.\n", "summary_of_the_review": "This paper proposed a novel approach for multi-label knowledge distillation. The proposed modules are reasonable with high performance.", "strength_and_weaknesses": "Strength:\n\nThe background and the motivation of the setting is well-introduced. The motivation of the work is reasonable and logical.\n\nThe proposed modules including the label decoupling and the embedding-based lobal correlation learning are clearly introduced with motivation and explanations. The motivation is rational and the explanation is reasonable.\n\nExperiments demonstrate the effectiveness of the proposed model.\n\nWeaknesses:\n\nEmbedding based label structural knowledge extraction/learning is a general approach for multi-label learning. To this end, there are one module which could be considered as a novel module. More explanations about why/how the embedding strategy is novel could be introduced.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MvRYc5osnsF", "reviewer": "Reviewer_NCc1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes doing knowledge distillation (KD) for multi-label ML models via KD approaches at both logit and feature representation levels.  The major novelty comes from the two proposed label-wise embedding distillation approaches, i.e., LED-CD and LED-ID, which leverage the multi-label info to implement the relation-based knowledge distillation. The experimental results, however, are insufficient for justifying or providing insights into the proposed approach.", "summary_of_the_review": "Though the paper proposes an interesting idea of label-wise embedding distillation leveraging the multi-label info, the experimental setup in the paper, however, makes it difficult to justify the effectiveness of the proposed distillation method. Because of that, I recommend rejecting the paper. I also suggest that the authors improve the literature survey, implement a stronger baseline, and conduct experiments verifying the loss design in Eq. (4) and (7). ", "strength_and_weaknesses": "# Strength\n* The paper is well-written and easy to understand.\n\n# Weakness\n* The literature survey for knowledge distillation in this paper is incomplete. The survey only covers two KD categories (logit-based and feature-based) and misses the relation-based category [1]. This miss is critical as the major novelty of this paper comes from proposing losses in this third KD category.\n* The proposed KD approach requires labeled data, which limits its application. I would question if Eq. (4) and (7) need to consider only the distances between positive label embeddings. It may also work if we simply include the embedding distances for all label pairs in the loss – disregarding whether the labels are positive or not. Would be good to have experiments verifying the design decision for Eq. (4) and (7).\n* Some claims in the paper lack supports from the experiments. For example, the authors claim the LED-ID loss would leverage spatial relations between labels. It would be good to have experimental results supporting such a claim.\n* Some experimental setup lacks explanation, e.g., how are the weights for MLD, LED-CD, and LED-ID losses determined as 10, 100, and 1000 in this paper?\n* The baselines are considerably weak as they are all with a single type of knowledge distillation, while the proposed L2D combines multiple distillation losses (i.e., BCE, MLD, LED-CD, and LED-ID). The proposed L2D loss may have the best results simply because of the loss combination (a known approach for performance improvement). A fair baseline should also be a system combining multiple losses.\n\n\n\n[1] Gou, J., Yu, B., Maybank, S.J. et al. Knowledge Distillation: A Survey. Int J Comput Vis 129, 1789–1819 (2021). https://doi.org/10.1007/s11263-021-01453-z  (available at https://arxiv.org/pdf/2006.05525.pdf )", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HK9ro0fiUON", "reviewer": "Reviewer_SHPH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a distillation method on multi label image classification task. It introduces a multi label logits distillation task and intra-class/instance embedding distillation loss. And performs experiments on COCO and VOC dataset demonstrating the effectiveness of the\nproposed method. The source code is provided for reproducibility.", "summary_of_the_review": "This paper is well-written and the experiments result shows the improvement over the previous method on certain setting. But more experiments are needed to show the improvements over previous methods. Also there is a concern about the novelty.", "strength_and_weaknesses": "Strength:\n1. The approach is simple and easy to implement.\n2. The experiments result shows the improvement over the previous method on resolution 224x224.\n3. The code is released for reproducing the results.\n\nWeakness:\n1. Although the title is Multi-Label Knowledge Distillation (MLKD), and as stated in first contribution this paper proposes a general MLKD framework, it is only about multi label image classification.\n2. The experiment part only covers VOC and COCO with 224x224 resolutions. However, previous work [1] [2] mainly working on 448x448, and thus it is hard to compare and validate the result of this work. Also it would be better to include the result on NUS-WIDE.\n3. There seems not enough novelty by only adding a intra-class/instance embedding distillation loss.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8NS8buOV0g4", "reviewer": "Reviewer_c6wM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents the multi-label knowledge distillation to address the issue of multiple semantic labels in multi-label learning scenarios. MKD and LED are introduced for network learning. Experiments on several datasets demonstrate its superiority.", "summary_of_the_review": "My main concern lies in the novelty, since both multi-label classification and KD have been well studied, and this paper does not provide some impressive new design for this task.", "strength_and_weaknesses": "Strength:\n\n(1) The organization is good, which makes the paper easy to follow.\n\n(2) The method is simple, and the results on several datasets seem good.\n\nWeaknesses:\n\n(1) The presentation could be improved. For example, the abbreviation \"MLL\" lacks explanation. The meaning of the red line in Figure 1 is also unclear, and I guess it is the result of the teacher.\n\n(2) For the BCE loss, it is commonly used for multi-label classification. The simple distillation of MLD for this predicted probability is not novel. Moreover, the structural relation by class-aware label-wise embedding distillation and instance-aware label-wise embedding distillation is similar to the global graph and local graph-based similarity consistency. In this case, this paper can be regarded as a combination of existing multi-label learning and KD methods, and the overall novelty is not satisfying. Though the authors present several different distance measurements, the overall contribution is still insufficient.\n\n(3) The authors mainly compare with traditional KD methods. Please explain how these methods deal with the multi-class situation in the paper.\n\n(4) I would like to see a detailed comparison with existing multi-label methods and KD methods in the loss function, respectively.\n\n(5) According to the results in Tables 1 and 2, the L2D even achieves much better results than the teacher. Please explain this phenomenon.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject