Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes doing knowledge distillation (KD) for multi-label ML models via KD approaches at both logit and feature representation levels.  The major novelty comes from the two proposed label-wise embedding distillation approaches, i.e., LED-CD and LED-ID, which leverage the multi-label info to implement the relation-based knowledge distillation. The experimental results, however, are insufficient for justifying or providing insights into the proposed approach.
strength_and_weaknesses: # Strength
* The paper is well-written and easy to understand.

# Weakness
* The literature survey for knowledge distillation in this paper is incomplete. The survey only covers two KD categories (logit-based and feature-based) and misses the relation-based category [1]. This miss is critical as the major novelty of this paper comes from proposing losses in this third KD category.
* The proposed KD approach requires labeled data, which limits its application. I would question if Eq. (4) and (7) need to consider only the distances between positive label embeddings. It may also work if we simply include the embedding distances for all label pairs in the loss – disregarding whether the labels are positive or not. Would be good to have experiments verifying the design decision for Eq. (4) and (7).
* Some claims in the paper lack supports from the experiments. For example, the authors claim the LED-ID loss would leverage spatial relations between labels. It would be good to have experimental results supporting such a claim.
* Some experimental setup lacks explanation, e.g., how are the weights for MLD, LED-CD, and LED-ID losses determined as 10, 100, and 1000 in this paper?
* The baselines are considerably weak as they are all with a single type of knowledge distillation, while the proposed L2D combines multiple distillation losses (i.e., BCE, MLD, LED-CD, and LED-ID). The proposed L2D loss may have the best results simply because of the loss combination (a known approach for performance improvement). A fair baseline should also be a system combining multiple losses.



[1] Gou, J., Yu, B., Maybank, S.J. et al. Knowledge Distillation: A Survey. Int J Comput Vis 129, 1789–1819 (2021). https://doi.org/10.1007/s11263-021-01453-z  (available at https://arxiv.org/pdf/2006.05525.pdf )
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to understand. The major novelty comes from the two proposed label-wise embedding distillation approaches, i.e., LED-CD and LED-ID, which leverage the multi-label info to implement the relation-based knowledge distillation. However, the experiments in the paper are insufficient to justify the effectiveness of the proposed approach.
summary_of_the_review: Though the paper proposes an interesting idea of label-wise embedding distillation leveraging the multi-label info, the experimental setup in the paper, however, makes it difficult to justify the effectiveness of the proposed distillation method. Because of that, I recommend rejecting the paper. I also suggest that the authors improve the literature survey, implement a stronger baseline, and conduct experiments verifying the loss design in Eq. (4) and (7).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a novel method for knowledge distillation in multi-label learning scenarios. Multi-label learning aims to solve the problem where multiple positive labels exists in a given single sample. However, general knowledge distillation methods mainly focus on single-label learning scenario while ignoring the latent label correlations/structural knowledge across the labels or the label embedding.

This work proposed a specific design framework which preserves the label structural knowledge in knowledge distillation training process. More specifically, a label decoupling with the one-versus all reduction strategy is deployed, and label-wise embedding is further explored to get the label structural knowledge across different label. Experimental results demonstrate the effectiveness of the proposed model.

strength_and_weaknesses: Strength:

The background and the motivation of the setting is well-introduced. The motivation of the work is reasonable and logical.

The proposed modules including the label decoupling and the embedding-based lobal correlation learning are clearly introduced with motivation and explanations. The motivation is rational and the explanation is reasonable.

Experiments demonstrate the effectiveness of the proposed model.

Weaknesses:

Embedding based label structural knowledge extraction/learning is a general approach for multi-label learning. To this end, there are one module which could be considered as a novel module. More explanations about why/how the embedding strategy is novel could be introduced.

clarity,_quality,_novelty_and_reproducibility: This paper clearly introduces the motivation and the proposed models. The training parameters are introduced in the paper. The novelty is fine.
summary_of_the_review: This paper proposed a novel approach for multi-label knowledge distillation. The proposed modules are reasonable with high performance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a multi-label knowledge distillation method. On the one hand, it exploits the logit's informative semantic knowledge by label decoupling with the one-versus-all reduction strategy. On the other hand, it enhances the distinctiveness of the learned feature representations by leveraging the structural information of label-wise embeddings. 
strength_and_weaknesses: Strength
- The authors tackle a relatively new problem, knowledge distillation for multi-label classification, and they propose a method suitable to address the problem of the existing approaches.
- The proposed method is straightforward and shows favorable performance.


Weakness
- This work is not the first work to apply the knowledge distillation technique to the multi-label classification model. As the authors mentioned, Song et al. and Xu et al. also apply knowledge distillation to multi-label classification models. There are several more works with similar frameworks such as [1]. In the introduction, the authors should not only introduce the “distillation for multi-label classification” but also explain what the problem of the existing frameworks is and how the proposed method addresses the problem. The authors should explain why the proposed method shows favorable performance.
[1] Multi-label image classification via knowledge distillation from weakly-supervised detection. ACM MM 2018.
- The proposed method itself is simple and straightforward. Therefore, it would be better to further analyze the design choices of the proposed model to claim the impact of the proposed method. For example, why is the loss function in Eqn (6) used? Is there any better option for the authors to try for the loss function?
- It would be better to have more experimental results in the main paper. For example, instead of the results for the VOC dataset in Fig. 4, it would be more informative to show the result with different architectures like Tables 1 and 2. Also, the results on the NUS-WIDE dataset, which is a famous multi-label classification dataset, are missing.
- Finally, it would be better to have an analysis of the CAM before and after applying the proposed method, which is popularly used in knowledge distillation literature. For the analysis of Fig. 5, it is hard to see the correlation due to the large class number. I suggest analyzing the VOC dataset to better visualize the correlation between classes.





clarity,_quality,_novelty_and_reproducibility: The proposed method is clear and novel. The quality of this paper looks favorable overall.
summary_of_the_review: Although it would be better to add several more results in the main paper, this paper addresses an interesting problem, and the performance of the proposed model seems favorable.


-----after rebuttal-----

After reading the comments from the other reviewers, I think the novelty of the proposed method is not strong enough.
Therefore, I decided to reduce the score to 'below threshold.'
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents the multi-label knowledge distillation to address the issue of multiple semantic labels in multi-label learning scenarios. MKD and LED are introduced for network learning. Experiments on several datasets demonstrate its superiority.
strength_and_weaknesses: Strength:

(1) The organization is good, which makes the paper easy to follow.

(2) The method is simple, and the results on several datasets seem good.

Weaknesses:

(1) The presentation could be improved. For example, the abbreviation "MLL" lacks explanation. The meaning of the red line in Figure 1 is also unclear, and I guess it is the result of the teacher.

(2) For the BCE loss, it is commonly used for multi-label classification. The simple distillation of MLD for this predicted probability is not novel. Moreover, the structural relation by class-aware label-wise embedding distillation and instance-aware label-wise embedding distillation is similar to the global graph and local graph-based similarity consistency. In this case, this paper can be regarded as a combination of existing multi-label learning and KD methods, and the overall novelty is not satisfying. Though the authors present several different distance measurements, the overall contribution is still insufficient.

(3) The authors mainly compare with traditional KD methods. Please explain how these methods deal with the multi-class situation in the paper.

(4) I would like to see a detailed comparison with existing multi-label methods and KD methods in the loss function, respectively.

(5) According to the results in Tables 1 and 2, the L2D even achieves much better results than the teacher. Please explain this phenomenon.
clarity,_quality,_novelty_and_reproducibility: Please refer to my detailed comments above.
summary_of_the_review: My main concern lies in the novelty, since both multi-label classification and KD have been well studied, and this paper does not provide some impressive new design for this task.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a distillation method on multi label image classification task. It introduces a multi label logits distillation task and intra-class/instance embedding distillation loss. And performs experiments on COCO and VOC dataset demonstrating the effectiveness of the
proposed method. The source code is provided for reproducibility.
strength_and_weaknesses: Strength:
1. The approach is simple and easy to implement.
2. The experiments result shows the improvement over the previous method on resolution 224x224.
3. The code is released for reproducing the results.

Weakness:
1. Although the title is Multi-Label Knowledge Distillation (MLKD), and as stated in first contribution this paper proposes a general MLKD framework, it is only about multi label image classification.
2. The experiment part only covers VOC and COCO with 224x224 resolutions. However, previous work [1] [2] mainly working on 448x448, and thus it is hard to compare and validate the result of this work. Also it would be better to include the result on NUS-WIDE.
3. There seems not enough novelty by only adding a intra-class/instance embedding distillation loss.
clarity,_quality,_novelty_and_reproducibility: The paper is writing clearly and easy to follow. The class/instance-aware embedding distillation loss seems common in other areas so not sure if this counts as a novel contribution.
summary_of_the_review: This paper is well-written and the experiments result shows the improvement over the previous method on certain setting. But more experiments are needed to show the improvements over previous methods. Also there is a concern about the novelty.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors present a knowledge distillation (KD) solution for multi-label ML models. The proposed idea exploits the informative semantic knowledge from the logits by label decoupling with the one-versus-all reduction strategy. Moreover, it enhances the distinctiveness of the learned feature representations by leveraging the structural information of label-wise embeddings. Experimental results are reported on several datasets.

The main contributions are the class-aware label-wise embedding distillation (LED-CD) and the instance-aware label-wise embedding distillation (LED-ID), which use multi-label info to put forth relation-based knowledge distillation.

The key strength of the paper, as pointed out by all reviewers, is the quality of the  presentation style along with the good organization of the content. The technical novelty appears instead to be limited/incremental, and some aspects of the contributions seem to exist in prior art (class/instance-aware embedding distillation loss seems common in other areas). The authors’ response did not changed the reviewers’ opinions. Another main problem is the experimental validation, which has actually generated additional concerns after the authors’ rebuttal ( MS-COCO performance reported by the authors for ​​ResNet-101 appear to be not reliable, and  the parameter sensitivity analysis generated some new concerns).

justification_for_why_not_higher_score: The technical novelty is not sufficient. Experimental results lacking in some respect.
Decision: Reject