Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: - This paper is established by a discovered phenomenon that in distillation, not all the classes’ performance is improved although the student is improved significantly. 

- Thus, the authors explore a method that focuses on promoting the performance of the poorly-performed labels, which also lead to the overall promotion while distillation.

- The authors propose two methods to achieve it, one is the distillation with adaptive mixing weights and the other is the distillation with per-class margins.

main_review: - The authors aim at improving the performance of the underperforming groups of classes, which can be easily observed. Thus, for me, this observation is not surprising enough. So in section 3 (are distillation’s gains uniform), the authors use big space of the main paper to tell us the conclusions, which are already known by the researchers in the field of distillation. These conclusions are mainly roughly drawn from the simple experiments and lack the theoretical support. In this way, this paper is more or less like an experimental report.

- In the main paper, it is said that the labels that are not well learned by the teacher performance even worse in the student network with direct distillation. But I just wonder if it is also a kind of knowledge to be learned? Is it necessary to just simply eliminate this?

- The proposed two methods are too straightforward and intuitive and not new. So, the contribution of the main method is limited. 


- Then about the experiments. Does the author mainly focus on the self-distillation? I don’t think in the situation of self-distillation, the problem of the unbalanced performance should be well focused. What about the experiments in transfer learning? There are more poorly performed labels.



summary_of_the_review: - please restate the contribution of the proposed method.

- should need more experiments. like in transfer learning. And need more analysis, like what is the performance of the best-performance group of labels.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper identified an underexplored problem with distillation, fairness of distillation on different subgroups. This paper further proposed two methods to tackle this challenge. Experiments are conducted on CIFAR-100, ImageNet, and their long-tailed variants.
main_review: **Strengths**

The identified problem, distillation performance on different subgroups, is highly related to the fairness of machine learning, which is an interesting and important research topic. Experiments on balanced and long-tailed datasets further evaluate how distillation performs diversely on different subgroups.

This paper proposed two methods for improving subgroup performance, AdaWeight with adaptive mixing weights and AdaMargin with per-class margins.

This paper is clearly written and well organized. The writing quality is satisfying.

**Weaknesses**

The improvements of AdaAlpha and AdaMargin on ImageNet LT, a larger dataset for long-tailed classification, are marginal. Specifically, AdaAlpha achieves only an improvement of 0.15 mean per-class accuracy, and the training of AdaMargin diverges. It is not clear whether AdaAlpha works on large datasets. This marginal improvement limits the contribution of the proposed methods. Another large-scale dataset, iNaturalist 2018, may be helpful to further evaluate the effectiveness of AdaAlpha and AdaMargin.

The contribution of AdaMargin is somehow limited. As shown in Table 3, AdaMargin achieves a significant improvement of worst-1/10 per-class accuracy on CIFAR-100. However, the improvement on ImageNet compared to one-hot and distillation are very marginal. Also, the training of ImageNet LT diverges. These results show that AdaMargin is not a good choice in practice, which limits its technical contribution. 

For the long-tailed setting, this paper evaluated the performance on worst-1, worst-10, worst-100 subgroups. The popular subgroups in long-tailed recognition are many-shot, medium-shot and few-shot. It is okay to choose not to report the performances on the normal subgroups, but an explanation on why not using these subgroups is welcomed.
summary_of_the_review: Overall, this paper proposed an interesting and important perspective on the fairness of distillation, which will inspire future works to pay more attention to the fairness of distillation. The writing quality is good. The contributions to empirical analysis are acknowledged by my side.

The weakness comes from the technical contributions. In particular, it seems that the proposed AdaMargin does not work well in practice. This limits the contributions of the methods part. The contribution of AdaAlpha is clear.

Considering both the strengths and weaknesses, I vote for acceptance at this stage. My initial score is between 6 and 8.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to have a closer look at (logit matching) knowledge distillation, and explores whether the gains are uniform across all portions of the test data or if instead there are gains on some portions of the data but damaged performance on others. This is justified by the potential of KD to amplify biases. The authors quickly show that indeed the gains are not uniform, some classes being affected negatively by KD so that the bad performance of the teacher on some classes is amplified on the student. Then the authors conduct a number of experiments to study the impact of different aspects of training (e.g. architectures involved and aspects of data like class imbalance) over this amplification of class-specific poor performance. They are then able to identify what classes are likely to be negatively affected and follow that with the proposal of two modifications of the basic KD formulation with the aim to improve KD effect on the worse-performing classes. Experiments focus on CIFAR-100 and ImageNet, including long-tail variants.
main_review: The paper is a pleasure to read, very well written, clear and convincing with their arguments, and do some good work (with small caveats) regarding the experimental support of the claims. I also like that ImageNet is thoroughly used rather than sticking to datasets like CIFAR.

My main issues however are 1) while biases in learning is clearly a topic that has relevance nowadays, the focus of this paper ends up being on some specific aspect of a specific technique that might cause bias amplification. Thus, the focus is quite narrow 2) The results are not impressive. Performance improvements, even on the specific focus group (lower-performing classes) the gains are not very strong or consistent. In more detail:

Regarding 1, generally speaking, I think the community has too much of a fascination with the logit matching knowledge distillation method. Knowledge distillation is widely used and very important, but the specific logit matching flavour of KD is a 2015 paper that has been thoroughly improved multiple times. I am always a bit confused about why the community keeps producing a large corpus of works dissecting small aspects of logit matching KD, while forgetting about the whole literature that followed that seminal paper. (I have to admit here though that logit matching KD still performs admirably against some of the other methods that are supposed to work better, and that it also tends to give gains when combining with other KD methods). 

Still, nowadays the state of the art is well above logit matching KD. I would be much more interested in the paper if it was applicable across KD methods (or at least on top of a couple of state of the art methods). For example, is it relevant for feature distillation (Heo et al., A Comprehensive Overhaul of Feature Distillation, ICCV'19)? or can it be combined with Xu et al., Knowledge Distillation Meets Self-Supervision, ECCV'20? 

Issue 2: The main results, shown in Table 3, are not very impressive. On ImageNet, (logit matching) distillation marginally hurts overall performance. The proposed method shows that it can keep the performance to roughly "no distillation" level, and for ImageNet LT the worst-1 and worst-10 accuracy are basically 0% for all (thus no improvement) and the worst-100 actually takes a small hit (quite irrelevant given the overall poor performance, but still). Also, the average performance is improved marginally. So all in all, the CIFAR-100 are the only convincing set of results.

Comparison with other approaches is also somewhat absent, e.g. Zhou et al (2021) (I think the most related method?) is only compared in one setting, and a table with a wide set of comparisons against other state of the art distillation methods is also absent (see for example Tables 3 and 4 of the aforementioned Zhou et al (2021) paper).


Other aspects (not really important for the overall score and not necessary to reply):
- On Fig. 1 caption: shouldn't it be 2% absolute instead of 1% absolute?
- Page 2, bottom: the effect of the temperature is different (inverted) if > 1 or < 0, so the last phrase of the page is incorrect.
- Eq. 2 is quite trivial, I'm not sure it is necessary to spell it out.
- Section 3.1, the hyperparameter choices are not in appendix C, but in B.2 I believe.
- Same paragraph, +0.4 improvement over the baseline is hardly a success... I'm not sure this kind of claim is too healthy
- Table 1 could do with a CIFAR baseline so we can compare with line 3. Also, when comparing the long tail behavior with the standard behavior, we see less variance (less avg. gain, better worse performance), which feels kind of contradictory with the train of thought of the paper?
- The last paragraph of section 3.4 looks unsubstantiated to me.
- The analysis, especially for the experiments involving less classes, could do with some statistics - if the per-class performance is drawn from a normal distribution, it is clear that the worst performing one will depend on the number of classes (i.e. samples drawn from the distribution), so it is not surprising that reducing the number of classes of ImageNet means the performance of the worse class improves.
- It feels like the formulation is calling for a per-sample weighting... in the end the average per-class performance is used as proxy but it seems to me that what matters is the behavior of the teacher for a specific instance.
- Zhou et al. (2021) looks like the most related work, but I feel it is not explained properly in the related work. Explaining that method better, and highlighting the differences to the current proposal would improve the paper in my opinion.


summary_of_the_review: The paper has some interesting aspects and does a good deep-dive into some very specific aspect of knowledge distillation, but the focus is very narrow and the improvements obtained do not look impressive enough. The score would be around 4 if that existed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work the authors examine the impact of distillation on certain sub-groups of the data. More specifically, they demonstrate that even though distillation can increase the overall accuracy of a model, it can harm the accuracy on certain subclasses. This includes both self-distillation, as well as regular distillation setups. To overcome this, the authors propose using class-specific weights, that have been tuned according to a holdout set, to tune the importance of each class during distillation, as well as using softmax cross-entropy with margins tuned according to the difficulty of each class. Furthermore, it is also demonstrated that these phenomena can also impact the fairness of the model.

main_review: 
+ The paper is well-written and easy to follow and brings out an interesting issue with distillation
+ An easy to implement solution is proposed for addressing the issue highlighted.
- It is not clear if the same classes are "problematic" for each model. In other works, if I train a model 10 times do we expect the drop in performance to be in the same 10 classes? 
- I would expect a more detailed statistical analysis to validate that there is indeed a consistent decrease in classes in a statistically significant manner. This should include training multiple models and evaluating their performance.
- When measuring the worst1/10 performance, we measure the performance on the initial worst 10 classes or there might be a reduction in the performance in other classes? In any case, the average performance is increasing. However, it would be useful to know if the proposed method can negatively impact some classes.
- AdaAlpha/AdaMargin are not compared with more advanced distillation methods. I understand that the issue highlighted by the authors is important. However, it is not clear if this also affects the more recent distillation approaches (e.g., those performing online distillation, distillation from ensembles, using knowledge from intermediate layers, etc.) as well or if it only concerns vanilla distillation.

summary_of_the_review: Overall, I enjoyed reading this paper, which brings out an important issue in distillation. However, I would feel more confident in the reported results if a more detailed statistical analysis was provided, as well as some comparisons with more recent distillation approaches, which leaves me with many open questions. Furthermore, there is no discussion on why distillation brings out this behavior from a theoretical standpoint.
review_title: null

Meta-review: This paper studies knowledge distillation and explores why distillation gains are not uniform. Reviewers consistently find this paper an interesting read, but had common concerns on generalizability and limited improvements/contributions.
In general, reviewers mostly gave a score that is below the acceptance threshold, or expressed concerns otherwise. Summing these up, we conclude this paper is of interest to the ICLR audience, but current form is not ready yet for acceptance.  

Summary Of Reasons To Publish:
interesting analysis of the causes of non-uniform gains in distillation 

Summary Of Suggested Revisions:

 (1) the improvements are marginal and (2) the contribution of AdaMargin is limited, (3) generalizability to other KDs
Decision: Reject