Title: Understanding the Success of Knowledge Distillation -- A Data Augmentation Perspective
Authors: Huan Wang
Suhas Lohit
Michael Jeffrey Jones
Yun Fu
Venue: ICLR 2022
Keywords: knowledge distillation
data augmentation
mixup
cutmix
model compression
active learning

Abstract: Knowledge distillation (KD) is a general neural network training approach that uses a teacher model to guide a student model. Many works have explored the rationale for its success. However, its interplay with data augmentation (DA) has not been well understood so far. In this paper, we are motivated by an interesting observation in classification: KD loss can take more advantage of a DA method than cross-entropy loss \emph{simply by training for more iterations}. We present a generic framework to explain this interplay between KD and DA. Inspired by it, we enhance KD via stronger data augmentation schemes named TLmixup and TLCutMix. Furthermore, an even stronger and efficient DA approach is developed specifically for KD based on the idea of active learning. The findings and merits of our method are validated with extensive experiments on CIFAR-100, Tiny ImageNet, and ImageNet datasets. We achieve new state-of-the-art accuracy by using the original KD loss armed with stronger augmentation schemes, compared to existing state-of-the-art methods that employ more advanced distillation losses. We also show that, by combining our approaches with the advanced distillation losses, we can advance the state-of-the-art even further. In addition to very promising performance, this paper importantly sheds light on explaining the success of knowledge distillation. The interaction of KD and DA methods we have discovered can inspire more powerful KD algorithms. 


Review discussion:
The paper has the following strengths:
S1: Explaining the rationale behind the success of KD approaches from the point of view of DA approaches is very interesting and the paper makes good progress in this aspect. Figure 2(a) is very illustrative. It clearly demonstrates that compared to applying DA approaches to CE losses, applying DA approaches to KD losses has the potential of bringing more performance gains.
S2: The experimental results presented in the paper are quite extensive and impressive. The paper conducts experiments on 3 datasets: CIFAR-100, Tiny ImageNet, and ImageNet, varying from tens of thousands of images to millions of images. The paper applies a variety of network architectures such as VGG, ResNet, Wide-ResNet, etc. to implement student models and teacher models. The paper does a wide range of ablation studies like studying effects of using more training iterations, exploring effects of using various DA approaches or data picking schemes, etc.
S3: The paper is well-presented. It is easy to follow the logic flow of the paper and understand the proposed approach.
The paper has the following weaknesses:
W1: Although the explanation in Section 3.1 why DA approaches are more beneficial to KD losses than CE losses makes sense, it is not mathematically rigorous. It is not clear how to reason from a specific example of the advantage of using DA approaches in KD losses in Figure 2(a) to the hypotheses regarding the optimal number of training epochs in Equation (2). The last two inequations in Equation (2) are sort of known hypotheses: stronger DA approaches by design can be applied to a wide range of training losses including KD losses and are expected to outperform a standard DA approach regardless of the choice of training losses. These hypotheses are disconnected from the main purpose of the paper: as indicated by the title, the paper aims to explain why KD approaches are so successful. It is highly recommended that the paper adds more theoretical details of how interplay of KD losses and CE losses is related to the success of KD.
W2: I am a bit concerned about the technical depth of the paper.
Although overall the ideas make sense, these ideas are pretty straightforward. It is important for the paper to elaborate what is the challenge and how the proposed approach solves this challenge.
For example, it is not clear what is the challenge of adapting stronger DA approaches when using the KD losses and the adaptation method proposed in the paper seems to be straightforward.
W3: The related work can be improved. As mentioned in the paper, an effective idea for KD is to directly match response-based knowledge, feature-based knowledge, etc. between student models and teacher models [1]. To improve the process of transferring various types of knowledge between student models and teacher models, there are many algorithms like adversarial distillation [2], multi-teacher distillation [3], etc.
W4: The proposed approach does not outperform some baseline approaches in a few experimental settings.
For example, in Table 5, the baseline approach proposed by Hinton et al., 2014 performs the best on ImageNet under top-5 accuracy when using ResNet18 as student models. It would be better to explain the reason why the proposed approach under-performs the baseline approach in such experimental setting. The baseline approach by Hinton et al. is proposed around 7 years ago and it would make the conclusions more convincing if the paper could compare against more recent works [1].
[1] Knowledge Distillation: A Survey
[2] Kdgan: Knowledge distillation with generative adversarial networks
[3] Deep model compression: Distilling knowledge from noisy teachers
NO.
We greatly thank Reviewer 146T (R3)  for your constructive comments! Your concerns are addressed below.
`R3-Q1`: why DA approaches are more beneficial to KD losses than CE losses makes sense, it is not mathematically rigorous.  It is highly recommended that the paper adds more theoretical details of how interplay of KD losses and CE losses is related to the success of KD.
`R3-A1`: Admittedly, this paper is more of an empirical study than a mathematically proven theory. Thank R3 for letting us know about this problem! We shall make it more rigorous in our future version.
`R3-Q2`:  I am a bit concerned about the technical depth of the paper.
Although overall the ideas make sense, these ideas are pretty straightforward. It is important for the paper to elaborate what is the challenge and how the proposed approach solves this challenge.
For example, it is not clear what is the challenge of adapting stronger DA approaches when using the KD losses and the adaptation method proposed in the paper seems to be straightforward.
`R3-A2`: The proposed algorithm of TLCutMix is indeed not deep, while we take this simplicity as an *advantage* of our method instead of a weakness, as we show we can simply enhance the original KD loss function via a stronger DA scheme *without using any bells and whistles*. The performance improvement is non-trivial (e.g., on ImageNet, our KD+TLCutMix+pick_200 improves the original KD by over 1% top-1 accuracy). More importantly, the most valuable point of our paper, we think, is *not* one specific algorithm, but the generic framework to explain the synergistic interplay between KD and DA.
That is, we find a general way that can always enhance KD performance, as long as we find a stronger DA.
`R3-Q3`: The related work can be improved.
`R3-A3`: Thank R3 for pointing out the related works! We'll cite these papers in our new version.
`R3-Q4`: The proposed approach does not outperform some baseline approaches in a few experimental settings.
For example, in Table 5, the baseline approach proposed by Hinton et al., 2014 performs the best on ImageNet under top-5 accuracy when using ResNet18 as student models. It would be better to explain the reason why the proposed approach under-performs the baseline approach in such experimental setting. The baseline approach by Hinton et al. is proposed around 7 years ago and it would make the conclusions more convincing if the paper could compare against more recent works [1].
`R3-A4`: It is also pretty strange to us that our method has a better top-1 accuracy while left behind in the top-5 metric. Currectly we do not have a specific reason for this. We shall evaluate our method on more teacher-student pairs and see if we can have some insightful observations.
Indeed, the original KD method is pretty old.
But meanwhile, we can see this from a different perspective -- Equipped with our proposed DA scheme, the 7-year-old  baseline KD approach, now can beat CRD, which is one of the SOTA distillation methods. We believe this is a sign to show the effectiveness of our method.
But anyway, greatly thank R3 for letting us know your concern. We shall include more recent KD methods in our new version to evaluate our proposed DA scheme.
Hope our responses help resolve your concerns! *If you have any questions regarding our feedback, please let us know!*
As acknowledged by the author response, this paper could be much improved by adding rigorous explanations of how interplay of KD losses and CE losses is related to the success of KD, providing insightful observations why the proposed method has a better top-1 accuracy with a worse top-5 accuracy, etc.
Strength
- A new DA technique that is effectively applied to the existing state-of-the-art knowledge distillation technique is presented.
- Many experiments prove that the proposed method is effective.
- Overall, the readability of the paper is good
Weakness
- DA is generally a widely used technique to improve the performance of DNN, and the explanation of the principle of DA's performance improvement and its effect is somewhat trivial.
Therefore, contribution 1 claimed by the authors appears to be over-claimed.
- It is well known that DA differs in the degree of performance change depending on the learning configuration.
However, the authors show that the proposed DA strategy is effective for only one thing: CutMix.
Further analysis seems necessary.
- There are doubts as to whether the authors' DA strategy is an effective technique only for knowledge distillation, and it appears to be a general DA algorithm.
For example, it is necessary to check the degree of performance improvement when the student network learns the data generated by the proposed DA strategy.
- Since the proposed DA strategy generates twice as much data as a general data provider, even if it learns the same iteration, it can be considered unfair in terms of cost.
- Comparison techniques lack the latest techniques, and it seems necessary to reinforce them through sufficient surveys.
NO.
We greatly thank Reviewer T234  (R2) for your constructive comments! Your concerns are addressed below.
`R2-Q1`: DA is generally a widely used technique to improve the performance of DNN, and the explanation of the principle of DA's performance improvement and its effect is somewhat trivial.
Therefore, contribution 1 claimed by the authors appears to be over-claimed.
`R2-A1`: This paper focuses on the interplay between KD and DA, not DA itself.
Although the effectiveness of DA is well-known (actually we do not claim any contribution from the DA side alone), its interplay with KD has been less explored.
`R2-Q2`: It is well known that DA differs in the degree of performance change depending on the learning configuration.
However, the authors show that the proposed DA strategy is effective for only one thing: CutMix.
Further analysis seems necessary.
`R2-A2`: "However, the authors show that the proposed DA strategy is effective for only one thing: CutMix". With all due respect, this statement may not be true.
Apart from CutMix, we also show our method is effective on mixup, the standard DA scheme (flip and crop) in the main paper, also cutout (see our appendix, Tab. 10).
`R2-Q3`: There are doubts as to whether the authors' DA strategy is an effective technique only for knowledge distillation, and it appears to be a general DA algorithm.
For example, it is necessary to check the degree of performance improvement when the student network learns the data generated by the proposed DA strategy.
`R2-A3`: The proposed DA scheme is *not* a general DA algorithm as it demands the teacher to provide pseudo labels. This is why it is only discussed in the context of knowledge distillation instead of the general classification case. "it is necessary to check the degree of performance improvement when the student network learns the data generated by the proposed DA strategy" -- if not using the proposed DA scheme in a KD context, then the method degrades to the original CutMix (using the label assigned by the CutMix paper). Their performance has been shown in Tab. 1 (last row), where clearly, it underperforms our KD+TLCutMix, showing the merit of our proposed TLCutMix scheme.
`R2-Q4`: Since the proposed DA strategy generates twice as much data as a general data provider, even if it learns the same iteration, it can be considered unfair in terms of cost.
`R2-A4`: This is indeed a cost in our method.
However, by our observation, this only increases the training cost by around 30%, which is still bearable given we obtain a stronger performance.
`R2-Q5`: Comparison techniques lack the latest techniques, and it seems necessary to reinforce them through sufficient surveys.
`R2-A5`: Despite not being the latest, CRD is one of the SOTA methods as far as we know. In this regard, our method can beat it on ImageNet *simply using the original KD loss function*, showing the strong performance of our method.
Meanwhile, we shall include more SOTA KD methods to evaluate the proposed framework.
Hope our responses help resolve your concerns! *If you have any questions regarding our feedback, please let us know!*
Pros
* KD has many applications in different fields, but why it is successful is still an open research problem. The paper aims to investigate empirically and specifically the effects of DA on KD, which is a worthy problem.
* The advantage of using KD is obvious that the teacher will provide more information regarding the soft outputs, semantically similar classes that train the student better, and definitely with more examples, e.g., via data augmentation would be more useful.
* New DA method is proposed.
Cons
* The research problem is not well-stated.
* The experimental results are not sufficient to support the claims.
* The claim understanding the success of KD in DA perspective is not supported well in the paper.
* The hypothesis (Eq .2) needs more rigorous experiments to support.
NO.
We greatly thank Reviewer zS3x (R1) for your constructive comments! Your concerns are addressed below.
`R1-Q1`: The key concern about the paper is the question being tackled in the paper is unclear, and the lack of rigorous experimentation to support the hypothesis.
For example, the purpose of investigating interplay is not well-stated. Why is it important and useful?
Also, after reading the paper I do not understand the conclusion after investigating the interplay between KD and DA?
`R1-A1`: When we say "the success of knowledge distillation". Implicitly, we mean "the success of knowledge distillation" against the CE (cross-entropy) loss.
Then, previous works mainly think this is attributed to the "dark knowledge" hidden in the output classes.
Namely, they analyze it from the network *output* side, while in this paper, we analyze it from the *input* side, which states, KD loss can harvest more gains than CE loss from the *same* data augmentation.
That is, the success of KD is also partly attributed to the fact that *the KD loss is more effective in exploiting the data provided by a certain DA scheme*. To our best knowledge, this has not been proposed before. It is important because (1) we can better understand why KD loss performs better than CE loss (2) it can help us achieve stronger KD performance even not changing the loss function (which is also the reason it is *useful*).
`R1-Q2`: why DA improves KD are not clear.  We observe data augmentation can improve most training models, it is reasonable and not too surprising it will improve the student model.
`R1-A2`: The statements of "DA improves KD" or "DA improves the student model" are talking about the effectiveness of DA. They are well-known, admittedly (see page 2, "The first two observations are well-recognized by existing works (they simply reiterate the effectiveness of KD and DA, respectively"). This paper focuses on the *relative* advantage of KD over CE under the same DA. Especially, with a *stronger* DA, this relative advantage is *more* pronounced.
`R1-Q3`: Another concern about the performance is that the improvements are minor compared to the original KD and most of the experiments are less than 1%.
`R1-A3`: On CIFAR-100, our method is better than the KD counterpart (comparing "KD+TLCutMix+pick_960" vs "KD_960" or "KD+TLCutMix+pick" vs. "KD") by over 1% on *5/7 pairs*: ResNet32x4/ResNet8x4, VGG13/VGG8, VGG13/MobileV2, Res50/VGG8, ResNet32x4/ShuffleNetV2. R1's concern seems to rest on our ImageNet experiment, where KD is well-known not so effective itself.
Yet our method still improves KD under this hard case by around 0.4% top-1 accuracy, which is a non-trivial improvement on ImageNet.
Also, the ImageNet result is the *new SOTA* KD result simply using the original KD loss, to our best knowledge.
`R1-Q4`: By the way, not sure how the authors defined a “stronger” DA? It looks like the “stronger” one will give better accuracy to students which may be vague.
`R1-A4`: This is indeed an open question in our paper. Currently, it is defined by a stronger classification accuracy. We'll explore this further in our future work.
`R1-Q5`: Does the author consider the combination of multiple DAs together a “strong” DA?
`R1-A5`: Yes! This is definitely in our consideration. E.g., for the Fig.1, we show the combination of Flip+Crop, which is stronger than Flip alone, and delivers better KD performance as expected. How to combine even more DA schemes effectively falls into our future work considering there are so many DA schemes currently.
`R1-Q6`: In the framework, can the author explain the reason to keep x and x’? Does that help to boost the performance? As in Eq. 1 we already have the CE part using the original samples x?
`R1-A6`: We keep x for the reason explained on page 5: "The consideration of keeping both inputs is to maintain the information path for the original input x so that we can easily see how the added information path of x' leads to a difference." Although CE loss already includes x, yet it is for *CE* itself; x can be further exploited in the *KL divergence* loss term, thus intuitively keeping x is also beneficial.
In practice, we do see keeping x helps boost the performance.
`R1-Q7`: Many related state of the art data augmentations are missing.
`R1-A7`: Thank R1 for pointing out these related works. We shall cite them in our new version.
`R1-Q8`: the hyper parameter $\alpha$ may lead to different observations and behaviors. Would be good to study on that.
`R1-A8`: The alpha is inherited from the CRD paper, which is a pretty standard setting for KD. We will include study results on it in our new version.
Hope our responses help resolve your concerns! *If you have any questions regarding our feedback, please let us know!*

Meta-review:
role: PC
data: comment: All of the reviewers recommended rejecting this paper.
There were concerns that the underlying research questions being probed were not expressed clearly enough.
Reviewers were concerned that the experimental work was not sufficient to warrant acceptance.
Other concerns included the technical depth of the paper, the degree to which related work was discussed, placed in context and compared with empirically.
The AC recommends rejecting this paper.

Decision: Reject