Understanding the Success of Knowledge Distillation -- A Data Augmentation Perspective
Authors: Huan Wang
Suhas Lohit
Michael Jeffrey Jones
Yun Fu
Venue: ICLR 2022 Submitted (2022)
Keywords: knowledge distillation
data augmentation
mixup
cutmix
model compression
active learning

Abstract: Knowledge distillation (KD) is a general neural network training approach that uses a teacher model to guide a student model. Many works have explored the rationale for its success. However, its interplay with data augmentation (DA) has not been well understood so far. In this paper, we are motivated by an interesting observation in classification: KD loss can take more advantage of a DA method than cross-entropy loss \emph{simply by training for more iterations}. We present a generic framework to explain this interplay between KD and DA. Inspired by it, we enhance KD via stronger data augmentation schemes named TLmixup and TLCutMix. Furthermore, an even stronger and efficient DA approach is developed specifically for KD based on the idea of active learning. The findings and merits of our method are validated with extensive experiments on CIFAR-100, Tiny ImageNet, and ImageNet datasets. We achieve new state-of-the-art accuracy by using the original KD loss armed with stronger augmentation schemes, compared to existing state-of-the-art methods that employ more advanced distillation losses. We also show that, by combining our approaches with the advanced distillation losses, we can advance the state-of-the-art even further. In addition to very promising performance, this paper importantly sheds light on explaining the success of knowledge distillation. The interaction of KD and DA methods we have discovered can inspire more powerful KD algorithms. 

Under review as a conference paper at ICLR 2022
UNDERSTANDING THE SUCCESS OF KNOWLEDGE DIS- TILLATION - A DATA AUGMENTATION PERSPECTIVE E
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge distillation (KD) is a general neural network training approach that uses teacher model to guide student model. Many works have explored the rationale for its success. However, its interplay with data augmentation (DA) has not been well understood far. In this paper, we are motivated by an inter- esting observation in classification: KD loss can take more advantage of DA method than cross-entropy loss simply by training for more iterations. We present a generic framework to explain this interplay between KD and DA. Inspired by it, we enhance KD via stronger data augmentation schemes named TLmixup and TLCutMix. Furthermore, an even stronger and efficient DA approach is devel- oped specifically for KD based on the idea of active learning. The findings and merits of our method are validated with extensive experiments on CIFAR-100, Tiny ImageNet, and ImageNet datasets. We achieve new state-of-the-art accuracy by using the original KD loss armed with stronger augmentation schemes, com- pared to existing state-of-the-ar methods that employ more advanced distillation losses. We also show that, by combining our approaches with the advanced distil- lation losses, we can advance the state-of-the-art even further. In addition to very promising performance, this paper importantly sheds light on explaining the suc- cess of knowledge distillation. The interaction of KD and DA methods we have discovered can inspire more powerful KD algorithms.
INTRODUCTION
Deep neural networks (DNNs) are the best performing machine learning method in many fields of interest (LeCun al., 2015; Schmidhuber, 2015). How effectively train deep network for classification has been a central topic for decades. In the past several years, efforts have mainly focused on better architecture design (e.g., batch normalization (Ioffe & Szegedy, 2015), residual blocks (He et al., 2016), dense connections (Huang et al., 2017)) and better loss functions (e.g., label smoothing (Szegedy et al., 2016; Müller al., 2019), contrastive loss (Hinton, 2002), large- margin softmax (Liu et al., 2016)) than the standard cross-entropy (CE) loss. Knowledge distillation (KD) (Hinton et al., 2014) is a training framework that falls the second group. In KD, a stronger network called teacher - is introduced to guide the learning of the original network - called student by minimizing the discrepancy between the representations of the two networks.
LKD (1 - LcE(V) + ar ar" (p /t)p//////////) /T):
(I)
where DEL represents KL divergence (Kullback, 1997); E (0, 1) is a factor to balance the two loss terms; Leg denotes the cross-entropy loss; is the one-hot label and p(*) p(()) stands for the teacher's output and student's output, respectively (which are probability distributions over the
classes); is temperature constant (Hinton et al., 2014) to smooth predicted probabilities. KD allows us to train smaller, more efficient neural networks without compromising on accuracy, which facilitatees deploying deep learning in resource constrained environments (e.g., mobile devices). The effectiveness of KD has been seen in many tasks (Chen et al., 2017; Wang et al., 2020; Jiao et al., 2019; Wang & Yoon, 2021). Meanwhile, many works have investigated the reason behind its success, such as class similarity structure (Hinton et al., 2014) and regularization (Yuan et al., 2020). However, few works have paid attention its interplay with the input image data augmentation (DA), a technique to obtain more data through various transformations (Shorten & Khoshgoftaar, 2019). In this paper, will show that data augmentation is also important dimension to explain

---

Under review as a conference paper at ICLR 2022
the success of KD. Moreover, our findings show we can achieve much better performance simply using the original KD loss equipped with a stronger data augmentation scheme.
Our proposed algorithms are inspired from in- teresting observations shown in Fig. 1, where plot the student test enror curves when the model is trained for different numbers of epochs using KD loss vs. CE loss'. Three data
Without DA
Flip
Flip Crop
CE
CB
That crear nate (70)
28.0
augmentation scenarios are examined: not us-
30.0
ing DA at all (Without DA). only using the hor-
izontal flip (Flip), and using both the horizontal flip and random crop (Flip+Crop). We have the following observations. (1) Within each plot, KD loss delivers lower test error than CE loss. (2) When DA is used (comparing the middle or right plot to the left), both CE and KD curves improved. (3) When DA is used (compar- ing the middle or right plot to the left), the op- timal number of training epochs is postponed and the postponement is greater for KD than CE (the optimal number epochs is postponed from 180 to 480 for KD versu from 60 to 120 for CE). (4) When stronger DA employed (comparing the right plot to the middle), the optimal number of epochs is further postponed with even lower test error. The first two obser- vations are well-recognized by existing works (they simply reiterate the effectiveness of KD and DA. respectively), while the last two observations are new discoveries of this work, concerning the interaction between KD and DA. In other words, KD and DA, as two common techniques to im- prove the performance of DNNs, are actually not independent. This paper explains the interplay between KD and DA and leverages it for stronger KD performance using only the standard KD loss.
94.0
(126)
22.0
30.0
CE
240
450
720
M6D
200
450
720
150
240
450 720
950
Number of total training epoclas
Figure 1: Test error rate of ResNet20 on CIFAR- 100 when trained for different numbers of epochs (the teacher is ResNet56 for KD). Each result is obtained by averaging 3 random runs (shaded area indicates the std) "Flip" refers to horizontal flip; "Crop" refers to random crop. Both are standard data augmentation schemes in classification. The optimal number of training epochs and its test er- ror are highlighted in red.
Specifically, we explain why KD is able to exploit DA more than the CE loss. Owing to the random transformations in data augmentation, the input data are not fixed over different epochs. Different views of each image are presented over the training process. When KD loss is used, the teacher maps these different views to different targets. As illustrated in Fig. 2(a), these targets with different probability structures can reveal more information of the data, thus helping the student more. In contrast, when CE loss is adopted, the target is fixed regardless of the different views of the input. The extra information is thus lost. This observation inspired us to develop two stronger data aug- mentation techniques (TLmixup, TLCutmix) that are tailored for KD. We further tap into the idea of active learning to make TLCutmix even better, that is, our TLCutmix+pick method.
In summary, these are the main contributions of this work:
We make novel observations of the interaction between KD and DA. We explain how DA methods are more suited to be exploited by KD which uses teacher outputs as labels instead of the ground-truie -ruth one-hot labels used by CE loss.
Inspired by the above, we propose to enhance the original KD loss with stronger data augmentation schemes (by adapting mixup (Zhang et al., 2018) and CutMix (Yun et al., 2019) to KD). It is shown that these methods are more reasonably applied in the KD case than in the CE case.
We further propose an even stronger data augmentation method specifically for knowledge distillation using the idea of active learning (Settles, 2011).
We show empirically better results simply by using the original KD loss combined with the proposed DA scheme, compared to state-of-the-ar KD methods, which adopt more advanced distillation losses.
Notee data points one curve in Fig. 1 are from independent experiments of different total number of epocks; the learning rate schedule proportionally scaled based on the total numher of epochs.
2

---

Under review as a conference paper at ICLR 2022
turget (KD)
Target (CF)
Stundard DA
00.13.002...7000.55
10,0,
Teacher (fixed)
0:05, 0.600
10,0,
Raw inpur
Input
KD loss
Inpui
[0,05, 0.22, 0.55 10.0,1.00
Stronger DA
Student
Different of the input
(a)
mapped Seferent biggds
(b)
Input
Figure 2: Interplay between knowledge distillation (KD) and data augmentation (DA). (a) Illustra- tion of the difference of supervised target between the KD loss and cross-entropy (CE) loss. An input is transformed to different versions (called views in this paper) owing to data augmentation. KD loss provide extra information the student by mapping these views to different targets, while the CE loss cannot. (b) Illustration of KD with the proposed data augmentation framework. The standard DA consists of random crop and horizontal flip. The stronger DA refers to any data augmentation scheme more advanced than the standard one. In this paper, we propose three stronger DA schemes: TLmixup, T.CCutMxx TI.CutMix+pick (see Sec. 3.2 for details).
RELATED WORK
Knowledge distillation: The general idea of knowledge distillation is to guide the training of student model through a stronger pretrained teacher model (or an ensemble of models). It was pioncered by Buciluă et al. (Buciluă et al., 2006) and later refined by Hinton et al. (Hinton al., 2014), who coined the term. Since its debut, knowledge distillation has seen extensive application in vision and language tasks (Chen al., 2017: Wang et al., 2020; Jiao et al., 2019; Wang & Yoon, 2021). Many variants have been proposed regarding the central question in knowledge distillation, that is, how to define the knowledge that is supposed to be transferred from the teacher to the student. Examples of such knowledge definitions include feature distance (Romero et al., 2015), feature map attention (Zagoruyko & Komodakis, 2017). feature distribution (Passalis & Tefas, 2018), activation boundary (Heo et al., 2019), inter-sample distance relationship (Park et al., 2019; Peng et al., 2019; Liu et al., 2019; Tung & Mori, 2019), and mutual information (Tian et al., 2020). Over the past several years, the progress has been made primarily at the output end (i.e., through a better loss function). contrast to previous works, our goal in this paper is improve the KD performance at the input end with the help of data augmentation. We will show this path is as effective and also has much potential for future work.
Deep neural networks are prone to overfitting, i.e., building input-target con- nection using undesirable or irrelevant features (like noise) the data. Data augmentation is a pre- vailing technique to curb overfitting (Shorten & Khoshgoftaar, 2019). In classification tasks, data augmentation aims to explicitly provide data with label-invariant transformations (such as random crop, horizontal flip, color jittering) in the training so that the model can learn representations bust to those nuisance factors. Recently, some advanced data augmentation methods were proposed, which not only transform the input, but also transform the target based certain corresponding relations. For example, mixup (Zhang et al., 2018) linearly mixes two images with the labels mixed by the same linear interpolation: manifold mixup (Verma et al.. 2019) is similar to mixup but con- ducts the mix operation in the feature level instead of pixel level; CutMix (Yun et al., 2019) pastes patch cut from an image onto another image with the label decided by the area ratio of the two parts. Now that the input and target are transformed simultaneously. the key is to maintain a semantic correspondence between the new input and new target. Although these methods have been proven effective, one lingering concern is about the reasoning behind them. Specifically, it is easy to come up with examples where the semantic correspondence is poorly kept (see Fig. 5 for examples on CutMix). Unlike these methods, which focus general classification using the cross-entropy loss, our work investigates the interplay between data augmentation and knowledge distillation loss and the proposed data augmentation specifically for knowledge distillation.
Data augmentation:
3

---

Under review as a conference paper at ICLR 2022
One recent work (Das et al., 2020) also conducts empirical study about the impact of data augmen- tation on knowledge distillation. However, their exploration is very different from ours: they first apply data augmentation (e.g., mixup/CuttMix) to the teacher then conduct the distillation step as usual (no extra data augmentation in this step); our investigation is the exact opposite to their setup: train the teacher usual (not applying mixup/CutMix), then in the distillation step we employ a more advanced data augmentation (e.g., mixup/CutMixx. Interestingly, they conclude that the teacher trained with mixup/CutMix hurts the student's generalization ability, while we consistently see a performance boost by using a stronger DA in the distillation step during student training.
3 PROPOSED METHOD
3.1 INTERPLAY OF KD, DA, AND TRAINING ITERATIONS
We first introduce a framework to explain the phenomenon that KD can exploit DA more simply by training for more iterations. Training for more iterations means presenting more examples to the network. Over iterative training. the presented examples are not exactly the same among different epochs because of the random transformations of data augmentation. Different versions of an image produced by data augmentation can be regarded as multi-views of that image (Wu et al., 2018; Tian et al., 2020). We term this kind of data difference Fig. 2(a), when the CE loss is employed, different views of an image are mapped to a single point in the target space (the hard label). In contrast, when the KD loss is used, different views of the data are mapped to a group of points in the target space through the teacher, which can reveal richer information around that class. By richer, we specifically mean two sources. First the class structure information provided by the soft labels instead of the hard labels, i.e., the well-known dark knowledge (Hinton et al.. 2014). Second the information provided by the different input views from data augmentation.
input view diversity. As depicted in
Concretely, in Fig. 2(a), although the three views of the input share the same main class "dog", the target probability vectors are different: compared with the first view, the second one has more lawn in it, thus the teacher has larger predicted probability in the "lawn" class; similarly, the third view has more sunshine, thus larger probability in the "sunset" class. These subtle changes in class- wise probabilities are beneficial for the student to learn. However, the CE loss is used, all the three views are mapped to the same one-hot label, not putting the extra information to good use. More training iterations keep producing new data views to the student, making it afford more training epochs without overfitting. In contrast, for CE, the target is fixed for different views, little new information is added in longer training. Thus, the student can only afford fewer extra epochs.
Denote the optimal number of training epochs as N, lowest test error E. The synergistic interplay between KD and DA can be summarized in the following hypotheses.
(-DA) - (-wl DA) > NOE - Nr(wl DA) KD
KD
CE
NDDA*)
NADA and E(A*') < EKD
where "DA" refers to a data augmentation method; "DA+" refers to another data augmentation method stronger than "DA"; "w/o DA" means not using any data augmentation. These inequalities are empirically verified in our experiments (Fig. 1, Fig. 3). The first inequality suggests the ad- vantage of KD over CE: given the same DA scheme, KD loss can make more use of extra training iterations. The second suggests we can obtain better accuracy simply by training for more epochs using a stronger DA method. This leads us to investigate stronger DA methods for KD, as follows.
3.2 PROPOSED ALGORITHMS FOR IMPROVED KD
(1) KD-TLmmxxup/LLCuMMi We continue exploration with two existing data augmentation techniques that are more advanced than the standard random crop and flip: mixup (Zhang et al., 2018) and CutMix (Yun al., 2019). They are initially proposed in the CE case. Here upgrade them for KD, resulting in TLmixup and TLCurMix (TL is short for teacher-labeled).
Specifically, let zo denote the data, denote the transformed data by the standard augmentation (random crop and flip). Illustrated in Fig. 2(b). we propose to add mixup/CutMix following x to obtain x'. Unlike the common data augmentation where only the transformed input is fed into the
4

---

Under review as a conference paper at ICLR 2022
Wilnom DA
Flip
Flip ClpppCopp
Wilhon DA
Flip
Flip-OCop
17.5
AMA 50
12.30
22..0
100, (0:00)
310
25.02
CP
20.70 700
27.5
CF
XD
Dulo 0.0
Number troining spochs
Number tromiing spochs
(a) WRN-1662 (teacher: WRN-40-2)
(b) VGG8 (teacher: VGG13)
Figure 3: Test error rate of WRN-16-2 and VGG8 on CIFAR-100 when trained for different numbers of epochs, using KD or cross-entropy (CE) loss, with or without data augmentation (DA). Every error rate averaged by 3 random runs (shaded area indicates the stddev). Consistent with Fig. 1, when DA is used, the optimal number of epochs is postponed and postponed more for KD than CE. When a stronger DA is used, the optimal number of epochs is postponed even more with smaller optimal test error.
network, we keep both the input and x' for the training (as such, the number of input examples during training is increased). The consideration of keeping both inputs is to maintain the information path for the original input that can easily see how the added information path of x' leads to a difference.
For æ, its loss is still the original KD loss, consisting of the cross-entropy loss and the KL divergence (Eq. 1). Of special note is that, for x', its loss is only the KL divergence, i.e., we do not use the labels assigned by mixup or CulMix because they can be misleading and do not perform well as we will show later (Tab. 1. Fig. 5). In fact, not using the hard label has another bonus. A dataset augmenta- tion scheme which employs CE loss has provide corresponding labels as supervisory information. In order to maintain the semantic correspondence, it cannot admit very extreme transformations for data augmentation.
contrast, in the mixup/CutMixx+B mentation scheme need not worry about the labels as they are assigned by the teacher. Therefore, it admits a broader set of transformations to expose the teacher's knowledge more completely.
setting described above, the data aug-
Between TLmixup and TLCutMix, we will empirically show TLCutMix is more favorable (Tab. 1) (and both of them are significantly better than the standard augmentation, random crop and flip). Therefore, choose TLCutMix as base to develop our next algorithm as follows.
(2) KD+TLCutMixxppic. Our next algorithm is an even stronger DA scheme tailored to KD, based on the idea of active learning (Settles, 2011). In active learning, the learner enjoys the freedom to query the data instances to be labeled for training by an oracle (i.e., the teacher in our case) (Settles, 2011). Since the augmented data vary in their quality, we can introduce certain criterion to pick the more valuable data for the student. We tap into the idea of hard examples (Micaelli & Storkey, 2019) define the criterion. Specifically, we measure the hardness by the KL divergence between the teacher's ' output and the student's 's output (p(*) and p(*) are the teacher and student output probabilities over classes),
d= /T.
We sort the augmented samples by their d's in ascending order and pick a subset with the largest d's. Notably, the criterion d has exactly the same form that the student is supposed to minimize in This design makes an adaptive competition:
Eq. (I); while here we pick samples to maximize when the student is updated, the criterion made of the KL divergence will also be updated. Each time, it makes sure the hardest samples are selected for the student.
Other common choices for the criterion include the teacher's entropy the student's 's entropy (larger entropy implying more uncertainty meaning the sample is harder). They only take into account one-side information, either the teacher's student's. Conceivably, they are not good as the KL divergence criterion, which considers the information from both sides. This choice will be empirically justified in experiments (Tab. 2).
5

---

Under review as a conference paper at ICLR 2022
Table 1: KD test accuracy comparison on CIFAR-100 when using different DA schemes. Each experiment is run times and the mean and (standard deviation) are reported. De- fault DA: random crop and flip. Note (I) KD+CutMix is much worse than KD alone Res56/Res20; (2) KD+TLCutMix consistently
Table 2: KD test accuracy compari- son on CIFAR-100 when using different data picking schemes for the proposed "KD+TLCCuMixx<pic"" method. "T/S" refers to teacher/student, "ent." is short for entropy, and "kid" stands for KL divergence. The mean and (std) of 3 runs reported for each entry.
outperforms KD+CutMix on all the 3 pairs.
Teacher
WRN-40-2
ResNer56 ResNer20
VGG13
Teacher Student
WRN-40-2
ResNet56 ResNer20
VGG13
Student
WRN-16-2
VGG8
WRN-16-2
VGG8
Teacher Acc.
75.61
72.34
74.64
Teacher Ace.
75.61
72.34
74.64
Student Acc.
73.26
69.06
70.36
Student Acc.
73.26
69.06
70.36
KD (default DA)
74.92 (0.28) 70.66 (U.24) 72.98 (0.19)
TLCutMix
75.34 (0.19)
70.77 (0.17) 74,16 (0.18)
KD+TLmixxu
25.33 (0.07) 71.00 (0.16) 73.79 (0.18)
+Pick (T ent.)
775.46 (0.07)
20,88 (0.10)
74.16 (4.18(00118)
KD+TLCuuMix 75.34 (0.19) 70.77 (0.17) 74.16 (0.18)
+Pick (S ent.)
75.52 (0.06) 70.84 (0.12) 74.16 (0.48)
KD+CutMix
75.25 (0.13) 69.76 (0.19) 73.75 (0.16)
+Pick (TVS kid)
75.59 (0.22) 70.99 (0.20) 74,43 (0.20)
EXPERIMENTAL RESULTS
Datasets and networks. We evaluate our method on the CIFAR-100 (Krizhevsky, 2009), Tiny ImageNet', and ImageNet (Deng et al., 2009) object recognition datasets. CIFAR-100 has 100 object classes (32x32 RGB images). Each class has 500 images for training and 100 images for testing. ImageNet is now the standard large-scale benchmark dataset in image classification, which has 1000 classes (224x 2244 RGB images), over 1.2 million images in total. Tiny ImageNet is a small version of ImageNet with 200 classes (64 x 64 RGB images). Each class has 500 images for training, 50 for validation and 50 for testing. To thoroughly evaluate our methods, we benchmark them on various standard network architectures: VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016), WRN (Wide-ReNNet) (Zagoruyko & Komodakis, 2016), MobileNetV2 (Sandler et al., 2018), ShuffleNetV2 (Ma al., 2018). Our code and trained models will be made publicly available.
Evaluated methods. In addition to the standard cross-entropy training and the original KD method (Hinton et al., 2014), we also compare with the state-of-the-ar distillation approach Con- trastive Representation Distillation (CRD) (Tian et al., 2020). It is important to note that our method focuses on improving KD by using better inputs, while CRD improves KD at the output end (i.e. a better loss function). Therefore, they are orthogonal and we will show they can be combined together to deliver even better results.
Hyperparameter settings. The temperature 7 of knowledge distillation is set to 4. Loss weight = 0.9 (Eq. equation 1). (1) For CIFAR-100 and Tiny ImageNet, training batch size is 64; the original number of total training epochs is 240, with learning rate decayed at epoch 150, 180, and 210 by multiplier 1/10. The initial Ir 0.05. (2) For ImageNet, training batch size is 256; the original number of training epochs is 100, with learning rate decayed at epoch 30, 60, 90. The initial learning rate is 0.1. All these settings are the same as CRD (Tian et al., 2020) for fair comparison with it. Note, in our experiments of total epochs is scaled by a factor k the epochs after which learning rate is decayed is also be scaled by k. For example, if we train network for CIFAR-100 --0 for 480 epochs (k: = in total, the epochs after which the learning rate is decayed will be 300, 360, and 420.
will show the results of more training iterations. If the number
We use PyTorch (Paszke et al., 2019) conduct all our experiments. For CIFAR-100, we adopt the pretrained teacher models from CRD https://gitubb.co/HHbbbb http://...ooooobooooooogeeiiiiie Long/RepDisttile)) for fair compar- ison with it. For Tiny ImageNet, we train our own teacher models. For ImageNet, we adopt the standard torchvision models as teachers.
4.1 CIFAR-100
Effect of more training iterations. In Sec. 1, we presented Fig. 1 to show the advantage of KD loss over CE loss in exploiting extra epochs. Here we show more results in Fig. 3 on different network architectures to confirm the finding is general. In line with the ResNet case (Fig. 1), extra training

---

Under review as a conference paper at ICLR 2022
Table 3: Student test accuracy comparison CIFAR-100. Each result is obtained by 3 random runs, mean (std) accuracy reported. The best results are in bold and second best underlined. The subscript 960 means the total number of training epochs (default: 240).
Teacher
WRN-40-2
ResNet56 ResNe120
ResNer32x4
VGG13
VGG13 MobileNetV2
ResNer50 VGG8
ResNer32x4
Student
WRN-16-2
ResNet8x4
VGG8
ShufleNetV2
Teacher Ace.
75.61
72.34
79.42
74.64
74.64
79.34
79.42
Student Ace.
73.26
69.06
72.50
70.36
64.60
70.36
71.82
KD (Hinton et al., 2014)
74.92 (0.28) 70.66 (0.24) 73.33 (0.25) 72.98 (0.19)
67.37 (0.32)
73.81 (0.13) 74.45 (0.27)
KD960 (Hinton al., 2014)
75.68 (0.12) 71.79 (0.29) 73.14 (0.06) 74.00 (0.34) 68.77 (0.05) 74.04 (0.25) 74.64 (0.30)
KD+TLCutMix
75.34 (0.19) 70.77 (0.17) 74.91 (0.20) 74.16 (0.18) 68.79 (0.35) 74.85 (0.23) 76.61 (0.18)
KD+TL.CutMix-pickk
75.59 (0.22) 70.99 (0.20) 74.78 (0.35) 74.43 (0.20) 69.49 (0.32) 74.95 (0.18) 76.90 (0.25)
XD+TLCutMix+ppiceaa
76.41 (0.10) 71.66 (0. 15) 75.12 (0.18) 75.00 (0. (0.17) 70.47 (0.12) 76.13 (0.16) 77.90 (0.30)
C et al., 2020)
75.64 (0.21) 71.63 (0.15) 15) 75.46 (0.25) 74.29 (0.12) 69.94 (0.05) 74.58 (0.27) 76.05 (0.09)
CRD+-I.CutMixx-icc
75.96 (0.27) 71.41 (0.26) 76. (0.53) 74.65 (0.12) 69.95 (0.22) 75.35 (0.22)
76.93 (0.11)
CRD-TI.CutMix-piickon
76.61 (0.01) 72.40 (0.20) 75.96 (0.29) 75.41 (0.10) 70.84 (0.05) 76.20 (0.22) 78.51 (0.27)
Table 4: Student test accuracy comparison on Tiny ImageNet. Each result is obtained by 3 random runs, mean (std) accuracy reported. The best results are in bold and second best underlined. The subscript 480 means the total number of training epochs (default: 480).
Teacher
WRN-40-2
ResNet56 ResNer20
ResNer32x4
VGG13
VGG13 MobileNetV2
ResNer50 VGG8
ResNet32x4 ShuffleNerV2
Student
WRN-16-2
ResNet8xd
VGG8
Teacher Acc. Student Acc.
61.28
58.37
64.41
62.59
62.59
68.20
64.41
58.23
52.53
55.41
56.67
58.20
56.67
62.07
KD (Hinton et al., 2014)
58.65 (0.09) 53.58 (0.18) 55.67 (0.09) 61.48
(0.36)
59.28 (0.13) 60.39 (0.16) 66.34 (0.11)
KDaso (Hinton al., 2014)
59.20 (0.30) 54.23 (0.24) 55.49 (0.11) 61.72 (0.10)
59.27 (0.08) 60.10 (0.30) 65,81 (0.11)
KD+TLCutMix
59.06 (8)18) 53.77 (0.33) 56.41 (0.04) 62.17 (0.11) 60.48 (0.30) 61.12 (0.18) 67.01 (0.30)
59.22 (0.05) 53.66 (0.05) 86.82 (0.23) 62.32 (0.18) 60.53 (0.18) 61.40 (0.26) 67.08 (U.13)
UD+TI.CutMix++pickkss
60.07 (0.04) 54.25 (0.07) $7.54 (0.23) 62.60 (0.25) 60.66 (0.15) 61.95 (0.14) 67.35 (0.21)
CRD (Tian et al., 2020)
60.79 (0.24) 55.34 (0.02) 59.28 (0.13) 62.92 (0.31) 62.38 (0.19) 62.03 (0.16) 67.33 (0.13)
CRD+TT.CCtMixx-picc
60.72 (0.09) 54.99 (0.16) 59.65 (0.24) 63.39 (0.10)
62.54 (0.22) 62.85 (0.18) 67.64 (0.18)
CRD+xICCuMixxxpick..
60.99 (0.33) 55.68
(0.22) 60.13 (0.13) 63.60
(0.20)
62.79
(0.03) 62.60 (0.17) 67.70 (0.35)
also brings more performance gains with KD loss on WRN and VGG. The gains are more or less up to the particular pairs but the trends are consistent: When DA is used, the optimal number of training epochs is higher and even more so for KD than CE; when stronger DA is employed, the optimal number of epochs is further significantly higher and produces lower test error. These results support the proposed hypotheses in Eq. (2).
Exploring different data augmentation schemes. In Tab. I we compare three different DA schemes on CIFAR- 100: the default, TLmixup, and TLCutMix. It has been shown in the origi- nal papers of mixup and CutMix that they improve accuracy over the standard data augmentation in the CE case. However, it does not mean naively combining KD and CuMixx/mxxp as it is can always bring performance improvement. As seen, CutMix is actually at odds with KD on the ResNec56/RRssee)) pair while our TLCutMix consistently improves the performance on all three pairs. On the other pairs, original CutMix is also not as effective as our adapted TLCutMix. These confirm that using the teacher's 's output for distillation for the augmented data is critical. TLCutMix better than TLmixup in general, so we choose it as base to develop TLCutMixxpick.
Exploring different data picking schemes. In Tab. 2, we compare the three potential schemes of selecting more informative data for the student: entropy of the teacher's output ("T ent."), entropy of the student output ("S ent."), and the KL divergence between the teacher's and student's outputs ("T/S kld"). As shown, the KI. divergence scheme performs best. This is expected as either the teacher entropy student entropy alone does not reveal the whole picture.
Benchmark on 100. The results are shown in Tab. 3. We have the following observations. (1) KD can be improved by training for more iterations (960 epochs vs. 240), owing to the effect of data augmentation (only one exception is ResNeet32x4Reesettxx).... This is not true for CE alone. This is a novel observation which shows the optimal number of training epochs for KD w/ DA is significantly different from that of CE (2) Comparing the row "KD+TLCutMix" to "KD", we see the proposed TI.CutMix scheme improves the accuracies of all teacher-student pairs. On 5 out of
7

---

Under review as a conference paper at ICLR 2022
the 7 pairs, the improvement is very significant (more than 1 percentage point). (3) Comparing the row "XD+TTCCuuMix+picc"" to "KD+TLCutMix", we see out of the 7 pairs are improved further, showing the proposed data picking scheme works in most cases. (4) Finally, *KD+TLCuuMix+pick scheme can be combined with more training iterations, which delivers even higher accuracies. (5) If comparing our best results TDD+TT.CutMixxpiccssa with those of CRD (though this is not an apples-to-apples comparison since the two methods focus on different aspects to improve KD), we can see our approach outperforms CRD on 6 out of the 7 pairs.
worth emphasizing that we achieve this simply using the original KD loss (Hinton et al., 2014), with no bells and whistles. This justifies one of our motivations in this paper, i.e., existing KD methods (Peng et al., 2019; Park al., 2019; Tian et al., 2020) mainly improve KD at the oulpul layer through better loss functions, while we propose to improve KD at the inpul end and show this path is just as promising.
In the last two rows of Fig. 3, when CRD (Tian et al., 2020), the state-of-the-an KD algorithm, is armed with our proposed "TT. CutMix+pick" and more training iterations, its results can be further advanced consistently. This demonstrates that the proposed schemes work with those methods focusing on better KD loss functions, In the Appendix Tab. 9, we present the results of applying TLCutMix to another 5 KD methods. All of the evaluated pairs see accuracy gains; half of them are even improved by more than 1% point.
general and can readily
Further remarks. Observation (1) above has another implication to the community in addition to improving the performance of KD. It tells us that the number of training iterations can have a big impact on the performance of KD method. Unaware of this issue, if authors of KD paper compare their method to others by directly citing numbers from other papers and the training epochs happen to be different, then the comparison may well be unintentionally unfair from the beginning.
4.2 TINY IMAGENET
In this section evaluate the proposed schemes on more challenging dataset- Tiny ImageNet. Similar to the case on CIFAR-100, we have results on different teacher-student pairs, shown in Tab. 4. For more training iterations, we train for 480 epochs instead of 960 to save time. Most claims on the CIFAR-100 dataset are also validated here: (1) "KD+TLCutMix" is better than KD, is better than "KD+TLCutMix", verified on 6 pairs. The exception pair is ResNet566ResSee200, where adding data picking decreases the accuracy slightly by 0. 11%. (3) When "KD=+T.CutMix+pick is equipped with more training it- erations, we obtain the best performance. The main difference from CIFAR-100 results lies in the comparison between "KD4s0" " and "KD". In the CIFAR- 100 case with standard augmentation, more training iterations consistently improves the accuracy on 6 pairs, while here only 3 are improved. We believe this because the standard DA scheme random crops and horizontal flips cannot pro- duce diverse enough data on this challenging dataset (Tiny ImageNet vs. CIFAR-100). In contrast, using the stronger DA scheme (TLCutMix+pick) and more training iterations does show significant improvement in all 7 out of 7 cases.
which is verified on all pairs. (2)
We also evaluate the compatibility of our DA methods with the state-of-the-an CRD, shown in the last two rows of Tab. 4. Our "TLCutMix+pick" method further advances the prior sfate-of-the- art on 5 pairs. When CRD+TI CuttMiiiiMi is trained for 480 epochs (instead of 240), further improvement can be seen 6 of 7 pairs.
4.3 IMAGENET
We further evaluate our methods on the ImageNet dataset, shown in Tab. 5. "KD+TI.CutMix" improves original KD (from 70.66 to 71.05 top-1 accuracy). When data picking added, it does not help here. Possible reasons will be analyzed later. When the student is trained for 200 epochs with KD=TLCuuMix+pick, it delivers the new state-of-the-art top-1 performance.
We also present the result of original KD trained for 200 epochs. Interestingly, it matches the previous ofate-of-the-an method CRD and beats many other KD methods without any additional loss terms. However, this is not an aaples-to-apple comparison, as these methods are trained for 100 epochs. Yet it is clear indication that the interplay between KD and DA is useful even on a large-scale dataset.
8

---

Under review as a conference paper at ICLR 2022
CALE BestiTook
Table 5: Top-1 and Top-5 accuracy (%) of the student ResNet18 on ImageNet validation set. The subscript 200 indicates the total number of training epochs is 200 (the original one is 100).
Acerngg divergeene mito
1.45
1.46
1.88
Top-1 acc.
Top-5 acc.
Teacher (ResNet34)
73,31
91.42
1.30
Student (ResNet18)
69.75
89.97
KD (Hinton al., 2014)
70.66
89.88
SP (Tung & Mori, 2019)
70.62
89.80
0M 1122
9.5
0.55
1.0
Ronation
AT (Zagoruyko & Komodakis, 2017)
70.70
90.00
Figure Mean KL divergence ratio r (Eq. (4))
CRD (Tian
2020)
71.38
90.49
KDau (Hinton et al., 2014)
71.38
90.59
over iterations different datasets. The itera-
KD+TLCutMix
71.05
90.36
tions are normalized into range [0, 1] for casy
<D-rTI.CutMix+xiick
70.78
90.04
comparison since the total numbers of iterations are different on the 3 datasets.
71.76
90.58
Cross-dataset analysis. Here we investigate how the proposed method of KD+TT.Cuuii--pic is affected by the size and nature of the dataset. The resnet teacher-student pairs of Res56/Res20 and Res34/Res 18 are of particular interest as the boost in performance for these pairs are lower than other network architectures.
The picking scheme is proposed based on the idea of active learning (Sec. 3.2). Intuitively, it can work only if the picked data has more information to the student network than those randomly presented. Since we adopt the KL. divergence between the teacher's output and the student's output measure the amount of information in the input data, we can compare this metric two different sets of data, ie., picked randomly vs. picked based on KI. divergence. Specifically, we define average KI. divergence ratio
EN'' de Nr E dz
T
(4)
where d; stands for the KL divergence for the i-th sample defined in Eq. (3); N denotes the number of total samples in batch; Np denotes the number of sample picked based on KI. divergence (N, = NN2 in our experiments); note that r > 1. Larger r means the picked samples have more information than the average samples. Then we compare on different datasets over the training process of "KD+TL CuttxxxcIMi Results are shown in Fig. 4. As seen, in terms of r, CIFAR- average; meanwhile, comparing the results of CIFAR-100 (Tab. 3). Tiny ImageNet (Tab. 4), and ImageNet (Tab. 5) we see the accuracy gains brought by data picking also show the same trend of CIFAR- 100 > Tiny ImageNet >> ImageNet, in accordance with our expectation. This validates the soundness the metric r we introduced. The r on ImageNet is clearly lower than the other two, meaning there is significant information difference between the picked data and the average data, which may well explain the scheme on ImageNet. Note that the root cause of this problem actually lies in the data augmentation part since it cannot produce more informative samples, the subsequent data-picking has no scope to expand its value. How to obtain an even stronger scheme than TLCutMix remains elusive for now, which we will investigate as part of our future work. Also note, Res36/Res20 delivers the lowest among the three pairs. This likely explains why the picking scheme especially not effective on the original resnet pairs (Res56/Res20, Res34/Res18)
100 Tiny ImageNet >> ImageNet
of the data picking
5 CONCLUSION
We carefully investigate the interplay between knowledge distillation (KD) and data augmentation (DA) in this paper. Unlike the cross-entropy loss, KD can exploit DA by training for more epochs. The proposed input view diversity framework explains the interplay well and inspires us to de- velop three new data augmentation methods specifically for KD. Extensive experiments demonstrate the merits of our methods across various networks on CIFAR-100, Tiny ImageNet, and ImageNet datasets. Our method achieves the new state-of-the-an using only the vanilla KD loss with no bells and whistles, showing the potential of improving KD from the input side rather than better KD loss function. Our paper can also help the community build a more standard benchmark of KD algorithms, paying particular attention to the DA schemes and number of training epochs.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In CVPR, 2019. 12, 13
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil, Model compression. SIGKDD, 2006. 3
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. NeurIPS, 2017. 1,3
Deepan Das, Haley Massa, Abhimanyu Kulkarni, and Theodoros Rekatsinas. An empirical analysis of the impact of data augmentation on knowledge distillation. arXiv preprint arXiv:2006.03810, 2020. 4
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Terrance DrVries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017. 13
K. He, X. Zhang, Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016. 1,6
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. AAAI, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. In NeurlPS Workshop, 2014. 1, 3, 4, 6, 7, 8, 9,9,11 13
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771-1800, 2002.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In CVPR, 2017. 1
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 10351, 2019. 3,
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. 6
Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 521(7553):436, 2015.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolu- tional neural networks. ICML, 2016.
Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In CVPR, 2019.
Ningning Ma. Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, 2018. 6
Paul Micaelli and Amos Storkey. Zero-shot knowledge transfer via adversarial belief matching. arXiv preprint 1905.099990963 2019. 5
Rafael Müller, Simon Korbblith, and Geoffrey E Hinton. When does label smoothing help? In NeurIPS, 2019.
10

---

Under review as a conference paper at ICLR 2022
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019. 3, 8
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018. 3, 12, 13
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 2019. 6
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In ICCV, 2019. 3, 8, 12, 13
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICI.R, 2015. 3
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85-117, 2015.
Burr Settles. From theories to queries: Active learning in practice. In AISTATS Workshop on Active Learning and Experimental Design, 2011. 2,
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):60, 2019. 1,3
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016. 1
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020. 3, 4, 6, 7, 8, 9, 13
Frederick Tung and Greg Mori. Similarity-preservin knowledge distillation. CVPR, 2019. 3, 9, 12, 13
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019. 3
Huan Wang, Yijun Li, Yuehai Wang, Haoji Hu, and Ming-Hsuan Yang. Collaborative distillation for ultra-resolution universal style transfer. In CVPR, 2020. 1,
Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual intelli- gence: A review and new outlooks. TPAMI, 2021. 1,3
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, 2018.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In CVPR, 2020. 1
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In ICCV, 2019. 2, 3, 4
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016. 6
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In ICLR, 2017. 3, 9, 12, 13
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018. 2.
11

---

Under review as a conference paper at ICLR 2022
A NETWORK PARAMETERS AND FLOPS
The number of parameters and FLOPs (FLoating-poin OPerations) of each model on the CIFAR- 100, Tiny ImageNet, and ImageNet datasets are presented in Tab. 6, Tab. and Tab. respec- tively. For the number of parameters (or FLOPs), we only count them in the convolutional and fully-comnected layers (BN layers not included) following the common practice.
Training cost of TLCutMix+iick. The proposed picking scheme only increases the wall-clock training time by around 30%. which is reasonable considering the performance gains.
Table 6: Model complexity statisties of each teacher-student pair on the CIFAR-100 dataset. The number of FLOPs is measured by the number of Multiply and Add operations in convolutional and fully-connected layers. Input image size: 32x32x3.
Teacher
WRN-40-2 ResNet56 ResNer32x4
VGG13
VGG13
ResNet50 VGG8
ResNet32x4
Student
WRN-16-2 ResNet20 ResNet8x4
VGG8
MobileNetV2
ShuffteNerV2
Teacher #Params (10%)
2.2497
0.8574
7.4239
9,4563
9,4563
23.6521
7.4239
Student #Params (10%)
0.7015
0.2768
1,2308
3,9621
0.7945
3.9621
1.3393
Compression ratio
3,2072x
3.0979x
6.0319x
2.3867 x
11.9027
5,9696
5,5430x
Teacher #FLLPs (10")
0,6552
0,2515
2.1661
0.5699
0.5699
2.5960
2.1661
Student #FLOPs (109)
0,2022
0,0816
0,3541
0,1924
0.0116
0.1924
0.0863
Speedup ratio
3,2399
3.0808x
6.1164x
2.9621
49.1399x
13,4939
25.0915x
Table Model complexity statistics of each teacher- sttuent pair on the Tiny ImageNet dataset. The number of FLOPs is measured by the number of Multiply and Add operations in convolutional and fully-connected layers. Input image size: 64x64x x.3.
Teacher
WRN-40-2 ResNet56 ResNec32x4
VGG13
VGG13
ResNee50
ResNe132x4
Student
WRN-16-2 ResNet20 ResNet8x4
VGG8
MobileNetV2
VGG8
ShuffleNerV2
Teacher #Params (10%)
2.2626
0.8639
7.4496
9,5076
9,5076
23.8570
7.4496
Student #Params (10%)
0.7144
0,2833
1.2565
4.0134
0,9226
4.0134
1,4418
Compression ratio
3.1674x
3.0498x
5.9289
2.3690x
10.3056x
5.9444x
5.1667 x
Teacher #FLOPs (109)
2.6208
1.0060
8.6642
1.8263
1.8263
10.3833
8.6642
Student #FLOPs (10%)
0,8089
0.3265
1.4165
0,5428
0.0459
0,5428
0,3449
Speedup ratio
3.2400x
3.0809x
6.1168x
3.3643x
39.8097
19.1276x
25.1210x
Table 8: Model complexity statistics of ResNet34 (teacher) / ResNet18 (student) on the ImageNet dataset. The number of FLOPs is measured by the number of Multiply and Add operations in convolutional and fully-connected layers. Input image size: 224 x 224xx3.
Teacher #Params (10") Student #Params (10") Compression ratio Teacher #FLOPs (10") Student #FLOPs (10") Speedup ratio
21.7806
11,6799
1.8648x
7.3275
3.6281
2.0196x
B COMBINING TLCUTMIX WITH OTHER KD METHODS
In the main paper, we show that our proposed method can advance the previous state-of-the-art method (CRD) even further on most pairs (Tabs. and 4). It interesting to see if this bonus can translate to more KD methods. Therefore, here we combine TLCutMix with five more KD methods see how works. The five methods are AT (Zagoruyko Komodakis, 2017). CC (Peng et al., 2019), SP (Tung & Mori, 2019), PKT (Passalis & Tefas, 2018), 8), and VID (Ahn et al., 2019). Note, these five methods we select are the top-performing KD methods (besides CRD) based on the CIFAR- 100 results in the CRD paper (it is easy improve mediocre KD method).
Results are shown in Tab. 9. When equipped with our proposed TLCutMiX, all these methods see accuracy gains, although some teacher-student pairs see more improvement than others. For ex- ample, SP+TLCutMix and VID+TLCutMix only have marginal gains on the 10/Nese1// pair. Except them, all the other pairs significant accuracy improvement. There are 42 results in total in Tab. 9. Half of them (21 pairs) are improved by more than % point. Several (5 pairs) even improved by over 2% points.
12

---

As explained in the main paper, our method focuses on the input end to improve KD, while methods like AT, CC, SP, PKT and VID focus on the output end (i.e., better loss function). Therefore, they are complementary. The results here reiterate one of our contributions in this paper: most Under review as a conference paper at ICLR 2022
Table 9: Student test accuracy (standard deviation) of different KD methods on CIFAR-100 100 when equipped with the proposed TLCutMix. The results of the different KD methods are directly cited from the CRD paper Table (Tian et al., 2020) (where they did not report the standard deviation of the accuracies except for the original KD method (Hinton et al.. 2014), so the standard deviation of accuracies of these methods are missing here as well). AT, CC, SP, PKT, and VID methods include the original KD loss as part of their loss functions. Each our result is obtained by 3 random runs, mean (std) accuracy reported. Accuracy gains are colored in red (this table best viewed in color).
Teacher Student
WRN-40-2
ResNet110
ResNet32x4
VGG13
VGG13
ResNe150 VGGS
ResNet32s4
WRN-16-2
ResNet20
ResNet8x4
VGGS
MobileNetV2
ShuffleNetV2
Teacher Ace.
75.61
74.31
79.42
74.64
74.64
79.34
79.42
Student Acc.
73.26
69.06
72.50
70.36
64.60
70.36
71.82
KD (Hinton et 2014)
74.92 (0.28)
70.67
(0.27)
73.33 (0.25)
72.98 (0.19)
67.37 (0.32)
73.81
(0.13)
74.45 (0.27)
KD+TL.CutMix (ours)
75.34 (0.19)
71.19 (0.23)
74.91
(0.20)
74,16 (0. 18)
68.
(0.35)
74.85
(0.23)
76.61 (0.18)
Acc. gain
+0.42
+1.58
+1.18
+1.42
+1.04
+2.16
AT (Zagonuyko & Komodakis, 2017)
75.32
70.97
74.53
73.48
65.13
74.01
75,39
AT+TLCutMix (ours)
75.65 (0.27) 71.66
(0.07) 75.68
(0.13) 74.02
(0.15) 67.20 (0.24)
74.67
(0.14) 76.25 (0.13)
Acc. gain
+0.33
+0.69
+1.15
+0.54
+2.07
+0.66
+0.86
CC (Peng et 2019)
75.09
70.88
74,21
73.04
68.02
73.48
74.71
CC+TI.CuMMi (ours)
75.75 (0.27) 71.41 (0.24)
75.54 (0,28)
74.35
(0.27)
(0.46)
74.76
(0.09)
76.78 (0.18)
Acc. gain
40.66
40.53
+1.33
+1.31
+0.42
+1.28
+2.07
SP (Tung Moni 2019)
74.98
71.02
74.02
73.49
68.41
73.52
74.88
SP+TLCuuMix (ours)
75.29 (0.39) 71.10
(0.07)
74.96
(0.13) 74.10
(0.27)
68.79
78.79 (0.24)
74.77
(0.33)
76.24 (0.14)
Acc. gain
+0.31
+0.08
+0.94
+0.61
+0.38
+1.25
+1.36
PKT (Passalis & Tefas, 2018)
75,33
70.72
74.23
73.25
68.13
73.61
74.66
PKT+TLCutMix (ours)
75.85 (0.42)
71.33
(0.06)
75.44
(0.08)
74.30
(0.18)
68.98
(0.60)
74.70
(0.32)
76.79 (0.10)
Ace. gain
+0.52
+0.61
+1.21
+1.05
+0.85
+1.09
+2.13
(Ahh et al., 2019)
75.14
71.10
74.56
73.19
68.27
73,46
74.85
VID+TLCutMix (ours)
75.66 (0.21)
71.13
(0.27)
75.40
(0.17)
74.24
(0.12)
69.70
(0.22)
74.67
(0.17)
76.90 (0.17)
Acc. gain
+0.52
+0.03
+0.84
+1.05
+1.43
+1.21
+2.05
Table 10: KD+Cutout (DeVries & Taylor, 2017) vs. KD on the CIFAR-100 dataset.
Teacher Student
WRN-40-2
ResNet56 ResNet20
ResNet32x4 ResNet8x4
VGG13
VGG13
ResNet50
ResNet32x4
WRN-16-2
VGG8
MobileNetV2
VGG8
ShuffleNetV2
Teacher
75.61
72.34
79.42
74.64
74.64
79.34
79.42
Student
73.26
69.06
72.50
70.36
64.60
70.36
71.82
KD
74.92 (0.28) 70.66 (0.24)
73.33 (0.25)
72.98
(0.19)
67.37 (0.32)
73.81
(0.13)
74.45 (0.27)
KD+Cutout 75.54 (0.16) 70.86 (0.19) 74.32 (0.30) 74.03 (0.08) 68.22 (0.06) 73.98 (0.19) 75.20 (0.10)
existing KD papers seek to improve KD through a better loss function while we discover new axis improving KD through a stronger DA, which is just as promising.
C KD+CUTOUT ON CIFAR-100
In the main paper, we have shown that the KD loss can exploit more advanced data augmentation schemes (like mixup/CutMixx) for improved performance. Another popular image data augmentation method, which is stronger than the common random cut and horizontal flip, is cutout (DeVries & Taylor, 2017). In Tab. 10 we show KD can also work with cutout to consistently deliver stronger performances. This further confirms our finding that can simply boost the KD performance by using a stronger data augmentation.
D CUTMIX SAMPLE ANALYSIS
CutMix sample analysis and why KD is naturally suited to exploit CutMix. During the KD training of ResNee34/ResNer18 on ImageNet, we recorded the CutMix samples on which the teacher disagrees with the CutMix scheme on the label. We call this label disagreement issue As show in
13

---

Under review as a conference paper at ICLR 2022
ground
Arabian camel
beetle
0.544
airhip
0.538
0.587
quill
0,44
Tibetan
acoustic
Indian elephant
cab
0.486
483
0,133
0.776
terrier
guitar
cub
M44
Yorkshire terrir
0,300
shopping cart
0.220
coffeepot
ambulance
Lhasa
shopping basket
milk can
police van
0,20
silky terrier
0.145
solar dish
0.107
ca caldron
0.184
minivan
0004
Norfolk terrier
0,090
guineu pig 0.772
water jug
0.055
minibus
Tibetan terrier
chainlink fence
nnss
bucket
(8)
(b)
(d)
Figure 5: ImageNet CutMix samples where the main object in one of the images is longer vis- ible after CutMix augmentation. Below each sample, the first is the target probability assigned by CutMix and the second is the top-5 predicted probabilities by the teacher. These examples can be misleading when cross-entropy loss is used, but not for KD, as explained in the text.
Fig. 5. there exist cases where the image cut from one image covers the salient object in the other. For example, the cab (a) completely covers the ground beetle. In this case, using the label by CutMix does not make sense anymore. A similar problem appears on (b). Note that these misleading labels by CutMix are rectified when the teacher is employed to guide the student. The teacher assigns the correct label "cab" to (a) and "Yorkshire terrier" to (b) (which is still not the true label "Tibetan terrier" but is clearly more relevant and "Tibetan terrier" also the top-5 predictions). For (c) and (d), they pose problem more than occlusion: the foreground cut in (c) is labeled as "acoustic guitar", however, the cut too small for us to make it out without knowing the label. Meanwhile, be the
the background object "Arabian camel" is occluded. Then the grids in the picture turn most salient part. we look at the predictions of the teacher, "shopping cart" and "shopping basket" clearly make more sense than either of the original two labels. similar issue happens on (d), where the "Indian elephant" is largely occluded. The foreground cut is labeled "quill" but the bottle in the middle is more salient. Thus the teacher predicted as "coffeepot", "milk can",, In order how severe the label disagreement issue is, we counted the number of these synthetic samples and found that on more than half of the samples (52.1 % produced by CutMix, the teacher model and CutMix hold different view regarding the label. Many of these suffer from the problem shown in Fig. The KD loss can rectify these label mistakes. This further shows the interplay between KD and DA: KD thrives DA and in turn, some DA schemes more reasonable for KD (than CE) where a teacher can supply more relevant labels.
14


Reviews:
[{"review_id": "Z3X1dpW5tAC", "reviewer": "Reviewer_zS3x", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros \n\n* KD has many applications in different fields, but why it is successful is still an open research problem. The paper aims to investigate empirically and specifically the effects of DA on KD, which is a worthy problem. \n\n* The advantage of using KD is obvious that the teacher will provide more information regarding the soft outputs, semantically similar classes that train the student better, and definitely with more examples, e.g., via data augmentation would be more useful. \n\n* New DA method is proposed.\n\nCons\n\n* The research problem is not well-stated. \n\n* The experimental results are not sufficient to support the claims. \n\n* The claim understanding the success of KD in DA perspective is not supported well in the paper.\n\n* The hypothesis (Eq .2) needs more rigorous experiments to support. \n", "summary_of_the_paper": "This paper aims to explore the advantage of using DA for KD for training deep neural networks and try to explain the interplay between DA and KD regarding the label advantage. \n\nThe main idea delivered in the paper regarding the interplay is that instead of fixing labels for augmented samples when using CE, the teacher in KD can act on different augmented to provide the better label to train the student. The paper also comes up with the hypothesis of the training interactions between KD and DA (e.g. KD loss makes use of more training iterations and the “stronger” DA method leads to better accuracy if training more interactions. The experiments are conducted with DA methods of Flip, Flip + Crop, TLmixup, TLCutMix. The paper also proposes TLCutMix+pick, which improves the baseline TLCutMix with an active learning approach. \n\nExperiments conducted on CIFAR-100, Tiny ImageNet and ImageNet show the improvements of using DA over the baselines and the improvement of the proposed TLCutMix+pick. ", "summary_of_the_review": "The paper investigate the interesting problem, but more rigorous experiments and clear statement stated in the paper. The improvements of the paper are not substantial and few of definition and hypothesis are vague and not well-supported. \n\nDetails of Cons\n\n* The key concern about the paper is the question being tackled in the paper is unclear, and the lack of rigorous experimentation to support the hypothesis.  For example, the purpose of investigating interplay is not well-stated. Why is it important and useful? Also, after reading the paper I do not understand the conclusion after investigating the interplay between KD and DA?\n \n* The main things shown empirically in the paper are that DA is helpful to improve accuracy, which is different from the claim of the paper in understanding the success of KD via the DA perspective. rather than that, no other insights about understanding more about KD or fundamental reasons why DA improves KD are not clear (at least no empirical support). We observe data augmentation can improve most training models, it is reasonable and not too surprising it will improve the student model. The question is that the student model is improved because of simply more training data to be used or because the soft label provided by the teacher makes the training better (we observe in the baseline CE, simply adding more data performance gets improved)? One simple experiment to demonstrate that is to modify the current model using the soft-label of the original samples as the soft-label of the augmented sample in KD training. \n \n* Another concern about the performance is that the improvements are minor compared to the original KD and most of the experiments are less than 1%. \n\n* There are also concerns about the hypothesis (Eq. 2). I believe it would need rigorous experiments to demonstrate this because it depends a lot on the network architecture and dataset size as well as the data augmentation methods and hyper-parameters. Without that, it is difficult to make the final conclusion. Moreover, I wonder why and how this hypothesis is useful for the community to investigate? I understand when we have more training data provided to the model it will take more time for the model to  converge. By the way, not sure how the authors defined a “stronger” DA? It looks like the “stronger” one will give better accuracy to students which may be vague. \n\n* As claimed in the paper, the “stronger” DA will need to train more iterations to achieve good accuracy. Does the author consider the combination of multiple DAs together a “strong” DA?\n\n* In the framework, can the author explain the reason to keep x and x’? Does that help to boost the performance? As in Eq. 1 we already have the CE part using the original samples x? \n\n* Many related state of the art data augmentations are missing: \n\n[1] AutoAugment: Learning Augmentation Strategies from Data.\n[2] RandAugment\n[3] Fast AutoAugment.  ​\n\nMinor comments: \n\nFigures and plots are not high quality. \n\nEq. 1 is the combination of CE and KD; the hyper parameter $\\alpha$ may lead to different observations and behaviors. Would be good to study on that.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KqvUQxIcJX2", "reviewer": "Reviewer_T234", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n- A new DA technique that is effectively applied to the existing state-of-the-art knowledge distillation technique is presented.\n- Many experiments prove that the proposed method is effective.\n- Overall, the readability of the paper is good\n\nWeakness\n- DA is generally a widely used technique to improve the performance of DNN, and the explanation of the principle of DA's performance improvement and its effect is somewhat trivial. Therefore, contribution 1 claimed by the authors appears to be over-claimed.\n- It is well known that DA differs in the degree of performance change depending on the learning configuration. However, the authors show that the proposed DA strategy is effective for only one thing: CutMix. Further analysis seems necessary.\n- There are doubts as to whether the authors' DA strategy is an effective technique only for knowledge distillation, and it appears to be a general DA algorithm. For example, it is necessary to check the degree of performance improvement when the student network learns the data generated by the proposed DA strategy.\n- Since the proposed DA strategy generates twice as much data as a general data provider, even if it learns the same iteration, it can be considered unfair in terms of cost.\n- Comparison techniques lack the latest techniques, and it seems necessary to reinforce them through sufficient surveys.", "summary_of_the_paper": "- This paper points out the fact that in-depth analysis of data augmentation (DA) in the field of knowledge distillation is rarely performed.\n- In addition, the authors present a new DA strategy that simultaneously utilizes the data generated through the existing DA and the data generated through the stronger DA to improve the performance of the knolwedge distillation algorithm.\n- And it shows that additional performance improvement is possible with a learning algorithm inspired by active learning.\n- Through this, the authors show that it is possible to effectively improve performance even with the DA technique, which has not been well fused in the past.\n- The proposed augmentation techniques effectively converge with existing knowledge distillation algorithms to show that they can lead to further performance improvements, and as a result, state-of-the-art performance is achieved in various datasets and various teacher-student pairs.", "summary_of_the_review": "The analysis and proposal methods covered in this paper are interesting in terms of performance improvement.\nHowever, it is unclear whether the DA algorithm is for knowledge distillation only.\nIn addition, it is necessary to analyze whether the performance improvement is due to augmentation or a large amount of learning.\nData augmentation seems to be the most important contribution point of the authors, so it seems that clear verification is needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H4d4CB9ArqG", "reviewer": "Reviewer_146T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has the following strengths:\nS1: Explaining the rationale behind the success of KD approaches from the point of view of DA approaches is very interesting and the paper makes good progress in this aspect. Figure 2(a) is very illustrative. It clearly demonstrates that compared to applying DA approaches to CE losses, applying DA approaches to KD losses has the potential of bringing more performance gains.\nS2: The experimental results presented in the paper are quite extensive and impressive. The paper conducts experiments on 3 datasets: CIFAR-100, Tiny ImageNet, and ImageNet, varying from tens of thousands of images to millions of images. The paper applies a variety of network architectures such as VGG, ResNet, Wide-ResNet, etc. to implement student models and teacher models. The paper does a wide range of ablation studies like studying effects of using more training iterations, exploring effects of using various DA approaches or data picking schemes, etc.\nS3: The paper is well-presented. It is easy to follow the logic flow of the paper and understand the proposed approach.\n\nThe paper has the following weaknesses:\nW1: Although the explanation in Section 3.1 why DA approaches are more beneficial to KD losses than CE losses makes sense, it is not mathematically rigorous. It is not clear how to reason from a specific example of the advantage of using DA approaches in KD losses in Figure 2(a) to the hypotheses regarding the optimal number of training epochs in Equation (2). The last two inequations in Equation (2) are sort of known hypotheses: stronger DA approaches by design can be applied to a wide range of training losses including KD losses and are expected to outperform a standard DA approach regardless of the choice of training losses. These hypotheses are disconnected from the main purpose of the paper: as indicated by the title, the paper aims to explain why KD approaches are so successful. It is highly recommended that the paper adds more theoretical details of how interplay of KD losses and CE losses is related to the success of KD.\n\nW2: I am a bit concerned about the technical depth of the paper. Although overall the ideas make sense, these ideas are pretty straightforward. It is important for the paper to elaborate what is the challenge and how the proposed approach solves this challenge. For example, it is not clear what is the challenge of adapting stronger DA approaches when using the KD losses and the adaptation method proposed in the paper seems to be straightforward.\n\nW3: The related work can be improved. As mentioned in the paper, an effective idea for KD is to directly match response-based knowledge, feature-based knowledge, etc. between student models and teacher models [1]. To improve the process of transferring various types of knowledge between student models and teacher models, there are many algorithms like adversarial distillation [2], multi-teacher distillation [3], etc.\n\nW4: The proposed approach does not outperform some baseline approaches in a few experimental settings. For example, in Table 5, the baseline approach proposed by Hinton et al., 2014 performs the best on ImageNet under top-5 accuracy when using ResNet18 as student models. It would be better to explain the reason why the proposed approach under-performs the baseline approach in such experimental setting. The baseline approach by Hinton et al. is proposed around 7 years ago and it would make the conclusions more convincing if the paper could compare against more recent works [1].\n\n[1] Knowledge Distillation: A Survey\n[2] Kdgan: Knowledge distillation with generative adversarial networks\n[3] Deep model compression: Distilling knowledge from noisy teachers\n\n", "summary_of_the_paper": "This paper presents a generic framework to explain an observation that by using more training iterations, knowledge distillation (KD) losses benefit more from data augmentation (DA) than cross entropy (CE) losses do. The paper proposes an efficient DA approach to improve the performance of KD approaches by borrowing ideas from active learning. The paper shows that the proposed DA approach can improve the performance of a standard KD loss as well as more advanced KD losses. The paper also shows that the standard KD loss augmented with the proposed DA approach can outperform more advanced KD losses without using the DA approach.\n\nThe main contributions of the paper include: (1) This paper observes that DA approaches are more beneficial to KD losses than CE losses and explain the rationale behind this observation; (2) This paper proposes to enhance the performance of the tradition KD loss by adapting two stronger DA approaches named mixup and CutMix; (3) This paper further proposes an even stronger DA approach which is customized for KD losses and is built on top of ideas from active learning; (4) This paper empirically shows that incorporating the proposed DA approach into KD losses achieves new state-of-the-art accuracy on 3 datasets.\n", "summary_of_the_review": "Although the paper has several strengths, I am mostly concerned about the lack of arguments supporting the main purpose of the paper and the technical depth of the paper. Also, the related work and the experimental analyses of the paper can be further improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All of the reviewers recommended rejecting this paper.
There were concerns that the underlying research questions being probed were not expressed clearly enough.
Reviewers were concerned that the experimental work was not sufficient to warrant acceptance.
Other concerns included the technical depth of the paper, the degree to which related work was discussed, placed in context and compared with empirically.
The AC recommends rejecting this paper.
Decision: Reject