Year: 2021

Abstract:
Knowledge distillation (KD) is a general neural network training approach that uses a teacher model to guide a student model. Many works have explored the rationale for its success. However, its interplay with data augmentation (DA) has not been well understood so far. In this paper, we are motivated by an interesting observation in classification: KD loss can take more advantage of a DA method than cross-entropy loss simply by training for more iterations. We present a generic framework to explain this interplay between KD and DA. Inspired by it, we enhance KD via stronger data augmentation schemes named TLmixup and TLCutMix. Furthermore, an even stronger and efficient DA approach is developed specifically for KD based on the idea of active learning. The findings and merits of our method are validated with extensive experiments on CIFAR-100, Tiny ImageNet, and ImageNet datasets. We achieve new state-of-the-art accuracy by using the original KD loss armed with stronger augmentation schemes, compared to existing state-of-the-art methods that employ more advanced distillation losses. We also show that, by combining our approaches with the advanced distillation losses, we can advance the state-of-the-art even further. In addition to very promising performance, this paper importantly sheds light on explaining the success of knowledge distillation. The interaction of KD and DA methods we have discovered can inspire more powerful KD algorithms.

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) are the best performing machine learning method in many fields of interest (LeCun et al., 2015; Schmidhuber, 2015). How to effectively train a deep network for classification has been a central topic for decades. In the past several years, efforts have mainly focused on better architecture design (e.g., batch normalization (Ioffe & Szegedy, 2015), residual blocks (He et al., 2016), dense connections (Huang et al., 2017)) and better loss functions (e.g., label smoothing (Szegedy et al., 2016; Müller et al., 2019), contrastive loss (Hinton, 2002), largemargin softmax (Liu et al., 2016)) than the standard cross-entropy (CE) loss. Knowledge distillation (KD) (Hinton et al., 2014) is a training framework that falls in the second group. In KD, a stronger network – called teacher – is introduced to guide the learning of the original network – called student – by minimizing the discrepancy between the representations of the two networks,
LKD = (1− α)LCE(y, p(s)) + ατ2DKL(p(t)/τ, p(s)/τ), (1)
where DKL represents KL divergence (Kullback, 1997); α ∈ (0, 1) is a factor to balance the two loss terms; LCE denotes the cross-entropy loss; y is the one-hot label and p(t), p(s) stands for the teacher’s output and student’s output, respectively (which are probability distributions over the classes); τ is a temperature constant (Hinton et al., 2014) to smooth predicted probabilities. KD allows us to train smaller, more efficient neural networks without compromising on accuracy, which facilitates deploying deep learning in resource constrained environments (e.g., on mobile devices). The effectiveness of KD has been seen in many tasks (Chen et al., 2017; Wang et al., 2020; Jiao et al., 2019; Wang & Yoon, 2021). Meanwhile, many works have investigated the reason behind its success, such as class similarity structure (Hinton et al., 2014) and regularization (Yuan et al., 2020). However, few works have paid attention to its interplay with the input image data augmentation (DA), a technique to obtain more data through various transformations (Shorten & Khoshgoftaar, 2019). In this paper, we will show that data augmentation is also an important dimension to explain
the success of KD. Moreover, our findings show we can achieve much better performance simply using the original KD loss equipped with a stronger data augmentation scheme.
Our proposed algorithms are inspired from interesting observations shown in Fig. 1, where we plot the student test error curves when the model is trained for different numbers of epochs using KD loss vs. CE loss1. Three data augmentation scenarios are examined: not using DA at all (Without DA), only using the horizontal flip (Flip), and using both the horizontal flip and random crop (Flip+Crop). We have the following observations. (1) Within each plot, KD loss delivers lower test error than CE loss. (2) When DA is used (comparing the middle or right plot to the left), both CE and KD curves are improved. (3) When DA is used (comparing the middle or right plot to the left), the optimal number of training epochs is postponed and the postponement is greater for KD than CE (the optimal number of epochs is postponed from 180 to 480 for KD versu from 60 to 120 for CE). (4) When a stronger DA is employed (comparing the right plot to the middle), the optimal number of epochs is further postponed with even lower test error. The first two obser-
vations are well-recognized by existing works (they simply reiterate the effectiveness of KD and DA, respectively), while the last two observations are new discoveries of this work, concerning the interaction between KD and DA. In other words, KD and DA, as two common techniques to improve the performance of DNNs, are actually not independent. This paper explains the interplay between KD and DA and leverages it for stronger KD performance using only the standard KD loss.
Specifically, we explain why KD is able to exploit DA more than the CE loss. Owing to the random transformations in data augmentation, the input data are not fixed over different epochs. Different views of each image are presented over the training process. When KD loss is used, the teacher maps these different views to different targets. As illustrated in Fig. 2(a), these targets with different probability structures can reveal more information of the data, thus helping the student more. In contrast, when CE loss is adopted, the target is fixed regardless of the different views of the input. The extra information is thus lost. This observation inspired us to develop two stronger data augmentation techniques (TLmixup, TLCutmix) that are tailored for KD. We further tap into the idea of active learning to make TLCutmix even better, that is, our TLCutmix+pick method.
In summary, these are the main contributions of this work:
• We make novel observations of the interaction between KD and DA. We explain how DA methods are more suited to be exploited by KD which uses teacher outputs as labels instead of the ground-truth one-hot labels used by CE loss.
• Inspired by the above, we propose to enhance the original KD loss with stronger data augmentation schemes (by adapting mixup (Zhang et al., 2018) and CutMix (Yun et al., 2019) to KD). It is shown that these methods are more reasonably applied in the KD case than in the CE case.
• We further propose an even stronger data augmentation method specifically for knowledge distillation using the idea of active learning (Settles, 2011).
• We show empirically better results simply by using the original KD loss combined with the proposed DA scheme, compared to state-of-the-art KD methods, which adopt more advanced distillation losses.
1Note, data points on one curve in Fig. 1 are from independent experiments of different total number of epochs; the learning rate schedule is proportionally scaled based on the total number of epochs.

heading: 2 RELATED WORK
text: Knowledge distillation: The general idea of knowledge distillation is to guide the training of a student model through a stronger pretrained teacher model (or an ensemble of models). It was pioneered by Buciluǎ et al. (Buciluǎ et al., 2006) and later refined by Hinton et al. (Hinton et al., 2014), who coined the term. Since its debut, knowledge distillation has seen extensive application in vision and language tasks (Chen et al., 2017; Wang et al., 2020; Jiao et al., 2019; Wang & Yoon, 2021). Many variants have been proposed regarding the central question in knowledge distillation, that is, how to define the knowledge that is supposed to be transferred from the teacher to the student. Examples of such knowledge definitions include feature distance (Romero et al., 2015), feature map attention (Zagoruyko & Komodakis, 2017), feature distribution (Passalis & Tefas, 2018), activation boundary (Heo et al., 2019), inter-sample distance relationship (Park et al., 2019; Peng et al., 2019; Liu et al., 2019; Tung & Mori, 2019), and mutual information (Tian et al., 2020). Over the past several years, the progress has been made primarily at the output end (i.e., through a better loss function). In contrast to previous works, our goal in this paper is to improve the KD performance at the input end with the help of data augmentation. We will show this path is as effective and also has much potential for future work.
Data augmentation: Deep neural networks are prone to overfitting, i.e., building input-target connection using undesirable or irrelevant features (like noise) in the data. Data augmentation is a prevailing technique to curb overfitting (Shorten & Khoshgoftaar, 2019). In classification tasks, data augmentation aims to explicitly provide data with label-invariant transformations (such as random crop, horizontal flip, color jittering) in the training so that the model can learn representations robust to those nuisance factors. Recently, some advanced data augmentation methods were proposed, which not only transform the input, but also transform the target based on certain corresponding relations. For example, mixup (Zhang et al., 2018) linearly mixes two images with the labels mixed by the same linear interpolation; manifold mixup (Verma et al., 2019) is similar to mixup but conducts the mix operation in the feature level instead of pixel level; CutMix (Yun et al., 2019) pastes a patch cut from an image onto another image with the label decided by the area ratio of the two parts. Now that the input and target are transformed simultaneously, the key is to maintain a semantic correspondence between the new input and new target. Although these methods have been proven effective, one lingering concern is about the reasoning behind them. Specifically, it is easy to come up with examples where the semantic correspondence is poorly kept (see Fig. 5 for examples on CutMix). Unlike these methods, which focus on general classification using the cross-entropy loss, our work investigates the interplay between data augmentation and knowledge distillation loss and the proposed new data augmentation is specifically for knowledge distillation.
One recent work (Das et al., 2020) also conducts empirical study about the impact of data augmentation on knowledge distillation. However, their exploration is very different from ours: they first apply data augmentation (e.g., mixup/CutMix) to the teacher then conduct the distillation step as usual (no extra data augmentation in this step); our investigation is the exact opposite to their setup: we train the teacher as usual (not applying mixup/CutMix), then in the distillation step we employ a more advanced data augmentation (e.g., mixup/CutMix). Interestingly, they conclude that the teacher trained with mixup/CutMix hurts the student’s generalization ability, while we consistently see a performance boost by using a stronger DA in the distillation step during student training.

heading: 3 PROPOSED METHOD

heading: 3.1 INTERPLAY OF KD, DA, AND TRAINING ITERATIONS
text: We first introduce a framework to explain the phenomenon that KD can exploit DA more simply by training for more iterations. Training for more iterations means presenting more examples to the network. Over iterative training, the presented examples are not exactly the same among different epochs because of the random transformations of data augmentation. Different versions of an image produced by data augmentation can be regarded as multi-views of that image (Wu et al., 2018; Tian et al., 2020). We term this kind of data difference as input view diversity. As depicted in Fig. 2(a), when the CE loss is employed, different views of an image are mapped to a single point in the target space (the hard label). In contrast, when the KD loss is used, different views of the data are mapped to a group of points in the target space through the teacher, which can reveal richer information around that class. By richer, we specifically mean two sources. First is the class structure information provided by the soft labels instead of the hard labels, i.e., the well-known dark knowledge (Hinton et al., 2014). Second is the information provided by the different input views from data augmentation.
Concretely, in Fig. 2(a), although the three views of the input share the same main class “dog”, the target probability vectors are different: compared with the first view, the second one has more lawn in it, thus the teacher has larger predicted probability in the “lawn” class; similarly, the third view has more sunshine, thus larger probability in the “sunset” class. These subtle changes in class-wise probabilities are beneficial for the student to learn. However, if the CE loss is used, all the three views are mapped to the same one-hot label, not putting the extra information to good use. More training iterations keep producing new data views to the student, making it afford more training epochs without overfitting. In contrast, for CE, as the target is fixed for different views, little new information is added in longer training. Thus, the student can only afford fewer extra epochs.
Denote the optimal number of training epochs asN , lowest test error asE. The synergistic interplay between KD and DA can be summarized in the following hypotheses,
N (DA)KD −N (w/o DA) KD > N (DA) CE −N (w/o DA) CE ,
N (DA+)KD > N (DA) KD and E (DA+) KD < E (DA) KD ,
(2)
where “DA” refers to a data augmentation method; “DA+” refers to another data augmentation method stronger than “DA”; “w/o DA” means not using any data augmentation. These inequalities are empirically verified in our experiments (Fig. 1, Fig. 3). The first inequality suggests the advantage of KD over CE: given the same DA scheme, KD loss can make more use of extra training iterations. The second suggests we can obtain better accuracy simply by training for more epochs using a stronger DA method. This leads us to investigate stronger DA methods for KD, as follows.

heading: 3.2 PROPOSED ALGORITHMS FOR IMPROVED KD
text: (1) KD+TLmixup/TLCutMix. We continue our exploration with two existing data augmentation techniques that are more advanced than the standard random crop and flip: mixup (Zhang et al., 2018) and CutMix (Yun et al., 2019). They are initially proposed in the CE case. Here we upgrade them for KD, resulting in TLmixup and TLCutMix (TL is short for teacher-labeled).
Specifically, let x0 denote the raw data, x denote the transformed data by the standard augmentation (random crop and flip). Illustrated in Fig. 2(b), we propose to add mixup/CutMix following x to obtain x′. Unlike the common data augmentation where only the transformed input is fed into the
network, we keep both the input x and x′ for the training (as such, the number of input examples during training is increased). The consideration of keeping both inputs is to maintain the information path for the original input x so that we can easily see how the added information path of x′ leads to a difference.
For x, its loss is still the original KD loss, consisting of the cross-entropy loss and the KL divergence (Eq. 1). Of special note is that, for x′, its loss is only the KL divergence, i.e., we do not use the labels assigned by mixup or CutMix because they can be misleading and do not perform well as we will show later (Tab. 1, Fig. 5). In fact, not using the hard label has another bonus. A dataset augmentation scheme which employs CE loss has to provide corresponding labels as supervisory information. In order to maintain the semantic correspondence, it cannot admit very extreme transformations for data augmentation. In contrast, in the mixup/CutMix+KD setting described above, the data augmentation scheme need not worry about the labels as they are assigned by the teacher. Therefore, it admits a broader set of transformations to expose the teacher’s knowledge more completely.
Between TLmixup and TLCutMix, we will empirically show TLCutMix is more favorable (Tab. 1) (and both of them are significantly better than the standard augmentation, random crop and flip). Therefore, we choose TLCutMix as base to develop our next algorithm as follows.
(2) KD+TLCutMix+pick. Our next algorithm is an even stronger DA scheme tailored to KD, based on the idea of active learning (Settles, 2011). In active learning, the learner enjoys the freedom to query the data instances to be labeled for training by an oracle (i.e., the teacher in our case) (Settles, 2011). Since the augmented data can vary in their quality, we can introduce certain criterion to pick the more valuable data for the student. We tap into the idea of hard examples (Micaelli & Storkey, 2019) to define the criterion. Specifically, we measure the hardness by the KL divergence between the teacher’s output and the student’s output (p(t) and p(s) are the teacher and student output probabilities over classes),
d = DKL(p(t)/τ, p(s)/τ). (3)
We sort the augmented samples by their d’s in ascending order and pick a subset with the largest d’s. Notably, the criterion d has exactly the same form that the student is supposed to minimize in Eq. (1); while here we pick samples to maximize it. This design makes an adaptive competition: when the student is updated, the criterion made of the KL divergence will also be updated. Each time, it makes sure the hardest samples are selected for the student.
Other common choices for the criterion include the teacher’s entropy or the student’s entropy (larger entropy implying more uncertainty meaning the sample is harder). They only take into account one-side information, either the teacher’s or student’s. Conceivably, they are not as good as the KL divergence criterion, which considers the information from both sides. This choice will be empirically justified in our experiments (Tab. 2).

heading: 4 EXPERIMENTAL RESULTS
text: Datasets and networks. We evaluate our method on the CIFAR-100 (Krizhevsky, 2009), Tiny ImageNet2, and ImageNet (Deng et al., 2009) object recognition datasets. CIFAR-100 has 100 object classes (32×32 RGB images). Each class has 500 images for training and 100 images for testing. ImageNet is now the standard large-scale benchmark dataset in image classification, which has 1000 classes (224×224 RGB images), over 1.2 million images in total. Tiny ImageNet is a small version of ImageNet with 200 classes (64×64 RGB images). Each class has 500 images for training, 50 for validation and 50 for testing. To thoroughly evaluate our methods, we benchmark them on various standard network architectures: VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016), WRN (Wide-ResNet) (Zagoruyko & Komodakis, 2016), MobileNetV2 (Sandler et al., 2018), ShuffleNetV2 (Ma et al., 2018). Our code and trained models will be made publicly available.
Evaluated methods. In addition to the standard cross-entropy training and the original KD method (Hinton et al., 2014), we also compare with the state-of-the-art distillation approach Contrastive Representation Distillation (CRD) (Tian et al., 2020). It is important to note that our method focuses on improving KD by using better inputs, while CRD improves KD at the output end (i.e., a better loss function). Therefore, they are orthogonal and we will show they can be combined together to deliver even better results.
Hyperparameter settings. The temperature τ of knowledge distillation is set to 4. Loss weight α = 0.9 (Eq. equation 1). (1) For CIFAR-100 and Tiny ImageNet, training batch size is 64; the original number of total training epochs is 240, with learning rate decayed at epoch 150, 180, and 210 by multiplier 1/10. The initial lr is 0.05. (2) For ImageNet, training batch size is 256; the original number of training epochs is 100, with learning rate decayed at epoch 30, 60, 90. The initial learning rate is 0.1. All these settings are the same as CRD (Tian et al., 2020) for fair comparison with it. Note, in our experiments we will show the results of more training iterations. If the number of total epochs is scaled by a factor k, the epochs after which learning rate is decayed is also be scaled by k. For example, if we train a network for CIFAR-100 for 480 epochs (k = 2) in total, the epochs after which the learning rate is decayed will be 300, 360, and 420.
We use PyTorch (Paszke et al., 2019) to conduct all our experiments. For CIFAR-100, we adopt the pretrained teacher models from CRD (https://github.com/HobbitLong/RepDistiller) for fair comparison with it. For Tiny ImageNet, we train our own teacher models. For ImageNet, we adopt the standard torchvision models as teachers.

heading: 4.1 CIFAR-100
text: Effect of more training iterations. In Sec. 1, we presented Fig. 1 to show the advantage of KD loss over CE loss in exploiting extra epochs. Here we show more results in Fig. 3 on different network architectures to confirm the finding is general. In line with the ResNet case (Fig. 1), extra training
2https://tiny-imagenet.herokuapp.com/
also brings more performance gains with KD loss on WRN and VGG. The gains are more or less up to the particular pairs but the trends are consistent: When DA is used, the optimal number of training epochs is higher and even more so for KD than CE; when a stronger DA is employed, the optimal number of epochs is further significantly higher and produces lower test error. These results support the proposed hypotheses in Eq. (2).
Exploring different data augmentation schemes. In Tab. 1 we compare three different DA schemes on CIFAR-100: the default, TLmixup, and TLCutMix. It has been shown in the original papers of mixup and CutMix that they improve accuracy over the standard data augmentation in the CE case. However, it does not mean naively combining KD and CutMix/mixup as it is can always bring performance improvement. As seen, CutMix is actually at odds with KD on the ResNet56/ResNet20 pair while our TLCutMix consistently improves the performance on all three pairs. On the other pairs, original CutMix is also not as effective as our adapted TLCutMix. These confirm that using the teacher’s output for distillation for the augmented data is critical. TLCutMix is better than TLmixup in general, so we choose it as base to develop TLCutMix+pick.
Exploring different data picking schemes. In Tab. 2, we compare the three potential schemes of selecting more informative data for the student: entropy of the teacher’s output (“T ent.”), entropy of the student output (“S ent.”), and the KL divergence between the teacher’s and student’s outputs (“T/S kld”). As shown, the KL divergence scheme performs best. This is expected as either the teacher entropy or student entropy alone does not reveal the whole picture.
Benchmark on CIFAR-100. The results are shown in Tab. 3. We have the following observations. (1) KD can be improved by training for more iterations (960 epochs vs. 240), owing to the effect of data augmentation (only one exception is ResNet32x4/ResNet8x4). This is not true for CE alone. This is a novel observation which shows the optimal number of training epochs for KD w/ DA is significantly different from that of CE. (2) Comparing the row “KD+TLCutMix” to “KD”, we see the proposed TLCutMix scheme improves the accuracies of all teacher-student pairs. On 5 out of
the 7 pairs, the improvement is very significant (more than 1 percentage point). (3) Comparing the row “KD+TLCutMix+pick” to “KD+TLCutMix”, we see 6 out of the 7 pairs are improved further, showing the proposed data picking scheme works in most cases. (4) Finally, “KD+TLCutMix+pick” scheme can be combined with more training iterations, which delivers even higher accuracies. (5) If comparing our best results (KD+TLCutMix+pick960) with those of CRD (though this is not an apples-to-apples comparison since the two methods focus on different aspects to improve KD), we can see our approach outperforms CRD on 6 out of the 7 pairs. It is worth emphasizing that we achieve this simply using the original KD loss (Hinton et al., 2014), with no bells and whistles. This justifies one of our motivations in this paper, i.e., existing KD methods (Peng et al., 2019; Park et al., 2019; Tian et al., 2020) mainly improve KD at the output layer through better loss functions, while we propose to improve KD at the input end and show this path is just as promising.
In the last two rows of Fig. 3, when CRD (Tian et al., 2020), the state-of-the-art KD algorithm, is armed with our proposed “TLCutMix+pick” and more training iterations, its results can be further advanced consistently. This demonstrates that the proposed schemes are general and can readily work with those methods focusing on better KD loss functions. In the Appendix Tab. 9, we present the results of applying TLCutMix to another 5 KD methods. All of the evaluated pairs see accuracy gains; half of them are even improved by more than 1% point.
Further remarks. Observation (1) above has another implication to the community in addition to improving the performance of KD. It tells us that the number of training iterations can have a big impact on the performance of a KD method. Unaware of this issue, if authors of a KD paper compare their method to others by directly citing numbers from other papers and the training epochs happen to be different, then the comparison may well be unintentionally unfair from the beginning.

heading: 4.2 TINY IMAGENET
text: In this section we evaluate the proposed schemes on a more challenging dataset – Tiny ImageNet. Similar to the case on CIFAR-100, we have results on different teacher-student pairs, shown in Tab. 4. For more training iterations, we train for 480 epochs instead of 960 to save time. Most claims on the CIFAR-100 dataset are also validated here: (1) “KD+TLCutMix” is better than KD, which is verified on all pairs. (2) “KD+TLCutMix+pick” is better than “KD+TLCutMix”, verified on 6 pairs. The exception pair is ResNet56/ResNet20, where adding data picking decreases the accuracy slightly by 0.11%. (3) When “KD+TLCutMix+pick” is equipped with more training iterations, we obtain the best performance. The main difference from CIFAR-100 results lies in the comparison between “KD480” and “KD”. In the CIFAR-100 case with standard augmentation, more training iterations consistently improves the accuracy on 6 pairs, while here only 3 are improved. We believe this is because the standard DA scheme – random crops and horizontal flips – cannot produce diverse enough data on this challenging dataset (Tiny ImageNet vs. CIFAR-100). In contrast, using the stronger DA scheme (TLCutMix+pick) and more training iterations does show significant improvement in all 7 out of 7 cases.
We also evaluate the compatibility of our DA methods with the state-of-the-art CRD, shown in the last two rows of Tab. 4. Our “TLCutMix+pick” method further advances the prior state-of-theart on 5 pairs. When CRD+TLCutMix+pick is trained for 480 epochs (instead of 240), further improvement can be seen on 6 of 7 pairs.

heading: 4.3 IMAGENET
text: We further evaluate our methods on the ImageNet dataset, shown in Tab. 5. “KD+TLCutMix” improves original KD (from 70.66 to 71.05 in top-1 accuracy). When data picking is added, it does not help here. Possible reasons will be analyzed later. When the student is trained for 200 epochs with KD+TLCutMix+pick, it delivers the new state-of-the-art top-1 performance.
We also present the result of original KD trained for 200 epochs. Interestingly, it matches the previous state-of-the-art method CRD and beats many other KD methods without any additional loss terms. However, this is not an apples-to-apples comparison, as these methods are trained for 100 epochs. Yet it is a clear indication that the interplay between KD and DA is useful even on a large-scale dataset.
Cross-dataset analysis. Here we investigate how the proposed method of KD+TLCutmix+pick is affected by the size and nature of the dataset. The resnet teacher-student pairs of Res56/Res20 and Res34/Res18 are of particular interest as the boost in performance for these pairs are lower than other network architectures.
The picking scheme is proposed based on the idea of active learning (Sec. 3.2). Intuitively, it can work only if the picked data has more information to the student network than those randomly presented. Since we adopt the KL divergence between the teacher’s output and the student’s output to measure the amount of information in the input data, we can compare this metric on two different sets of data, i.e., picked randomly vs. picked based on KL divergence. Specifically, we define average KL divergence ratio
r =
1 Np ∑Np i di
1 N ∑N j dj , (4)
where di stands for the KL divergence for the i-th sample defined in Eq. (3); N denotes the number of total samples in a batch; Np denotes the number of sample picked based on KL divergence (Np = N/2 in our experiments); note that r > 1. Larger r means the picked samples have more information than the average samples. Then we compare r on different datasets over the training process of “KD+TLCutMix+pick”. Results are shown in Fig. 4. As seen, in terms of r, CIFAR100 > Tiny ImageNet > ImageNet on average; meanwhile, comparing the results of CIFAR-100 (Tab. 3), Tiny ImageNet (Tab. 4), and ImageNet (Tab. 5) we see the accuracy gains brought by data picking also show the same trend of CIFAR-100 > Tiny ImageNet > ImageNet, in accordance with our expectation. This validates the soundness of the metric r we introduced. The r on ImageNet is clearly lower than the other two, meaning there is no significant information difference between the picked data and the average data, which may well explain the under-performance of the data picking scheme on ImageNet. Note that the root cause of this problem actually lies in the data augmentation part – since it cannot produce more informative samples, the subsequent data-picking has no scope to expand its value. How to obtain an even stronger scheme than TLCutMix remains elusive for now, which we will investigate as part of our future work. Also note, Res56/Res20 delivers the lowest r among the three pairs. This likely explains why the picking scheme is especially not effective on the original resnet pairs (Res56/Res20, Res34/Res18).

heading: 5 CONCLUSION
text: We carefully investigate the interplay between knowledge distillation (KD) and data augmentation (DA) in this paper. Unlike the cross-entropy loss, KD can exploit DA by training for more epochs. The proposed input view diversity framework explains the interplay well and inspires us to develop three new data augmentation methods specifically for KD. Extensive experiments demonstrate the merits of our methods across various networks on CIFAR-100, Tiny ImageNet, and ImageNet datasets. Our method achieves the new state-of-the-art using only the vanilla KD loss with no bells and whistles, showing the potential of improving KD from the input side rather than a better KD loss function. Our paper can also help the community build a more standard benchmark of KD algorithms, paying particular attention to the DA schemes and number of training epochs.

heading: A NETWORK PARAMETERS AND FLOPS
text: The number of parameters and FLOPs (FLoating-point OPerations) of each model on the CIFAR100, Tiny ImageNet, and ImageNet datasets are presented in Tab. 6, Tab. 7, and Tab. 8, respectively. For the number of parameters (or FLOPs), we only count them in the convolutional and fully-connected layers (BN layers not included) following the common practice.
Training cost of TLCutMix+pick. The proposed picking scheme only increases the wall-clock training time by around 30%, which is reasonable considering the performance gains.

heading: 21.7806 11.6799 1.8648× 7.3275 3.6281 2.0196×

heading: B COMBINING TLCUTMIX WITH OTHER KD METHODS
text: In the main paper, we show that our proposed method can advance the previous state-of-the-art method (CRD) even further on most pairs (Tabs. 3 and 4). It is interesting to see if this bonus can translate to more KD methods. Therefore, here we combine TLCutMix with five more KD methods to see how it works. The five methods are AT (Zagoruyko & Komodakis, 2017), CC (Peng et al., 2019), SP (Tung & Mori, 2019), PKT (Passalis & Tefas, 2018), and VID (Ahn et al., 2019). Note, these five methods we select are the top-performing KD methods (besides CRD) based on the CIFAR-100 results in the CRD paper (it is easy to improve a mediocre KD method).
Results are shown in Tab. 9. When equipped with our proposed TLCutMix, all these methods see accuracy gains, although some teacher-student pairs see more improvement than others. For example, SP+TLCutMix and VID+TLCutMix only have marginal gains on the ResNet110/ResNet20 pair. Except them, all the other pairs see a significant accuracy improvement. There are 42 results in total in Tab. 9. Half of them (21 pairs) are improved by more than 1% point. Several (5 pairs) are even improved by over 2% points.
As explained in the main paper, our method focuses on the input end to improve KD, while methods like AT, CC, SP, PKT and VID focus on the output end (i.e., a better loss function). Therefore, they are complementary. The results here reiterate one of our contributions in this paper: most existing KD papers seek to improve KD through a better loss function while we discover a new axis – improving KD through a stronger DA, which is just as promising.

heading: C KD+CUTOUT ON CIFAR-100
text: In the main paper, we have shown that the KD loss can exploit more advanced data augmentation schemes (like mixup/CutMix) for improved performance. Another popular image data augmentation method, which is stronger than the common random cut and horizontal flip, is cutout (DeVries & Taylor, 2017). In Tab. 10 we show KD can also work with cutout to consistently deliver stronger performances. This further confirms our finding that we can simply boost the KD performance by using a stronger data augmentation.

heading: D CUTMIX SAMPLE ANALYSIS
text: CutMix sample analysis and why KD is naturally suited to exploit CutMix. During the KD training of ResNet34/ResNet18 on ImageNet, we recorded the CutMix samples on which the teacher disagrees with the CutMix scheme on the label. We call this label disagreement issue. As show in
Fig. 5. there exist cases where the image cut from one image covers the salient object in the other. For example, the cab in (a) completely covers the ground beetle. In this case, using the label by CutMix does not make sense anymore. A similar problem appears on (b). Note that these misleading labels by CutMix are rectified when the teacher is employed to guide the student. The teacher assigns the correct label “cab” to (a) and “Yorkshire terrier” to (b) (which is still not the true label “Tibetan terrier” but it is clearly more relevant and “Tibetan terrier” is also in the top-5 predictions). For (c) and (d), they pose a problem more than occlusion: the foreground cut in (c) is labeled as “acoustic guitar”, however, the cut is too small for us to make it out without knowing the label. Meanwhile, the background object “Arabian camel” is occluded. Then the grids in the picture turn out to be the most salient part. If we look at the predictions of the teacher, “shopping cart” and “shopping basket” clearly make more sense than either of the original two labels. A similar issue happens on (d), where the “Indian elephant” is largely occluded. The foreground cut is labeled “quill” but the bottle in the middle is more salient. Thus the teacher predicted it as “coffeepot”, “milk can”, etc. In order to see how severe the label disagreement issue is, we counted the number of these synthetic samples and found that on more than half of the samples (52.1%) produced by CutMix, the teacher model and CutMix hold a different view regarding the label. Many of these suffer from the problem shown in Fig. 5. The KD loss can rectify these label mistakes. This further shows the interplay between KD and DA: KD thrives on DA and in turn, some DA schemes are more reasonable for KD (than CE) where a teacher can supply more relevant labels.

Review summaries:
- This paper points out the fact that in-depth analysis of data augmentation (DA) in the field of knowledge distillation is rarely performed. - In addition, the authors present a new DA strategy that simultaneously utilizes the data generated through the existing DA and the data generated through the stronger DA to improve the performance of the knolwedge distillation algorithm. - And it shows that additional performance improvement is possible with a learning algorithm inspired by active learning. - Through this, the authors show that it is possible to effectively improve performance even with the DA technique, which has not been well fused in the past. - The proposed augmentation techniques effectively converge with existing knowledge distillation algorithms to show that they can lead to further performance improvements, and as a result, state-of-the-art performance is achieved in various datasets and various teacher-student pairs.

This paper aims to explore the advantage of using DA for KD for training deep neural networks and try to explain the interplay between DA and KD regarding the label advantage.   The main idea delivered in the paper regarding the interplay is that instead of fixing labels for augmented samples when using CE, the teacher in KD can act on different augmented to provide the better label to train the student. The paper also comes up with the hypothesis of the training interactions between KD and DA (e.g. KD loss makes use of more training iterations and the “stronger” DA method leads to better accuracy if training more interactions. The experiments are conducted with DA methods of Flip, Flip + Crop, TLmixup, TLCutMix. The paper also proposes TLCutMix+pick, which improves the baseline TLCutMix with an active learning approach.   Experiments conducted on CIFAR-100, Tiny ImageNet and ImageNet show the improvements of using DA over the baselines and the improvement of the proposed TLCutMix+pick. 

This paper presents a generic framework to explain an observation that by using more training iterations, knowledge distillation (KD) losses benefit more from data augmentation (DA) than cross entropy (CE) losses do. The paper proposes an efficient DA approach to improve the performance of KD approaches by borrowing ideas from active learning. The paper shows that the proposed DA approach can improve the performance of a standard KD loss as well as more advanced KD losses. The paper also shows that the standard KD loss augmented with the proposed DA approach can outperform more advanced KD losses without using the DA approach.  The main contributions of the paper include: (1) This paper observes that DA approaches are more beneficial to KD losses than CE losses and explain the rationale behind this observation; (2) This paper proposes to enhance the performance of the tradition KD loss by adapting two stronger DA approaches named mixup and CutMix; (3) This paper further proposes an even stronger DA approach which is customized for KD losses and is built on top of ideas from active learning; (4) This paper empirically shows that incorporating the proposed DA approach into KD losses achieves new state-of-the-art accuracy on 3 datasets.