What Makes a "Good" Data Augmentation in Knowledge Distillation - A Statistical Perspective
Authors: Huan Wang
Suhas Lohit
Michael Jeffrey Jones
Yun Fu
Venue: NeurIPS 2022 Accept (2022)
Keywords: knowledge distillation
data augmentation
CutMix
CutMixPick
covariance of teacher-student cross-entropy
teacher's mean probability

Abstract: Knowledge distillation (KD) is a general neural network training approach that uses a teacher model to guide the student model. Existing works mainly study KD from the network output side (e.g., trying to design a better KD loss function), while few have attempted to understand it from the input side. Especially, its interplay with data augmentation (DA) has not been well understood. In this paper, we ask: Why do some DA schemes (e.g., CutMix) inherently perform much better than others in KD? What makes a "good" DA in KD? Our investigation from a statistical perspective suggests that a good DA scheme should reduce the covariance of the teacher-student cross-entropy. A practical metric, the stddev of teacher’s mean probability (T. stddev), is further presented and well justified empirically. Besides the theoretical understanding, we also introduce a new entropy-based data-mixing DA scheme, CutMixPick, to further enhance CutMix. Extensive empirical studies support our claims and demonstrate how we can harvest considerable performance gains simply by using a better DA scheme in knowledge distillation. Code: https://github.com/MingSun-Tse/Good-DA-in-KD.

What Makes a "Good" Data Augmentation in Knowledge Distillation - A Statistical Perspective
Huan Wang 1.2.t
1.2.f Suhas Lohit
Mike Jones?
Yun Fu'
Northeastern University, Boston, MA
MMERL, Cambridge, MA
Project: http://uuanwang.echhhccco
Wm 40 2/wm 16_2, CIFARI00
Standard
1.10
Manily
Teacher ((ieed)
Cogr Fii
S. test loss
1.05
ingue To
How to deffiee "ttrnggr" DA?
Stronger D.A.
Suakeu
Camichik OT mn))
Camrehht is cn
Spearman o 5667 (g vater 0.9395)
ip water 6:08%)
0.00425
0,00550 0.00475 0.00500 0.00525 0.00550
stddev
(a) Apply additional stronger DA KD
(b) S. test loss vs. T. stddev with different DA schemes
Figure 1: (a) Illustration of applying stronger data augmentation (DA) in addition the standard DA (random crop and flip) in knowledge distillation (KD). We ask: What makes "good" DA when is applied to KD in the manner of (a)? (b) We present proven proposition (Proposition 3.1) to answer this question rigorously, along with practical metric to evaluate the "goodness" of a DA. The proposed metric called stddev of teacher's 's mean probability (shorted T. stddev). As seen in (b), there is strong positive correlation (p-value 5% is typically considered statistically significant) between the student's ' test loss (S. test loss) and T. stddev, showing that T. stddev well captures the "goodness" of different DA schemes in KD. The most striking fact from this plot may be: T. stddev is purely calculated with the teacher (no any student used) while it can "predict" the relative order of the student's performance, implying the "goodness" of DA in KD probably is coudent-invarian
Abstract
Knowledge distillation (KD) is general neural network training approach that uses a teacher model to guide the student model. Existing works mainly study KD from the network output side (e.g., trying design a better KD loss function), while few have attempted to understand it from the input side. Especially, its interplay with data augmentation (DA) has not been well understood. In this paper, ask: Why do some DA schemes (e.g., CutMix) inherently perform much better than others in KD? What makes "good" DA in KD? Our investigation from statistical perspective suggests that a good DA scheme should reduce the covariance the feacher-student cross-entropy. A practical metric, the stddev of teacher's 's mean probability (T. stddev), is further presented and well justified empirically. Besides the theoretical understanding, we also introduce new entropy-based data-mixing DA scheme, CutMixPick, to further enhance CutMix. Extensive empirical studies support our claims and demonstrate how we can harvest considerable performance gains simply by using a better DA scheme in knowledge distillation.
"Thi paper originates from Huan's 's summer internship work at MERL. Coorreppnndigg author: slohit @0mer11mr1....mm com com
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Introduction
Deep neural networks (DNNs) are the de facto methodology in many artificial intelligence areas nowadays [23, 35]. How to effectively train a deep network has been a central topic for decades. In the past several years, efforts have mainly focused on better architecture design (e.g., batch normalization [19], residual blocks [13), dense connections [18]) and better loss functions (e.g., label smoothing [41, 28], contrastive loss [17]. large-margin softmax [24]) than the standard cross-entropp (CE) loss. Knowledge distillation (KD) [16] is a training method that falls into the second group. In KD, stronger network - called teacher- is introduced to guide the learning of the original network- called student by minimizing the discrepancy between the representations of the two networks,
LKD (1 aL--a. ) ar (p ( (T.D''/T))
where DxL. represents KL divergence [22]: E (0,1) is factor to balance the two loss terms; LCE denotes the cross-entropy loss: y is the one-hot label and p(), p(e) stand for the teacher's and student's s output probabilities over the classes; 7 is a temperature constant [16] to smooth predicted probabilities. KD allows to train smaller, more efficient neural networks without compromising on accuracy. which facilitates deploying deep learning in resource constrained environments (e.g., on mobile devices). KD has found plenty of applications in many tasks [6. 46, 12, 20, 47].
Most existing KD methods have attempted to improve by proposing better KD loss functions applied at the network outputs [32, 29, 433. Few works have considered KD from the input side. Especially, the interplay between KD and data augmentation (DA) [38] has not been well understood so far (note, by DA here, we mean the conventional DA concept: generating new input by transforming multiple inputs. A broader scope of DA may involve the neural network, e.g., dropout can be seen as kind of DA [3]. We do not consider this type of DA in this paper due to the limited length).
or
In this work, we ask: What makes a "good" data augmentation knowledge distillation? A clear answer to this question has many benefits. First, theoretically, it can help us towards better understanding about how data augmentation plays a role KD. Second, practically, it can bring us considerable performance gain in Fig. 2, we show test error rates using the standard CE loss (no teacher) vs. using KD loss (with a teacher). As seen, a stronger DA can lower the test error rate and admit more training iterations without overfitting in KD. It is pretty obvious to see that "Flip+Crop" is stronger than "Flip" alone in Fig. 2. However, for other DA schemes, such as Mixup [51] vs. Auto/ugmentt [7], which one is stronger? Not very clear. We thus desire principled way (e.g., concrete metric) to make the vague concept "stronger" exact. Presenting such a theoretically sound metric and empirically validating its effectiveness is the goal of this paper.
Intuitively, a good DA should enrich the input data and expose more knowledge of the teacher so that the student can generalize better. We for- malize this idea from a statistical learning per- spective. Specifically, we will show a good DA
Without DA
Flip
Flip+Croo Crop
62.0
KD
Test oroor rate (%6
39.39 55)
scheme is defined by a lower variance (or covari- ance) of the teacher's 's mean output probability over different input samples, which ultimately leads to a lower generalization gap for the stu- dent. The proposed theory is well justified by our extensive empirical studies on CIFAR 100 and Tiny ImageNet datasets with various pairs. The proposed theory well explains why Cut- Mix is better than other alternatives (such as Mixup [5]], AutoAugment [7]) in KD.
30.0
34.0 Sen 33
(120) 54.33)
30
20.0
720 28 03
28.0
550
730
Number of Sotall tesining epoccss
Figure 2: Test error rate of resnet20 on CIFAR100 when trained for different numbers of epochs with (KD) and without (CE) knowledge distillation (the teacher is resnet56 for KD). Each result is obtained by averaging 3 random runs (shaded area indi- cates the stddev). "Flip": random horizontal flip; "Crop": random crop. The optimal number of train- ing epochs and its test loss are highlighted in red.
In addition to the new theoretical results, we also propose an entropy-based data picking scheme to select more informative samples for KD, which can deliver even lower variance of the teacher's mean probability as well as lower gen- eralization error of the student.
We make the following contributions in this paper:
2

---

We present proven proposition (Proposition 3.1) that precisely answers what defines a better DA in KD: Given a fixed teacher, a better DA the one that gives a lower variance of the teacher's mean probability.
The proposition is well justified empirically on standard image classification datasets with many teacher-student pairs.
An entropy- based data picking scheme is introduced to further reduce the variance of the teacher's mean probability, which can further advance CutMix, the prior state-of-the-art DA approach among those evaluated in this paper.
Empirically, we show how the presented theory can benefit in practice we can simply enhance the existing KD methods by using stronger DA and prolonged training iterations.
2 Related Work
Knowledge Distillation (KD). The general idea of knowledge distillation is guide the training of student model through a (stronger) teacher model (or an ensemble of models). It was pioneered by Buciluă et al. [4] and later refined by Hinton al. [16], who coined the term. Since its debut, KD has seen extensive application in vision and language tasks [6, 14, 46, 20, 47]. Many variants have been proposed regarding the central question in KD, that is, how to define the knowledge transferred from the teacher to the student. Examples of such knowledge definitions include feature distance [33], feature map attention [50]. feature distribution [30], activation boundary [15]. inter-sample distance relationship [29, 32, 25, 44], and mutual information [43]. Several works [28, 37, 5] investigate the connection between label smoothing and knowledge distillation. Another line of works (e.g.. [42]) attempts to understand KD more theoretically. Over the past several years, the progress has been made primarily for intermediate feature maps and network outputs (i.e., through a better loss function). In contrast, our goal is to improve the KD performance at the input end with the help of data augmentation. We will show this path is as effective and also has much potential for future research.
Data Augmentation (DA) Deep neural networks are prone to overfitting, i.e., building input-target mappings using undesirable or irrelevant features (like noise) in the data. Data augmentation is prevailing techmique to curb overfitting [38]. In classification tasks, data augmentation aims to explicitly provide data with label-ime innariian transformations (such as random crop, horizontal flip, color jittering, Cutout [1I]) during training so that the model can learn representations robust to those nuisance factors. Recently, more advanced data augmentation methods were proposed, which not only transform the input, but also transform the target. For example, Mixup [51] linearly mixes two images with the labels mixed by the same linear interpolation. Manifold Mixup [45] is similar to Mixup but conducts the mix operation in the feature level instead of pixel level; CutMix [48] pastes a patch cut from an image onto another image with the label decided by the area ratio of the two parts. AutoAugment [7] is a strong DA method that finds an optimal augmentation policy from a large search space via reinforcement learning. When both the input and target are transformed simultancously, the key is to maintain semantic correspondence between the new input and new target. Unlike these methods, which focus on general classification using the cross-entropy loss, our work investigates the interplay between data augmentation and knowledge distillation loss and proposes new data augmentation specifically for knowledge distillation.
Some recent KD works also involve the utilization of DA in KD, such as [2, 8]. Especially, [2] also employs Mixup and prolonged training to enhance the student performance in KD, akin to ours. Yet it is worthwhile to note that our paper is subsiantially different from theirs, in that we are seeking the theoretical reason explaining how to define a better DA in KD to deliver better performance; these works mainly investigate in an empirical fashion, with no theoretical results presented. Mcanwhile, our work is not limited to one specific DA (see Sec. 5). We target a general theoretical understanding which can apply to broad scope of DA schemes (fortunately, experiments show, the proposed theory and the derived DA "goodness" measure indeed capture it). The fact that [2] utilizes Mixup [5]] and prolonged training to deliver 82.8% top-1 accuracy with resnet50 [13] on ImageNet [10] can be a direct proof that the principle proposed in this work can bring very promising practical benefits.
One recent work [9] also conducts empirical studies of the impact of DA on KD. They first apply DA (e.g., Mixup/CutMixx to the teacher training then conduct the KD step as usual (no extra DA in this step). Our investigation the exact opposite to their setup: We train the teacher as usual (no Mixup/CutMix), then in the KD step we employ more advanced DA (e.g., Mixup/CutMixx.
3

---

Interestingly, they conclude that the teacher trained with Mixup/CutMix hurts the student's general- ization ability, while we consistently see student performance boost via a stronger DA. Another recent work [40] utilizes KD to re-label mixed samples in Mixup to fix the inaccurate labelling problem of Mixup. Their work shows that KD) can be used make Mixup more generally useful, which is orthogonal to the topic of this work.
The work of [27] presents statistical perspective to understand how the softened probabilities in KD are better than the one-hot hard labels. Our work is inspired by their Bayes teacher notion. This said, our work is different from theirs in that we focus on explaining how data augmentation plays a role in KD and answer what characterizes a good DA, while they attempt to answer why the knowledge distillation loss is better than the standard cross-entropy loss.
Theoretical Investigation
3.1 Prerequisites: Multi-Class Classification with KD
Ja-i ~ DN,, where D is the joint distribution for input-output random variable pair (I, v), the goal in multi-class classification to pin down a predictor f: RC from hypothesis class H. where X' is the input space and C' refers to the number of classes. The predictor f is supposed to minimize the frue risk
Given a training set 5 ((I,, Bn)
R (f) def
[l,, f(x))),
(2)
where stands for the loss objective function (e.g, cross-entropy); the subscript D of Ro to emphasize that the true risk is defined on the data distribution. The true risk is approximated in practice on a separate test set.
For training, the predictor aims to minimize the empirical risk defined on the training sequence S:
1
(f
yyn (xx)))
where e, E 1 is a one-hot vector indicating the label y E [C] = 2.... CC. The subscript S of Rs is to emphasize the empirical risk is defined on the finite sampled data points.
In the context of KD, the one-hot hard target vector ey is replaced with a probability vector (O) E RC, giving us the empirical distilled risk of f:
(f)
del
N Ep (pn) log(((ar))
(4)
where the super-script t indicates the fixed teacher model. The theory concerning why Eq. (4) is better than Eq. (3) has been established in [27]. Interested readers may refer to their paper for more details. Next, look into how the data augmentation plays role in KD.
3.2 What Makes "Good" DA in KD?
Intuitively, a better DA should provide more information, i.e., expose more knowledge the teacher so that the student can absorb more and thus generalize better. We make this idea rigorous follows.
Proposition 3.1. Given a bounded loss function and a fixed teacher model with the empirical distilled risk defined in Eq. (4), for any predictor f, consider two sampled sequences S E 1 N and S₂ E N they made up of N elements sampled from the same distribution D. while not i.idd. (especially when data augmentation is employed). If the elements in S present a larger correlation than those in Sz, then the student's generalization gap trained on S₁ will be greater than that trained on Sz:
E SINDN
(Rss (f) (F))
E SaNDNN
(Rs, (f))
(5)
Proof. Let - (f( Ro (f). By the definition of variance, Es(A Vars[A]: + (Bs(A])². To make the notation clearer, define q(2.)= (x:) log(f(z:)).

---

(1) Since R (f) is a constant (albeit unknown).
N Σ N
EgLØr [ = [s]]ts([)] + Const Es
((x))) Const
(6)
Es (s()())) + Const
Ex, [g(xx)) + Const
N Ex [g(x)) Const = [xx( + Const,
N
where the second last equation is because each element x; in S is drawn from the same distribution
①. From the RHS of the last equation, we can clearly see that for S1, S2, Es[A] is the same.
(2) Then we consider Var Again, since Rg (f) is a constant, we only need to consider
N
N
N
Vars[Ats[)] = Varsl
d(;)) = N2
Cov
(
Σ ((w))
1=1
/=1
k=1
Varn; (((x;)) +2
Σ
Cov Covsla(zz).
ISjcKEN
(N- Varzlu(x)) +2
Σ
Covs(a(z,),
N
ISiCKEN
Var [g(x)] +
(ovsla(z)).
N
N2
ISjCKEN
where Covl.-] stands for covariance. From the last item in Eq. (7), we can see that the covariance part is different for different sampled S's. 5. If the samples in S₁ present a larger correlation than those S2, we will have Vars, [A] Vars, [A), which further leads increased generalization gap for
the student, ie., Es, [Rss (f) R. 2 Es, (Rs, (f) Rs (f)) The proof is finished. O
Practical Use: Stddev of Teacher's Mean Probability (T. stddev). Note in Proposition 3.1, the predictor (i.e., the student model) can be any one (not necessarily converged model). Each student would have an order different DA schemes regarding which is better "in its opinion". Presumably, different students will lead different such orders. For practical use, must pick a certain student as oracle to conduct the evaluation. Here, we can play a trick that can make it rather simple to use our theory- assume there is student that performs exactly the same the teacher (this assumption is not unpractical, since one straightforward example is to use the teacher as student). Then we have
Cov Cov[g(z;) ((xx)) Cov (p (x,)) log(f(r;)); (0) (IrE) log(f(xx)))
Cp (x,,) log(p (I;)), (5) (*x)) (og())))))))
In this case, we only need the covariance of the teacher's probability to measure the "goodness" of certain DA technique, no need for the student. Despite not using any information of the student, the proposed metric turns out correlate surprisingly well with the student's performance (see Fig. 4).
Based on Eq. (8), when using S vs. S2 as training sequence, the fundamental factor answering for the variance gap of 5ar boils down to the covariance in (x() (xx)), 1 That is, larger covariance in (p() (a:)) } leads larger covariance in (x.)) Tog(pp (0) (ax)) in Eq. (8),
which ultimately leads to a higher Vars[A) in Eq. (7).
The next step is to find a feasible way to estimate the covariance in (p(0) (x:)) Consider the averuge variable (denoted as u here) of several random variables (p) k
1
(Z)) K
(9)
Its variance (or equivalently, stddev) inherently takes into account the covariance among its addends. Therefore, the variance of proxy for the covariance in (p (xx)),,
m = Vars-(u),nng m E C m Σ (m:)2,m TR,
(10)
E
5

---

target (KD)
target (CF)
Standard DA
10.13, 0.210
D0,0,
Teacher (fiood)
10.10, 0.0.0..0500600..25
1.00,1.01
Raw inpuixx
Toput
KDD loss.
Inpul
0005,0.22.0.58.00 0.510.
10,0, 1.01
sunset dog lawn
Stronger DA
Student
Different views input
(a)
mupppd to different
mapped target
Input x''
Figure 3: (a) Illustration of the difference of supervised target between the KD loss and cross-entropy (CE) loss. An input is transformed to different versions owing to data augmentation. KD loss can provide extra information to the student by mapping these views to different targets, while the CE loss cannot. This work attempts to answer what characterizes a "good" data augmentation scheme in distillation. (b) Illustration of adapting an existing DA approach to KD. The standard DA consists of random crop and horizontal flip. This training framework is employed to empirically verify our proposed metric define a "stronger" data augmentation.
where K a pre-defined number of samples of set S to realize the averaging effect (e.g., K 640 in our experiments for CIFARI00 and Tiny ImageNet- see Appendix Sec. ?? for concrete calculation example). Note, u can be interpreted as the teacher's mean probability over K input samples.
If there is large covariance among (p()) [ m will be large (in element- sense). Then the average of m over classes, i.e., the m (we do this averaging simply because we desire a scalar metric), is good indicator to capture such covariance. We thus formally introduce m, the (averaged) T. stddev, as the proposed metric to measure the quality of data augmentation scheme. A lower m implies better DA by definition. In the experiments, will show this metric defined purely using the teacher can accurately characterize the generalization error of the students after distillation.
4 Evaluated Algorithms
4.1 Extend Existing DA Approaches for KD
Given an existing DA approach, this section explains how we adapt it properly to the case of KD. Specifically, let denote the raw data, denote the transformed data by the standard augmentation (random crop and flip). Illustrated in Fig. 3(b), we will add the DA following x to obtain x'. Unlike the common data augmentation where only the transformed input is fed into the network, keep both the input æ and x' for the training (as such, the number of input examples is doubled). The consideration of keeping both inputs is to maintain the information path for the original input so that we can easily see how the added information path of x' leads to difference.
For v. its loss is still the original KD loss, consisting of the cross-entropy loss and the KL divergence (Eq. (1)). Of special note is that, for x', its loss is only the KL divergence, i.e., we do not use the labels assigned by the DA algorithm (for example, in the original Mixup and CutMix, they assign a linearly interpolated label to the augmented sample) because these labels can actually be wrong (see Appendix Sec. ?? for concrete examples on ImageNet). In fact, not using the hard label has another bonus. A dataset augmentation scheme which employs CE loss has to provide corresponding labels as supervisory information. In order to maintain the semantic correspondence, it cannot admit very extreme transformations for data augmentation. In contrast, in the Mixup/CutMiix*X" setting described above, the data augmentation scheme need not worry about the labels as they are assigned by the teacher the recent work [2] refers this kind of utilization of DA as function matching of the teacher. As result, it admit broader set of transformations to expose the teacher's 's knowledge more completely. This reflects that data augmentation in KD has more freedom than in CE.
Among all the data augmentation techniques evaluated in this paper, we will show CulMix works best. The fundamental reason for its success, as we will show, that it achieves a much lower variance of the teacher's 's mean probability, implying it produces more diverse data than its counterparts.

---

4.2 CutMixPick: Enhancing CutMix with Entropy-Based Data Picking
In this section, we propose a data picking scheme to further reduce the variance of the teacher's mean probability. The idea is partly inspired by active learning [36]. In active learning, the learner enjoys the freedom to query the data instances to be labeled for training by an oracle (i.e., the teacher in our case) [36]. Since the augmented data vary in their quality, we can introduce certain criterion to pick the more valuable data for the student.
Intuitively, we regard a sample with more information is of higher quality. Therefore, we take Shannon's 's entropy of the teacher's output probability as a natural measure to select samples,
H (x)) -p( (z) (x))).
(11)
Note this formula is in exactly the same form of Eq. (8). Empirically, we will also show the data selected by this formula indeed results in lower m and better test loss for the student.
Concretely, given a batch of data, we first apply CutMix to obtain bunch of augmented samples. Then sort all the augmented samples by Eq. (11) in ascending order and keep the top r (a pre-defined percentage constant, r = 0.5 in our experiments) samples.
This simple technique can be rather effective according our empirical study. Another seemingly potential alternative is to use the student's 's entropy as the picking metric. The intuition behind this is that high-entropy sample in the view of the student can be regarded with these hard examples may expand the student's 's knowledge by squeezing its blind spots. Despite this intuitively plausible explanation, in practice, we will show this scheme actually under-performs the former student-agnostic scheme in Eq. (11), which is a bit surprising.
hard example, too. Learning
Experimental Results
Datasets and Networks. We evaluate our method primarily on the CIFARI00 [21] and Tiny ImageNet" datasets. CIFAR100 has 100 object classes (32x32 RGB images). Each class has 500
images for training and 100 images for testing. Tiny ImageNet a small version of ImageNet [10] with 200 classes (64x64 RGB images). Each class has 500 images for training. 50 for validation and
50 for testing. To thoroughly evaluate our methods, we benchmark them on various standard network architectures: vgg [39], resnet [13], wrn [49], MobileNetV2 [34], S2 [26]. We will also include results on ImageNet100 (a randomly drawn 100-class subset of ImageNet) and ImageNet.
Benchmark Methods. In addition to the standard cross-entropy training and the original KD method [16], we also compare with the ofate-of-the-art distillation approach, Contrastive Representa- tion Distillation (CRD) [43]. It is important to note that our method focuses on improving KD by using better inputs, while CRD improves KD at the outpur end (i.e., a better loss function). Therefore, they are orthogonal and we will show they can be combined together to deliver even better results.
Hyper-Parameter Settings. The temperature of knowledge distillation is set 4 following CRD [43]. Loss weight a = 0.9 (Eq. (1)). For CIFAR 100 and Tiny ImageNet, training batch size is 64; the original number of total training epochs is 240, with learning rate (LR) decayed at epoch 150. 180, and 210 by multiplier 1. The initial LR is 0.05, All these settings are the same as CRD [43] for fair comparison. Note, in our experiments we will present the results of more training iterations. If the number of total epochs is scaled by a factor k the epochs after which learning rate is decayed is also be scaled by k. For example, if we train a network for CIFARION 100 for 480 epochs (A: total, the learning rate will be decayed at epoch 300, 360, and 420. We use PyTorch [31] to conduct all our experiments. For CIFAR 100, we adopt the pretrained teacher models from CRD for fair comparison. For Tiny ImageNet and ImageNet100, we train our own teacher models. For ImageNet, we use torchvision models following CRD [43].
in
Data Augmentation Schemes. We investigate the following popular DA schemes:
Identity: This augmentation scheme simply makes a copy of each batch data during training, which should be the lower bound of all DA schemes discussed here since it adds new information.
. Flip: Random horizontal flip.
"htpss/ttiny-mmageeettnnooooor htw.://ggggenttteeouuuuuu.pooooo
7

---

Random horizontal flip and random crop. This is the standard DA extensively used in 2D image recognition task (such as on CIFAR and ImageNet datasets).
Flip+Cropp
Cutout [11]: Cutout occludes small random patch of an image.
AutoAugment [7]: AutoAugment is an ensemble of a collected DA schemes. The DA policy is automatically selected by reinforcement learning instead of manually.
Mixup [51]: Mixup applies linear interpolation between two inputs and applies the same linear interpolation to their labels to make the new label for the augmented sample.
CutMix [48]: CutMix cuts a small patch from source image and pastes another source image. The resulted image is considered as new input. The target for the new input is linear interpolation from the two source labels.
5.1 Empirical Verification of Our Proposition
Before presenting results, one point worth mentioning is that, in this section we use test loss instead of accuracy as the measure of student's performance KD, because (1) all the formulas in Sec. 3 are derived using numerical loss instead of accuracy; (2) more importantly, it is observed that accuracy can mismatch with loss - a model may achieve a better accuracy, meanwhile higher loss too [27], which we also observe several times in our experiments. Using accuracy as measure would prevent us from seeing the correlation between T. stddev and student's performance (see Appendix Fig. ?? for example). Potential extension of our theory from loss to accuracy is left for future work.
In Fig. 4, we plot the scatters of S. test loss and T. stddev. The x/y-axis value of each data point is averaged by at least three random runs (see Tabs ??/?? and Tabs ??/?? for detailed numbers).
(1) In terms T. stddev, there is rough trend (e-g., on the 13/vgg8 pair CIFAR 100): Identity < Flip < Flip+Crop < Cutout < AutoAuugment < Mixup < CutMix. These inequalities are well- aligned with our intuition. E.g., AutoAugment [7] includes Cutout [1]] in its transformation pool, thus should be stronger than Cutout. This is faithfully reflected by the T. stddev on many pairs.
(2) Obviously, S. test loss poses clear positive correlation with T. stddev. Per our theory, lower T. stddev should lead to better generalization risk for the student. This generally well verified in these plots. We do see some minor counterexamples. Possible reasons are: 1) The test loss is obtained on the test set with finite samples, which is only an approximation of the true risk defined on distribution; 2) The teacher's mean probability is also evaluated on finite data. Despite them, the general picture from Fig. 4 still confirms the positive correlation between S. test loss and T. stddev. The correlation is actually very significant the p-values indicate.
(3) Importantly, note we only need the teacher to define the quality of a certain DA in KD, no need for the student. This is more clear if we examine the results in Tabs. ??/?? and Tabs. ??/??. Taking vgg13/vgg 13vvgg8 and vgg 3//MobileNetV2 as an example, the students are starkly different but both the student's 's performance correlates well with T. stddev. This observation implies that the "goodness" of DA in KD is probably decide which DA should be used for the best performance simply using formula (Eq. (10)) with few network forwards, without having to train the students physically.
iintenntniaaiantt.. This, notably, is a great advantage in practice since we can
5.2 Boosting KD with Stronger DA
In this section, we present more results to show how the proposed theory can benefit in practice we can harvest considerable performance gain simply by using stronger DA in KD.
Prolonged Training. Notably, a stronger DA produces more diverse data, implying more information. Intuitively, it should take student model more training iterations (if the batch size does not change) to fully absorb the excessive information. That is, a stronger DA conceivably takes more training iterations fully exhibit its potential. This intuition is confirmed in Fig. 2 (and also more pairs wrn 40 1a_2//rn__aaa_ 16_ 2 and vgg 1ggl3//gg8 in the Appendix). Thus, in our experiments, we will also report results with prolonged training iterations for maximized performance.
Results on CIFAR100. The results on CIFAR100 dataset are shown in Tab.
(1) Comparing the row "KD+CurMix" to "KD", we see CutMix improves the student accuracies on all the pairs. On the pair resnet32x4/Shuffee22 the improvement very significant (more than percentage point). (2) Comparing the row "KD+CutMixPick" to "KD+CutMix", we see 6/7 pairs
8

---

wmn 11me_40__2 110140022210 2/wmm 2, CIFARIO0
resmet566/rsnee20, CIFAR100
Henile
Madiy
Plp
Ppp
1.16
Chep Fip
A-test loss
Calcat
$ kest loss
Groar
1.14
Masp
Cushie
Cu ox))
(podue 6,000)
1.12
OurmisPass an)
(p gewele 3.02%)
1.00
Syarmar 8 9567 (p-oster 0:001))
ChrmisPscc an)
Spuarner o 9135 gpoolla (ong)
1.10
ip (p-stae
0.00425 0.00450 0,00500
0.00525 0.00550
0.00375 0.00400 0.00255 0.00450
0.00500 0.00525
stidev
siddev
vgg 13/vg88 CIFAR100
RosNet50 CIFAR100
Mintay
ldeerry
Fiiz
1.2
Rip
tost loss
test
1.10
Misap
Aunakamann
Maug
Cuddii
Cutia
1.05
Cutmashik 15 cnt)
Penman: 8:9742 (ponice 9.0261
Cutmichik (8
Peanuz: 8,959 genaie RU009)
CatmiaPik vnl)
(p-elax 8.089
Catsichid
6.983 gerale 0.02%)
1,00
Kendal:: 1008 (p 6,90%)
1.0
Keddll 8.944
guolle 6.8099)
0.00400 0. 0.00425 0.004500 00475 0.00500 0.00525 0.00550
0.0044
0.0046
0.0048
0.0050 0.0052 0.0054 T. stddev
T. stidev
40 2/wm 16 2, Tiny ImageNet
resnet56/reamet20, Tiny ImageNet
1.96
1.76
Fli
Chop thlp
1.94
Lipp tlip
test loss
1.74
text loss
1.72
Mecep
1.92
Meap
QuasiaPicis
Pranser 9671 paller (.ee%)
1.90
Cuanisfici
08444 1869)
1.70
Chrmusttaee (I mn)
Speamar 9ee pponter i (994)
Crrmsthee (1) ax)
Speammn: IL7933 (p satar 25%0
Kendall: 7.7778 sa-sate (.2%))
Kendall rubee (p-vlte: 1.27%)
1.88
0.0024
0.0026
0. 0.0028
0.0050 0.0032 0.0034
0.0020
0.0022
0.0024
0,0026
0.0028
0.0050
T. siddev
T. stddev
vgg 33vvgg, Tiny ImageNet
ResNet50/vggR, Tiny ImageNet
Merstty
Ideerryy
1.80
Pip
llip
Fiip
Cuptitp
Teat loss
Crrerr
Iest Joss
Cannt
1.75
Antrongment
1.70
OurMix
Mixea
1.8
Maaz
Dakis
Chrmisthcc ds am)
Frmmer 987H y valur (.66))
Cammchdd (S ex)
Pureer 9.945 ip mo-oalue (181%))
1.65
Chrmanghee (1: an))
Spexmar
0.560 jp-silc 2.02%)
1.7
Chamachd
Spmeran: 1,0000 poaatee 80%)
10000 iperalas 0.829%)
0.0028 0.0030 0.0032 0.0034 0.0036 0.0038
0,0030
0.0032
0.0034
0.0036
0.0038
siddev
T. stddev
Figure 4: Scatter plots of T. stddev S. test loss of different pairs CIFAR100 and Tiny ImageNet. The detailed numbers are deferred to Appendix (Tab. ??, Tab. ??, Tab. ??, Tab. ??).
improved further, showing the proposed data picking scheme works in most cases. (3) Finally, "KD+CutMixPick" scheme can be combined with more training iterations (960 epochs), which delivers even higher accuracies.
(4) If comparing best results (KD+CutMixPickeen) with those of CRD (though this is not an apples-to-apples comparison since the two methods focus on different aspects to improve KD), can our approach outperforms CRD 6/7 pairs.
In the last two rows of Tab. 1, when CRD [43] is armed with our proposed "CutMixPick" and more training iterations, its results can be further advanced consistently. This demonstrates that our method is general and can readily work with those methods focusing on better KD loss functions.
Results on Tiny ImageNet. We also evaluate CutMix and CutMixPick on more challenging dataset, Tiny ImageNet. Similar to the case on CIFAR100, we have results on different teacher-studee pairs, shown in Tab. 2. For prolonged training, we train for 480 epochs instead of 960 to save time. Most claims on the CIFAR 100 dataset are also validated here: (1) "KD+CutMix" better than KD, which is verified on all pairs. (2) "KD+CuuMixPick" is better than "KD+CutMix", verified on 6/7 pairs. The exception pair is resnet56/resnet20, where adding data picking decreases the accuracy slightly by 0.11%. (3) When "KD+CuuMixPick is trained for prolonged iterations, the students perform best.

---

Table 1: Student test accuracy CIFAR100. Each result is obtained by 3 random runs, mean (std) accuracy reported. The best results are in bold and second best underlined. The subscript 960 means the total number of training epochs (default: 240).
Teacher Student
wm 40.2 2
resnet56
resnet32x4 resnet8x4
1g vgg8
1gg MobileNetV2
ResNet50 vgg8
resnet32x4
wm_16_2
resnet20
V2
Teacher Acc.
75.61 73.26
72.34
79.42
74.64
74.64
79.34
79.42
Student Acc.
69.06
72.50
70.36
64.60
70.36
71.82
KD [16]
74.92 (0.28) 70.66 (0.24) 73.33 (0.25) 72.98 (0.19) 67.37 (0.32)
73.81
(0.13) 74.45 (0.27)
KD+CutMix
75.34 (0.19) 70.77 (0.17) 74,91 (0.20) 74. (0.18)
68.79 (0.35)
74.85 (0.23) 76.61 (0.18)
KD+CutMixxPck
75.59 (0.22) 70,99 (0.20) 74.78 (0.35) 74.43 (0.20)
69.49 (0.32)
74.95 (0.18) 76.90 (0.25)
KDsea [16]
75.68 (0.12) 71.79 (0.29) 73.14 (0.06) 74,00 (0.34)
68.77 (0.05)
74.04 (0.25) 74.64 (0.30)
KD+CutMiixPickeen
76.41 (0.10) 71.66 (0.15) 75.12 (0.18) 75.00 (0.17)
70.47 (0.12)
76.13 (0.16) 77.90 (0.30)
CRD [43]
75.64 (0.21) 71.63 (0.15) 75.46 (0.25) 74.29 (0.12) 69.94 (0.05)
74.58 (0.27) 76.05 (0.09)
CRD+CutMixPick
75.96 (0.27) 71.41 (0.26) 76.11 (0.53) 74.65 (0.12)
69.95 (0.22)
75.35 (0.22) 76.93 (0.11)
CRD+CutMixPicckco
76.61 (0.01) 72.40 (0.20) 75.96 (0.29) 75.41 (0.10)
70.84 (0.05) 76.20 (0.22) 78.51 (0.27)
Table 2: Student test accuracy on Tiny ImageNet. The subscript 480 means the total number of
training epochs (default: 240).
Teacher Student
wmm_0__
resnet56 resnet20
resnet32x4 resnet8x4
1
1g MobileNetV2
ResNer50 vgg8
32snet32x4 V2
wm_16_2
vgg8
Teacher Acc.
61.28
58,37
64.41
62.59
62.59
68.20
64.41
Student Acc.
58.23
52.53
55.41
56.67
58,20
56.67
62.07
KD [16]
58.65 (0.09) 53.58 (0.18) 55.67 (0.09) 61.48 (0.36)
59.28
(0.13)
60.39
(0.16) 66.34 (0.1
KD+CutMix
59.06 (0.18) 53.77 (0.33) 56,41 (0.04) 62.17 (0.11)
60.48 (0.30)
61.12 (0.18) 67.01 (0.30)
KD+CutMixPicc
59.22 (0.05) 53.66 (0.05) 56.82 (0.23) 62.32 (0.18)
60.53
61.40 (0.26) 67.08 (0.13)
KD [16]
59,20 (0.30) (0.24) 55.49 (0.11) 61.72 (0.10)
59.27 (0.08) 60.10 (0.30) 65.81 (0.11)
KD++CutMixPickeen
60.07 (0.04) 54.25 (0.07) 57.54 (0.23) 62.60 (0.25)
60.66 (0.15)
61.95 (0.14) 67.35 (0.21)
CCR [43] I
60.79 (0.24) (5.04(0000) 59.28 (0.13) 62.92 (0.31)
62.3800.19
62.03 (0.16) 67.33 (0.13)
CRD+CutMixXick
60.72 (0.09) 54.99 (0.16) 59.65 (0.24) 63.39 (0.10)
62.54 (0.22)
62.85 (0.18) 67.64 (0.18)
CRD+CutMixPick.ss
60.99 (0.33) 55.68 (0.22) 60.13 (0.13) 63.60 (0.20)
62.79 (0.03)
62.60 (0.17) 67.70 (0.35)
We further evaluate our DA methods equipped with CRD [43], shown in the last two rows of Tab. 2. Our "CutMixPick" method further advances the prior SOTA on 5 pairs. When CRD+CuuMixPick is trained for 480 epochs (instead of 240), further improvement can be observed on 6 of 7 pairs.
Notably, we achieve the above performance boosting simply using the original KD loss [16], with no bells and whistles. This justifies one of our motivations in this paper, i.e., existing KD methods [32, 29, 43] mainly improve KD at the network output side via better loss functions, while we propose to improve KD at the input side and show this path is just as promising. Actually, this performance boosting effect is generic 5 top- top-performing KD methods on CIFAR AT [50]. CC [32], SP [44], PKT [30], and VID [1]. All the pairs see accuracy gains; half of them even improved by more than 1% point.
Apply DA to More KD Methods.
also applied CutMix to another
Results on ImageNet100 and ImageNet. These results are deferred to Appendix ??. In general, we observe that the correlation between T. stddev and S. test loss become weaker on ImageNet1 00 and ImageNet. This is because these two datasets are inherently harder than CIFAR 100 and Tiny ImageNet. Nevertheless, the v-luuue of the correlation is still below 5% on ImageNet100, i.e., still statistically significant, suggesting our theory can generalize to large-resolution (224 x 224) datasets.
6 Conclusion
In this paper, we attempt to precisely answer what makes a good data augmentation in knowledge distillation. By analyzing the generalization gap of the student under different sampling schemes, we reach the conclusion that good data augmentation scheme should reduce the variance of the cross-entropy (i.e., the distilled risk) between the teacher and student. Based on this, we propose a new metric, the stddev of the teacher's mean probability (T. stddev), as a feasible measure of the quality of data augmentation techniques. Empirical studies with various teacher-student pairs confirm the efficacy of the proposed metric for data augmentation quality KD. Besides the theoretical understanding, we also develop an entropy-based data picking scheme to further enhance the prior best augmentation scheme (CutMix) in KD Finally, we show how we can obtain considerable KD performance gains simply by using stronger DA guided by the proposed theory.
10

---

Acknowledgments and Disclosure of Funding
We thank the anonymous NeurIPS reviewers for giving us very helpful suggestions to improve this paper!
This work originates from Huan's 's internship at MERL, and is finished eventually after he returns to Northeastern University as a research assistant. This work is thus fully supported by MERL and Northeastern University. There is third-party funding or support in any form. There competing interests to disclose.
References
[1] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Varia- tional information distillation for knowledge transfer. In CVPR, 2019. 10
[2] Lucas Beyer, Xiaohua Zhai, Amélic Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In CVPR, 2022.
3,
[3] Xavier Bouthillier, Kishore Konda, Pascal Vincent, and Roland Memisevic. Dropout as data augmentation. arXiv preprint arXiv: 550.08870, 2015. 2
[4] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizii. Model compression. In
SIGKDD, 2006. 3
[5] Keshigeyan Chandrasegaran, Ngoc-Truun Tran, Yunqing Zhao, and Ngai-Man Cheung. Revisit- ing label smoothing and knowledge distillation compatibility: What was missing? In ICML,
2022.
[6] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. NeurlPS, 2017. 2,
[7] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe Le. Autoaugment:
Learning augmentation policies from data. In CVPR, 2019. 3,
[8] Wanyun Cui and Sen Yan. Isotonic data augmentation for knowledge distillation. arXiv preprint arXiv:2107.01412, 2021.
[9] Deepan Das. Haley Massa, Abhimanyu Kulkarni, and Theodoros Rekatsinas. An empiri- cal analysis of the impact of data augmentation on knowledge distillation. arXiv preprint
arXiv:2006.038/ 038700 2020. 3
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 3,
(11) Terrance De and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017. 8
[12] Yushu Feng, Huan Wang, Haoji Hu, and Daniel Yi. Triplet distillation for deep face recognition. In ICML Workshop, 2019. 2
[13] K. Hc, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, 2016. 3,
[14] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV, 2019. 3
[15] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In AAAI, 2019. 3
[16] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NeurIPS Workshop, 2014. 2, 10
[17] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural
Computation, 14(8): 2002. 2
[18] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In CVPR, 2017. 2
[19] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. 2

---

[20] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 1909. 10351, 2019. 2,3
[21] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[22] Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
[23] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 (7553))33
2015.
[24] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In JCML, 2016. 2
[25] Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In CVPR, 2019. 3
[26] Ningning Ma. Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. ECCV, 2018.
[27] Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In ICML, 2021. 4, 8
[28] Rafael Müller, Simon Korbblith, and Geoffrey Hinton. When does label smoothing help? In NeurlPS, 2019. 2,
[29] Wonpyo Park. Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019. 3, 10
[30] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018. 3, 10
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, pegf-peeformance deep learning library. In NeurlPS, 2019. 7
[32] Baoyun Peng, Xiao Jin, Jiaheng Liu. Dongsheng Li. Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In ICCV, 2019. 2, 3, 10
[33] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015. 3
[34] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. 7
Chich Chen.
[35] Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85-117, 2015. 2
[36] Burr Settles. From theories to queries: Active learning in practice. In AISTATS Workshop
Active Learning and Experimental Design, 2011. 1.11.7
[37] Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, and Marios Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study. In ICLR,
2021.
[38] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep
learning. Journal of Big Data, 6(1):60, 2019. 2,3
[39] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[40] Jy-yong Sohn, Liang Shang, Hongxu Chen, Jaekyun Moon, Dimitris Papailiopoulos, and Kangwook Lee. Genlabel: Mixup relabeling using generative models. In ICMI., 2022.
[41] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, 2016. 2
[42] Jiaxi Tang, Rakesh Shivanna. Zhe Zhao, Dong Lin, Anima Singh, Ed Chi, and Sugar Jain. Understanding and improving knowledge distillation. arXiv preprint 2rXh::2002.03532, 2020. 2020.
3
[43] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In
ICLR, 2020. 3, 7, 9, 10
12

---

[44] Frederick Tung and Greg Mori. Similarity-peseerving knowledge distillation. In CVPR, 2019.
10
[45] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICMI, 2019.
[46] Huan Wang. Yijun Li, Yuehai Wang. Haoji Hu, and Ming- Ming-Hsuan Yang. Collaborative distillation for ultru-resolution universal style transfer. In CVPR, 2020. 2,
[47] Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. TPAMI, 2021. 2,
[48] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In ICCV, 2019. 3,
[49] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016. 7
[50] Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the performance of convolutional neural networks via attention transfer. In /CLR, 2017. 3, 10
[51] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018. 2, 3,8
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See our Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See our Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] See Sec. 3.
(b) Did you include complete proofs of all theoretical results? [Yes] See Sec. 3.
3. If you experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either the supplemental material or as a URL)? [Yes] See our Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See our Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] All our
ImageNet//mageNett00
results are averaged by at least three random runs, mean and stddev reported.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See our Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See our Appendix.
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
We include the code link.
(d) Did you discuss whether and how consent was obtained from people whose data you're vou're using/curating? [Yes] The data we use are all publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] The data we use contains no personally identifiable information offensive content.
13

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [No] No crowdsoureing in this work.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [No] No crowdsourcing in this work.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [No] No crowdsourcing this work.
14


Reviews:
[{"review_id": "jZtCYAS3H9", "reviewer": "Reviewer_VLkK", "summary": "The major contributions of this paper are 2-fold:\n1) The authors suggest that given a fixed teacher model, a good data augmentation (DA) scheme is characterized by a lower variance of the teacher’s mean output probability. This claim is supported by statistical analysis and empirical results (image classification).\n\n2) An entropy-based sample picking scheme is introduced to synthetically reduce the variance of the teacher’s mean probability. This scheme (CutMixPick) further improves the results.", "questions": "Please see Weaknesses section above for a list of all questions.\n\nRemark : I managed to only study the main text. Unfortunately, I only skimmed through the supplementary, but didn't study in detail due to its length / my time-constraints.", "limitations": "The authors have discussed limitations / potential societal impacts in Supplementary Section 6.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1) Setups with lower variance (although it is not clear how $u$ is calculated, see weaknesses below) obtains acceptable improvements in CIFAR-100 and Tiny-ImageNet experiments.\n\n\n**Weaknesses:**\n\nThough this work is executed well and the image classification results show improvements, the novelty and significance of this work are limited. See below:\n1) The clarity of Section 3.2 can be improved. Though proposition 1 seems interesting, I have questions regarding the Std of the teacher’s mean output probability. It is unclear as to how $u$ in equation 9 is calculated. Particularly:\n- What is $S^*$ and how is it determined? Why is $K=640$?\n- Would it be reasonable to calculate $m$ using the variance of the output probabilities of the teacher **for the correct class** over all samples?\n2) Experiment coverage / results are limited: \n- Particularly, ImageNet results are not discussed and analyzed. Not that I have anything against CIFAR-100, Tiny-ImageNet analysis, but I tend to think that ImageNet results are critical to add merit to the findings (1000 classes, higher resolution:224 x 224).  Most KD works report results on ImageNet-1K ([1, 2, 3, 4, 5]). If compute is a problem, authors can consider using a smaller dataset for experiments: ImageNet-100 (A subset of ImageNet popular in the self-supervised learning community). I believe the authors can also use publicly available pre-trained ImageNet models if required: https://pytorch.org/vision/stable/models.html\n- What is the reason for not including Flip, Flip+Crop, Cutout, Auto-Augment, Mixup results in Tables 3, 4?\n3) For ‘Std of mean prob’ in Tables 1, 2, is it reasonable to use the training set as the training set is used for distillation (not the test set)?\n4) For CutMixPick experiments, what is the percentage of the training set used for distillation (after 'picking' / filtering)?\n\nOverall I enjoyed reading this paper. In my opinion, the weaknesses of this paper outweigh the strengths. But I’m willing to change my opinion based on the rebuttal. \n\n\n=====================\n\n[1] Müller, Rafael, Simon Kornblith, and Geoffrey E. Hinton. \"When does label smoothing help?.\" Advances in neural information processing systems 32 (2019).\n\n[2] Shen, Z., Liu, Z., Xu, D., Chen, Z., Cheng, K. T., & Savvides, M. (2021). Is label smoothing truly incompatible with knowledge distillation: An empirical study. In ICLR\n\n[3] Chandrasegaran, K., Tran, N. T., Zhao, Y., & Cheung, N. M. (2022). Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?. ICML\n\n[4] Heo, Byeongho and Kim, Jeesoo and Yun, Sangdoo and Park, Hyojin and Kwak, Nojun and Choi, Jin Young (2019). A Comprehensive Overhaul of Feature Distillation. ICCV\n\n[5] Tang, Jiaxi, et al. \"Understanding and improving knowledge distillation.\" arXiv preprint arXiv:2002.03532 (2020).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eaVXVhySeUk", "reviewer": "Reviewer_n9FN", "summary": "The paper establishes a connection between data augmentation (DA) and generalisation of student-teacher training, namely showing theoretically that given two subsequences of the original dataset D (where the subsequences differ by some data augmentation scheme), the subsequence that performs better is the one that induces less variation in the predicted probabilities of the teacher network.\n\n**Edited review**\n\nIn light of the authors _active discussion and contributions_ during this phase, as well as their addressal of my own concerns, I will tentatively raise my score from 4 to 7. Obviously this is conditional on some changes to the manuscript, namely:\n- Making the math more digestible (we already discussed this in great detail)\n- Preferring plots to large tables where applicable, or having plots supplement them. I leave this to the authors' discretion since space constraints will mean decisions will have to be made as to what should be a table vs what should be a plot, and what should be relegated to the appendix.\n- Further work on plotting Equation 8 vs its approximation vs both 'bad' and 'good' students. The authors have acknowledged this and even provided preliminary results.\n\nThanks, and good work.", "questions": "My questions mainly lie in the aforementioned weaknesses.", "limitations": "n/a", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- The efficacy of any DA scheme for student training could in principle be evaluated before the training of any student model, since computing the variance term $V[\\Delta]$ only requires the pretrained teacher model.\n\n- Experiments support the proposed theory, and are numerous, run over many different architectural variants as well as data augmentation schemes.\n\n- Proposal of an entropy-based scheme to improve results.\n\n# Weaknesses\n\nSome of the notation in Section 3.1 and 3.2 is incredibly confusing, and looks to be rushed. Firstly, I would recommend simply using $\\log$ instead of $\\mathcal{L}$ to denote the logarithm. Alternatively, if you want to be more abstract, you can simply define the cross-entropy loss as $\\mathcal{L}_{CE}(x, y; f) = y^T f(x)$ and it would make equation (6) more readable. In fact, you have already defined it as such in Equation (1).\n\nIn Section 3.2, I have several concerns. In Equation (6), in line 1 you have an expectation $S \\sim D^N$, yet in line 2 it confusingly turns into an expectation $x \\sim D$, yet you still have the inner summation. Why isn't it just this instead:\n\n$E_{x \\sim D} [ p(x) \\log f(x) ] + \\text{const}$\n\ni.e. the expectation implies the inner summation + multiplication over $1/N$, so having both seems redundant (and incorrect). Furthermore, in the inner multiplication you're missing the transpose, so it should be $p(x_n)^{T} \\log f(x_n)$, since that is how it is defined in equation (4).\n\nIn Equation (7), I am also not sure why the last line expands out into a variance + covariance term. Covariance to me implies that you're computing the variance between two sets of random variables; don't we only have one random variable? (That random variable being $\\Delta$, the empirical risk over whatever random draw $S \\sim D$.) Maybe I am missing something crucial, but it seems like one could just remove the last line of Equation (7), leaving us with:\n\n$V_x[ p(x)^T \\log f(x) ]$,\n\nthen say that the variance is going to be proportional to the $p^{(t)}$ and that $f$ is held constant. Actually, it seems like from Equation (9) that the covariance is over the $p$'s for the different classes. What is confusing is that, for a given $x$, you are indexing into its predicted probability for class $i$ as as $p^{(t)}(x_i)$, and $x_i$ is apparently the $i$'th example as you have implied in equations (3) and (4). If $p^{(t)}$ outputs a probability distribution over x, then you should be using $p^{(t)}(x)_i$ instead. \n\n\"Each element in S is drawn the same underlying distribution D.\" - I don't follow this, since it says earlier that $S_1$ and $S_2$ are elements from $D$ but not necessarily iid. If so, how can $E_{x \\sim S}$ turn into $E_{x \\sim D}$?\n\nTable 1: I appreciate that the results are averages over three runs, but please indicate the standard deviation. Since space seems a little tight in this table, maybe just indicate the stdevs below each number in parentheses. Also such information can be nicely condensed into a scatterplot with one axis denoting test accuracy and the other denoting mean stdev. When necessary, prefer plots over big tables, and leave big tables to the appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ClNwppdtx6z", "reviewer": "Reviewer_Tm5Y", "summary": "The paper presents a theoretical development of a straightforward to use measure of effectiveness of data augmentation techniques for distillation. It proposes that this measure need only be based on an analysis of the teacher, without consideration of the student. The paper also presents an enhanced version of the cut-mix data augmentation technique, which performs a distillation of the augmented dataset to maximize the information content of the data.", "questions": "Non-data manipulations, such as drop-out, can result in the same behavior as shown in figure 2a (of spreading different soft targets while keeping the same hard targets).\nThe paper (Bouthillier, X., Konda, K., Vincent, P., and Memisevic, R. Dropout as data augmentation. arXiv preprint arXiv:1506.08700, 2015.) even considers the dropout as a type of data augmentation. So, could your approach be made even more general (than the few types of augmentation you consider)?\n\nIn lines 127-129, why is a script L used to indicate the log function, rather than the more usual log() notation? Script letters are conventionally used to indicate spaces or sets. In line 139 the script L is used to indicate a loss function, adding to the confusion.\n\nIn line 154-156 the assumption is made of a \"well-trained\" student, that gives softmax outputs \"pretty equal\" to those of the teacher. However, in most practical applications of distillation, the student can be much smaller than the teacher (e.g. for network compression for edge devices). What happens to your conclusions in that case? This assumption is used to make the claim that we need only look at the teacher to judge the effectiveness of a DA method. So, is this claim still true if the assumption does not hold?\nIn table 1, (as was done in table 3) you should give the teacher and student test accuracies to see how well the \"well-trained\" assumption holds. For CIFAR-100 and the student/teacher networks used, I would expect a significant difference in softmax outputs (e.g. 74% for ResNet56 and 69% for ResNet20).\n\nIt seems surprising that an analysis of the student has no role to play in determining the effectiveness of a data augmentation technique that is being used to train that student. If no distillation was being done at all (just training using cross-entropy on the hard targets), DA still would be of some use, no? So, this suggests that one cannot just ignore the student in determining the effectiveness of the DA method. Perhaps, in light of the results in table 1, the teacher's effects dominate the student's effects in terms of judging DA performance, but the student may still have some influence. It would be nice to have some theoretical insight into why this is, rather than just throwing away the student based on the \"well-trained\" assumption.\n\nDoes the \"pick\" aspect of the Cut-Mix-Pick help with the other DA methods? Presumably doing the data distillation on the other augmentations would also improve their results.\n\nIn section 5.1 the authors say that they do not look at accuracy, as the derivations are based on error, and sometimes a network can give high accuracy but also high error. But accuracy is typically what people are interested in when using networks, not the value of the loss function. It may be that by focusing on loss (error) one is biasing the student to become more confident, like the teacher, and perhaps overly confident, leading to reduced accuracy. In any case, table 3 shows accuracies, so I am not sure why this was not done in section 5.1.", "limitations": "There is no discussion of limitations and potential negative societal impacts. However, distillation techniques, to the extent they can be used to train smaller networks, may positively affect societal impacts related to high power usage during training and inference time.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method yields an informative measure that, empirically, can be reliably used to judge the efficacy of different data augmentation schemes, at least for the methods considered.\n\nThe justification and derivation of the proposed measure is reasonable, but it (possibly) depends on an overly strong assumption on the student to be \"well-trained\". This may not be the case in some applications where the student has a much lower capacity than the teacher. However, the proposed measure may still be effective, but would need more detailed theoretical justification to show this.\n\nThe cut-mix-pick data augmentation technique demonstrates improved performance over the standard cut-mix method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0tPoRCfEnw", "reviewer": "Reviewer_JbXW", "summary": "Knowledge distillation (KD) is to improve a low-capacity student network's performance with a high-capacity teacher network. Current works focus on how to define KD loss. This work proposes to improve KD from data augmentation (DA). This work proposes an entropy-based data-mixing DA to improve two KD methods: KD and CRD.", "questions": "The main concern is the novelty of the proposed methods (see weakness). ", "limitations": "The authors have addressed the the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The method is simple and clearly presented.\n2. The experiments are done on different data augmentation techniques.\n3. Results on CIFAR-100 and TinyImageNet show increased performance.\n\nWeaknesses:\nThere are three contributions in the method part: (1) data augmentation (2) longer training and (3) cutmixpick. (1) and (2) are already investigated in works [1][2]\n[1] Cui, Wanyun, and Sen Yan. \"Isotonic Data Augmentation for Knowledge Distillation.\" IJCAI2021\n[2] Beyer, Lucas, et al. \"Knowledge distillation: A good teacher is patient and consistent.\" CVPR2022.\nThe improvement from (3) is very minimal.\n\n\nAfter rebuttal:\nThe authors have addressed my concerns. I increased my score to boardline accept after considering other reviewrs' comments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept