Role-Wise Data Augmentation for Knowledge Distillation
Authors: Jie Fu
Xue Geng
Bohan Zhuang
Xingdi Yuan
Adam Trischler
Jie Lin
Vijay Chandrasekhar
Chris Pal
Venue:  (2020)
Keywords: Data Augmentation
Knowledge Distillation

Abstract: Knowledge Distillation (KD) is a common method for transferring the ``knowledge'' learned by one machine learning model (the teacher) into another model (the student), where typically, the teacher has a greater capacity (e.g., more parameters or higher bit-widths). To our knowledge, existing methods overlook the fact that although the student absorbs extra knowledge from the teacher, both models share the same input data -- and this data is the only medium by which the teacher's knowledge can be demonstrated. Due to the difference in model capacities, the student may not benefit fully from the same data points on which the teacher is trained. On the other hand, a human teacher may demonstrate a piece of knowledge with individualized examples adapted to a particular student, for instance, in terms of her cultural background and interests. Inspired by this behavior, we design data augmentation agents with distinct roles to facilitate knowledge distillation. Our data augmentation agents generate distinct training data for the teacher and student, respectively. We focus specifically on KD when the teacher network has greater precision (bit-width) than the student network.

We find empirically that specially tailored data points enable the teacher's knowledge to be demonstrated more effectively to the student. We compare our approach with existing KD methods on training popular neural architectures and demonstrate that role-wise data augmentation improves the effectiveness of KD over strong prior approaches. The code for reproducing our results will be made publicly available.

Under review as a conference paper at ICLR 2020
ROLE-WISE DATA AUGMENTATION FOR KNOWLEDGE DISTILLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge Distillation (KD) a common method for transferring the "knowl- edge" learned by machine learning model (the teacher) into another model (the student), where typically, the teacher has a greater capacity (e.g.. more param- eters higher bit-widths). To our knowledge, existing methods overlook the fact that although the student absorbs extra knowledge from the teacher, both models share the same input data and this data the only medium by which the teacher's 's knowledge can be demonstrated. Due to the difference in model capacities, the student may not benefit fully from the same data points on which the teacher is trained. On the other hand, human teacher may demonstrate a piece of knowl- edge with individualized examples adapted to a particular student, for instance, in terms of her cultural background and interests. Inspired by this behavior, we design data augmentation agents with distinct roles to facilitate knowledge distil- lation. Our data augmentation agents generate distinct training data for the teacher and student, respectively. We focus specifically on KD when the teacher network has greater precision (bit-width) than the student network.
We find empirically that specially tailored data points enable the teacher's knowl- edge to be demonstrated more effectively to the student. We compare our ap- proach with existing KD methods on training popular neural architectures and demonstrate that role-wise data augmentation improves the effectiveness of KD over strong prior approaches. The code for reproducing our results will be made publicly available.
INTRODUCTION
Background and Motivation. In the educational psychology literature, is generally considered beneficial if teachers can adapt curricula based upon students' prior experiences (Bandura, 2002; Brumfiel, 2005; Gurlitt et al., 2006; Slotta Chi, 2006). These vary widely depending on students' cultural backgrounds, previous educational experiences, interests, and motivations. To help students with different prior experiences to comprehend, memorise, and consolidate a piece of knowledge, teachers may provide extra and customized teaching material during their teaching processes. For instance, when teaching the concept of the color pink, teacher may choose flamingos, sakura (cherry blossoms),
cream cones as the example, depending on student's background.
Knowledge distillation (KD) (Bucilua et al., 2006; Hinton et al., 2014) is a common framework for training machine learning models. works by transferring knowledge from higher-capacity teacher model to a lower-capacity student model. Most KD methods can be categorized by how they define the knowledge stored in the teacher (i.e., the "soft targets" of training as defined in existing literature). For instance, Hinton et al. (2014) originally proposed KD for neural networks, and they define the output class probabilities (i.e., soft labels) generated by the teacher as the targets for assisting the training of students. In follow up work, Romero et al. (2015) defined the soft targets via the feature maps in the teacher model's s hidden layers.
To train student network with KD effectively, it is important to distill much knowledge from the teacher as possible. However, previous methods overlook the importance of the medium by which the teacher's knowledge is demonstrated: the training data points. We conjecture that there exist examples, not necessarily seen and ingested by the teacher, that might make it easier for the student to absorb the teacher's knowledge. Blindly adding more training examples may not be beneficial

---

Under review as a conference paper at ICLR 2020
because it may slow down training and introduce unnecessary biases (Ho et al., 2019). The analogy with how human teachers adjust their teaching to their students' particular situations (e.g.. with the feedback gathered from the students during teaching) suggests that reasonable yet uninvestigated approach might be to augment the training data for both the teacher and student according to distinct policies.
In this paper, we study whether and how adaptive data augmentation and knowledge distillation can be leveraged synergistically to better train student networks. We propose a two-stage, role- wise data augmentation process for KD. This process consists of: (1) training a teacher network till convergence while learning a schedule of policies augment the training data specifically for the teacher; (2) distilling the knowledge from the teacher into student network while learning another schedule of policies to augment the training data specifically for the student. It is worth noting that this two-stage framework is orthogonal to existing methods for KD), which focus on how the knowledge to be distilled is define;; thus, our approach can be combined with previous methods straighforwardly.
Although our proposed method can in principle be applied to any models trained via KD, we focus specifically on how to use it to transfer the knowledge from full-precision teacher network into a student network with lower bit-width. Network quantization is crucial when deploying trained models on embedded devices, in data centers to reduce energy consumption (Strubell et al., 2019). KD-based quantization (Zhuang et al., 2018; Polino et al., 201 8) jointly trains a full-precision model, which acts as the teacher, alongside a low-precision model, which acts the student. Previous work has shown that distilling full-precision teacher's knowledge into a low-precision student, followed by fine-tuning, incurs noticeable performance degradation, especially when the bit-widths are below four (Zhuang et al., 2018; Polino et al., 2018). We show that it is advantageous to use adaptive data augmentation to generate more training data for the low- low-precision network based on its specific weaknesses. For example, low -weeeccision networks may have difficulties learning rotation- related patterns,' and the data augmentation agent should be aware of this and generate more such data points. One positive side-effect for demonstrating the effectiveness of our method is that the improvement brought by our proposed method is more significant compared to the experiments on all full-precision models.
RELATED WORK
Knowledge distillation. KD is initially proposed for model compression, where a powerful wide/deep teacher distills knowledge to narrow/shallow student to improve its performance (Hin- ton et al., 2014; Romero et al., 2015). In terms of the definition of knowledge to be distilled from the teacher, existing models typically use teacher's class probabilities (Hinton al., 2014) and/or intermediate features (Romero et al., 2015; Park et al., 2019; Seunghyun Lee, 2019; Hyun Lee et al., 2018). Among those KD methods that utilize intermediate feature maps, Relational KD (RKD) considers (Park et al., 2019) the intra-relationshiy in the same feature map, while Multi- Head KD (MHKD)(Senghhyun Lee, 2019) and KD using SVD (KD-SVD) (Hyun Lee al., 2018) utilize the inter-relationship across feature maps. By contrast, we propose to incorporate both the intra- and inter-relationships within and across feature maps.
Automated data augmentation. Manually applying data augmentation rules such as random ro- tating, flipping, and scaling are common practices for training neural models on image classification tasks (Krizhevsky et al., 2012; He et al., 2016). Several recent works attempt to automate the data augmentation process. Generative adversarial networks (Ratner al., 2017) and Bayesian optimiza- tion (Tran et al., 2017) have been used for this process. DeVris & Taylor (2017) augment training data in the learned feature space by injecting noise and interpolation. Lemley et al. (2017) learn how to combine pairs of images for data augmentation. AutoAugment (Cubuk et al., 2018) searches for the optimal data augmentation policies (e.g., how to rotate) based on reinforcement learning. However, the search process is computationally expensive. Population-based augmentation (PBA) (Ho al., 2019) uses an evolution- based algorithm to automatically augmenting data in an efficient way. In contrast to previous approaches, we study the effect of the training data for KD and propose to use automatic data augmentation to train the student better from her teacher.
We will visualize the learned schedules of policies in Section 5.5.
2

---

Under review as a conference paper at ICLR 2020
3 PRELIMINARIES
OOPULATION-BASHI AUGMENTATION (PBA)
PBA (Ho et al., 2019), as an evolutionary search algorithm, learns a dynamic per-epoch schedule of augmentation policies, denoted A. Since this schedule is epoch-based, it will re-create the augmented dataset every epoch. More concretely, PBA begins with population of models that small subset of the original training data. The weights of the worst
trained in parallel performing models in the population are replaced by those from better performing models (i.e., exploitation), and the policies are mutated new ones within the pre-defined policy search space (i.e., exploration). After training, PBA usually keeps the learned augmentation schedule of policies but discards the elementary parameters of the models. different model (e.g.. larger one) can then use the learned schedule to improve its training on the same task.
3.2 KNOWLEDGE DISTILLATION (KD)
Following the notations in (Park al., 2019), KD method aims minimize the objective function
Egeneral Lusk A LKD.
(1)
where À is hyper-parameter to balance the impact of the KD loss term.
In this paper, for classification tasks, Luask Ex,xx (FF Mrruhh, where X' refers to training sample space, U/nuth E y are the ground-truth labels, Fs(-) is the student network, and H(()) denotes the cross-entropy.
The KD term can be defined as
CKD
Σ (TT(( Z,ER
where F(.) is the function of the network and I(.) is a loss function to compute the difference between the teacher network and the student network.
For KD methods that use soft labels (Hinton et al., 2014), the objective can be defined as
Sooft
Σ
H(softmax (x:)), softmax (Ffinal (x:))):
(3)
2.EX
where ((aaa)(x)) is the feature map of the final layer.
There exist some KD methods that utilize the intermediate feature maps in complementary ways. For example, Relational KD (Park et al., 2019) considers the intra-relationships. That is, given the feature map of layer j. the KD loss can be formulated as:
Cintra Σ ((FF)())) &EE
(4)
where (--) refers to the potential function measuring the pairwise relationship inside feature map from student network or teacher network and (x;) is the feature map of layer j (which may include the final logits layer). Therefore, this ceature-based KD method includes the benefits of using soft labels.
On the other hand, some works (Seunghyun Lee, 2019: Hyun Lee et al., 2018) consider the inter- relationships, where the KD term can be formulated as:
Cinter
Σ
(a)))
(5)
Z.EX
Here, (e()) measures the inter-relationship between feature maps of different layers, i.e. k + j.
3

---

Under review as a conference paper at ICLR 2020
Augmented Training Dala
Original Traiting Dala
A Augmentation
Teacher
Agent
Network
Feedback
Figure I: Concept diagram (stage-o) of training a data augmentation agent for the Ny.
3.3 QUANTIZATION
In this work, DoReFa (Zhou et al., 2016) to quantize both weights and activations. The quantization function Q(-) defined as:
Tg = Q())()
round 1))
(6)
where r is the full- precision value, ra indicates the quantized value, Thbiss refers to the number of bits represent this value. With this quantization function, the quantization on weights defined as:
tanh(r)
+22
1
W/g 2 Q(2 max (aax(((hh())
The back-propg prokkaapppggaiinnno is approximated by the straight-througe estimator (Bengio et al., 2013) and the partial gradient ör w.r.t. the loss is computed as:
01 Øl dr
Or
al
~0
(8)
Dr, q Ör drg
THE PROPOSED METHOD
Our proposed method has two stages, which will described the following subsections. In the first stage, train teacher network, denoted as Nr. with the help of PBA-based augmentation. In the second stage, we further distill the knowledge from My. (pre-trained in the first stage) to the student network, denoted as Ms, while learning another augmentation schedule augment the training data for Ns.
4.1 STAGE-Q
In general, a teacher can provide better training signals for the student the teacher's performance increases (Mirzadeh al., 2019). As shown in Fig. 1, we apply PBA to learn a dynamic per-epoch schedule of augmentation policies, Ar. for Nr on a small subset of training data. That is, the augmentation agent's 's training signal is defined as the feedback of Ny accuracy subset of the dataset. After this, we use the discovered schedule Ar to augment the whole training dataset and re-train, Nr on it till convergence.
4.2
STAGE-33
KD methods have shown be effective improving the performance of lower-capacity networks using the knowledge from higher-capacity networks. In order to take advantage of this functionality, apply the KD methods together with data augmentation in stage-9, as shown in Fig. 2.
More concretely, we first use PBA to learn an epoch-based augmentation schedule As for Ns on subset of the dataset. Different from the schedule Ar learned in stage-a, As is learned based on the feedback (i.e., accuracy) from Ns, which is trained with KD. In other words, A's receives additional training signals from Nr that is pre-trained in stage-a. We then use the learned As to augment the whole training dataset, and re-train Ns on it with the distilled knowledge from Ny. Note that, because the learned schedule epoch- based, we do not use the discovered schedule Ar from stage-o to augment the training data as initialization.
It should be noted that our proposed method orthogonal any particular quantization method.
4

---

Under review as a conference paper at ICLR 2020
Teacher
Network
Augmented Student Training
fixed)
Original Training Data
A Augmentation
Data
Knowleedge
A gent
Student
Network
Feedback
Figure 2: Concept diagram (stage-3) to augment training datapoints for both Nr and Ns. The NT has been pre-trained using the method shown in Section 4.1, and fixed during training. The augmentation agent in stage-8 is designed to learn schedules of polices that are different from those learned in stage-a, and thus the agent only receives the feedback from Ng.
When Vg is low-looeeee www-precision network, following (Furlanello et al., 2018), we share the same network architecture between Ny and Ns. When Ns is full-precision network, it will have fewer layers compared to Nr.
EXPERIMENTS
5.1 SETTINGS
We evaluate approach two benchmark datasets: CIFAR-10 (Krizhevsky al., 2009) and CIFAR- 100. CIFAR-10 10 consists of 60,000 32x color images in 10 classes, with 6000 images per class and CIFAR-100 has 100 classes with 600 images per class. Both have 50,000 training images and 10,000 test images.
We search over "reduced" CIFAR-IO/CIFAR-I with 4,000 training images and 36,000 validation images, which is the same as in (Ho et al., 2019). All the data augmentation models are run with 16 total trials to generate augmentation schedules. Following PBA, stage-a, we run PBA create schedules over separate models and then transfer the CIFAR-10 policy to CIFAR-100. However, for student network training in stage-8, we empirically use the respective "reduced" dataset. The data augmentation approaches for the baselines include random crop and horizontal flipping operations. Following (Ho al., 2019), our policy search space has total of 15 operations, each having two magnitude and discrete probability values. We use discrete probability values from 0% to 100%, in increments of 10%. Magnitudes range from to
The models we evaluate include AlexNet (Krizhevsky et al., 2012) and ResNet18 (He et al., 2016). The number epochs is 200 and the batch size is set to 128. For full-precision network NT. the learning rate starts from 0.1 and is decayed by 0.1 after every 30% of the total epochs. we use SGD with a Nesterov momentum optimizer. The weight decay is set to x 10-4 For quantization, the learning rate is set to 10-3 and is divided by 10 every 30% of the total epochs. We use the pre-trained teacher network model as the initial point of student network. We use a smaller weight decay 10-5 assuming that less regularization is needed by the lower-precision networks. Following DoReFa (Zhou et al., 2016), the first layer and last layer are not quantized.
Following Clark al. (2019), during training, we gradually transit the student from learning based on the teacher to training based on the ground-truth labels. This heuristic provides the student with more rich training signals in the early stage but does force the student strietly mimic the teacher's is behaviors. As for the implementation, we decay the balancing syper-parameter À in the KD loss by 0.5 every 60 epochs.
5.2 COMPARING DIFFERENT KD METHODS
As mentioned in Section 3.2, there exist complementary KD methods considering both intra- and inter-relationships within and across feature maps. natural question is if it would be beneficial
Note this is not a hard constraint, we choose such strategy reduce the number of factors that might influence the final performance.
5

---

Under review as a conference paper at ICLR 2020
to combine them to further boost the performance together with data augmentation. Therefore, propose a simple extension to these complementary KD methods, dubbed as II-KD, by incorporating intraa-elationnhips inside the feature map and inter-relationships across different feature maps. We incorporate the two relationships into the final objective function as follows:
R = Loiginnll +
(LDD
Cinker KD
)
(9)
where we only use single balancing Sayper-paramenn À between the original loss and the distillation loss, which does not introduce extra hyyee-parameter
More precisely, our KD method incorporates components of three conventional KD methods: RKD (Park et al., 2019), MHGD (Seunghyun Lee, 2019) and KD-SVD (Hyun Lee et al., 2018). As shown in Eq. (9), we add the three KD terms together with equal coefficients. We use the loss function I(.) following their approaches. For the back-propagation we clip the gradient for KD loss as in KD-SVD, because this will smoothly post-processes the gradient to limit the impact of KD loss in training. For AlexNet we select the feature maps of ReLU layers after the convolution/max pooling layer. For ResNet18, we select the feature maps of the last ReLU layer of cach residual block.
We evaluate our proposed KD extension CIFAR-100 with ResNet18 for different bit-width set- tings by comparing with various KD methods. For the baseline methods, we use their default settings with fixed and pre-trained teacher network in the training stage and À for the knowledge distil- lation loss. We set = 0.4 for II-KD in Eq. (9), as we have two KD terms. Tab. 1 reports the results on various augmented KD methods. We observe that our proposed methods clearly outperforms the other KD methods on all the settings, though the improvements over MHGD and KD-SVD are not huge. The results also reveal that only relying soft labels is not as effective as utilizing multiple supervising signals from the teacher.
Accuracy (%) on CIFAR-100 with ResNet18 using different KD methods. We compare ours with the following methods: Soft labels (Hinton et al., 2014), DML (Zhang et al., 2018), RKD (Park et al., 2019), MHGD (Seunghyun Lee, 2019) and KD-SVD (Hyun Lee et al., 2018).
Table I:
Bit- Width
Soft labels
DML
RKD
MHGD
KD-SVD
II-KD
(Weight/ / Activation)
4/4
70.48
72.47
71.84
73.52
73.92
74.21
2/2
70.09
69.72
70.71
71.80
72.97
73.35
5.3 Is ROLF- AUGMENTATION WITH KD EFFECTIVE FOR QUANTIZAATION?
In this subsection, we aim to answer this question: is our two-stage role-wise augmentation with KD effective for network quantization? We conduct experiments on CIFAR-10 and CIFAR-100 100 datasets under full- 4-bit, and 2-bit settings.
From Tab. 2, we can observe that training with learned data augmentation schedules does not im- prove the performance of low- cow-precision networks too much. Similar to the results obtained in (Zhuang et al., 2018), transferring knowledge from the full-precision to the low-precision student usually helps the training of students, which is especially obvious on the CIFAR-100 dataset. Tab. 2 also clearly shows that our proposed pipeline consistently improves the performance of the low- precision student networks. For example, the 4-bit Ng is comparable with aill-precision reference without loss of accuracy for CIFAR- 10 and with loss of accuracy within 1.0% on CIFAR-100. When decreasing the precision to 2-bit, the results are still promising as compared with other baselines, even though there is a performance gap between the 2-bit and the full-precision models. For in- stance, our approach usually outperforms the strong baseline, only using II-KD, by more than 1.0%.
5.4 COMPARING SCHEDULES
Here we aim to answer this question: how effective is it if we use Ar. learned based on the feedback from Nr in stage-a, to dynamically augment the training dataset and train N's on it? Tab, 3 reports the accuracy comparison with different KD methods and augmentation schedules. We can clearly see that augmenting the training dataset for Nss with As consistently outperforms those using the transferred schedules Ar among different KD methods. This observation is consistent with our assumption that As has her own optimal augmentation schedule, As, that is different from Ay for Nr. In particular, blindly applying the teacher augmentation schedule Ar may negatively influence

---

Under review as a conference paper at ICLR 2020
Table 2: Accuracy (%) on CIFAR-10 and CIFAR-100 datasets with different bit-widths. Vanilla Training for 4-bit and 2-bit refers to training network based on DoReFa (Zhou et al., 2016) from scratch without learned data augmentation. Teacher after Stage-a refers to using learned schedules discovered by PBA to re-Irain Nr described in Section 4.1. Student with only II-KD refers to training Ns using II-KD but without the learned data augmentation. Student after Stage-$ refers to training Nss using II-KD and the learned data augmentation. For Vanilla Training and Teacher after Stage-a, we report the accuracy Nr, and for the rest we report the accuracy N's.
AlexNet CIFAR-10
AlexNet CIFAR-100
ResNet18 CIFAR- 10
ResNet18
Methods
CIFAR-100
32-bit
90.58
65.80
93.57
74,85
Vanilla Training
4-bit
89.72
60.25
90.97
69.81
2-bit
88.77
58.96
90.00
67.06
32-bit
91.62
66.40
94,49
75.19
Teacher after Stage-o
4-bit
90.06
60.65
91.47
70.24
2-bit
89,28
58.59
89.99
67.32
4-bit
90.55
65.55
91.42
73.85
Student with only II-KD
2-bit
89.18
63.49
90.60
72.44
4-bit
92.00
65.69
94.44
74.21
Student after Stage-8
2-bit
90.63
64.06
93.20
73.35
the training of Ng compared to only using KD. For example, the learned schedule based the teacher Ar degrades the performance of N by 0.58% for AlexNet on CIFAR-100 as compared to applying KD methods, as shown in Tab. 2.
Table 3: Accuracy (%) on CIFAR- 100 with 4-bit networks using different KD methods
AlexNet MHGD
ResNet18 MHGD
Methods
DMI.
Ours
DMI.
IIKD
Schedule based on teacher
61.61
61.62
64.97
71.78
69.76
73.46
Schedule based on student
63.73
63.47
65.69
72.47
73.52
74.21
5.5 ANALYZING THE LEARNED SCHEDULES
To analyze the difference on the discovered schedules between Nr (i.e., full-precision ResNet18) and Ns (i.e., 4-bit ResNet18), we report their augmented schedules quantitatively in terms of nor- malized probability and magnitude CIFAR- 100 in Fig. 3. We normalize the probability of each epoch by dividing the maximal summation of probabilities for all operations across all epochs.
It be that the discovered schedules As for Ns is quite different from Ar for Nr. In particular, for Ar, there is an emphasis on Brightness, Posterize, Rotate, Sharpness and Translate while As cares more about Contrast, ShearX and TranslateY. Furthermore, we observe that the probability and magnitude increase as the epoch evolves. For As. in the beginning, KD plays more important role, and there no augmentation operation before about epoch 50. As the training continues, the augmentation policies become more important. One possible reason is that, for low- precision networks, KD methods can provide rich training signals such that data augmentation does not help in the early training phases.
Furthermore, we observe that, compared to Ar. the schedule for student As evolves more smoothly in the sense that the policy updating frequency is lower. For example, the probability and magnitude values change about every 40 epochs for student, while the policies for teacher update about every 15 epochs. One possible reason is that for the low-precision Ns. KD methods make the training process more smooth and is not necessary to change the augmentation policies too frequently. This is consistent with the observations shown in Tab. 2 that KD can already provide useful training signals. Also, this validates our assumption that Ng has her own optimal augmentation schedule As that is different from Ar.

---

Under review as a conference paper at ICLR 2020
Thersned
Selire
64
Wyort
66
Swerk
Marproia
Shoppioo
04
Doudie
Treett
Opualias
Croad
02:
Coreriz
33
123
200
Epoch Number
Epoch Number
(a) Normalized plot of operation probability parame- ters over time for the teacher network Nr.
(b) Operation magnitude parameters over time for the teacher network Ny.
1.0
Trenslact
Seletre
08-
Owart
Swark
Probobiity
66-
Merpecss
Riate
Megettuue
Opaallas
12
02-
Col
Epoch Number
Epoch Number
(c) Normalized plot of operation probability parame- ters over time for the student network Ns.
(d) Operation magnitude parameters over time for the student network Ns.
Figure 3: Evolution of magnitude and probability parameters in the learned schedules. Each opera- tion appears in the parameter list twice, and take the mean values of the parameter.
5.6 FURTHER COMPARISON ON PULL-PRECISION NETWORKS
This subsection aims to verify the effectiveness of proposed methods settings where both Ny and N's are full-precision networks. We select ResNet18 as the teacher, and ResNet8 the student to check how our proposed methods affect the student network.
conventional
Tab. shows that our proposed method outperforms the standard baseline training. The discovered augmentation schedule further boosts the performance of shallow Ng based on II-KD, though the improvement is not that significant compared with the results obtained when W is low-precision network. This shows that our proposed method can be used for full precisiin training tasks.
Table 4: Accuracy (%) CIFAR-100 with full-precision ResNet8 as Ng and full-precision ResNet18 as My under different settings. Vanilla Training refers to training a full-precision net- work from scratch. Re-Training with PBA refers to using learned schedules discovered by PBA to re-train Ny as described in Section 4.1. Student with only II-KD refers to training Ns using I1-KD but without the learned data augmentation. Student after Stage-3 refers training Ns using II-KD and the learned data augmentation.
Vanilla Training
Re- Training with PHA
Student with only II-KD
After Stage-J
Accuracy
74.31
74.52
75.52
75.88
CONCLUSION
Previous literature on KD focuses on exploring the knowledge representation and the strategies for distillation. However, both the teacher and student learn from the same training data without adapt- ing the different learning capabilities. To address this issue, we propose customizing distinct agents to automatically augment the training data for the teacher and student, respectively. We have exten- sively studied the effect of combining data augmentation and knowledge distillation. Furthermore, we propose a simple feature- within and across feature maps. We have empirically observed that the student can lear better from the teacher with the proposed approach, especially in the challenging tow-precision scenario, and the learned schedules different for the teacher and student.
KD variant that incorporates both intra- and reationilas
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Albert Bandura. Social cognitive theory in cultural context. Applied Psychology, 51(2), 2002. doi: (44111//164-0597.0000 URL abs/10. 111.11111114405555
wwbboaryywwleee. wwwey. com/col//ooooooo
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 300.3333003333,,, 2013.
Geoff Brumfiel. Who has designs on your students' minds? Nature Publishing Group, 2005.
Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2006.
Kevin Clark; Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoe V Le. Bam! born-again multi-task networks for natural language understanding. arXiv preprint 1eXiv:1907.04822 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation policies from data, arXiv preprint arXiv:1805.0955: 09501, 2018
Terrance VeVries and Graham W Taylor. Dataset augmentation feature space. arXiv preprint 1702.0530,, 2017.
Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. arXiv preprint arXiv: /1001105500777 2018.
Johannes Gurlitt, Alexander Renkl, Michael Motes, and Sabine Hauser. How can we use con- cept maps for prior knowledge activation: different mapping-tasks lead to different cognitive processes. In Proceedings of the 7th international conference on Learning sciences. International Society of the Learning Sciences, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. IEEE Conf. Comp. Vis. Patr. Recogn., 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. In Proc. Adv. Neural Inf. Process. Syst. Workshops, 2014.
Daniel Ho, Eric Liang, Ion Stoica, Pieter Abbeel, and Xi Chen. Population based augmentation:
Efficient learning of augmentation policy schedules. In Proc. Int. Conf. Mach. Learn., 2019.
Seung Hyun Lee, Dae Ha Kim, and Byung Cheol Song. Self-supervised knowledge distillation using singular value decomposition. In Proc. Eur. Conf. Comp. Vis., 2018
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Proc. Adv. Neural Inf. Process. Syst., 2012.
Joseph Lemley, Shabab Bazrafkan, and Peter Corcoran. Smart augmentation learning an optimal data augmentation strategy. Ieee Access, 2017.
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh. Improved knowl- edge distillation via teacher assistant: Bridging the gap between student and teacher. arXiv preprint arXiv:102.003393. 2019.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2019.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quanti- zation. In Proc. Inf. Conf. Learn. Repren., 2018.

---

Under review as a conference paper at ICLR 2020
Alexander J Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learn- ing to compose domain- -omaan--peccffc transformations for data augmentation. In Proc. Adv. Neural Inf. Process. Syst., 2017.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. In Proc. Ini. Conf. Learn. Repren., 2015.
Byung Cheol Song Seunghyun Lee. Graph-based knowledge distillation by multi-head attention network. Proc. Brit. Mach. Vis. Conf., 2019.
James Slotta and Michelene TH Chi. Helping students understand challenging topics in science through ontology training. Cognition and instruction, 2006.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.0224 022433 2019.
Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and Ian Reid. A bayesian data augmenta- tion approach for learning deep models. Proc. Adv. Neurul Inf. Process. Syst., 2017.
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 2016.
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid. Towards effective low- bitwidth convolutional neural networks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2018 88
10


Reviews:
[{"review_id": "SJlEVGdpFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new method to distill a teacher model into a student model. They demonstrate improvements over existing distillation variants. The results are impressive for low-precision networks. \n\nHowever, I found a few problems with the paper:\n\nIn the author’s methods, there seem to be multiple steps:\n1. Train the teacher with PBA (stage-alpha).\n2. Train the student with PBA, using a subset of the data and then using the teacher to learn the augmentation policy (stage-beta). This is described in Section 4.2, but I found it incredibly confusing to get a clear picture since there are several moving parts here (augmentation, student solo training, distillation with teacher).\n3. Additionally in Section 5.2 (experiments), the authors propose the combined inter/intra distillation loss, which is named ‘II-KD’.\n\nIn Table 3, the accuracy of ResNet18 on CIFAR-100 with II-KD is the same as accuracy of Resnet18 Student after stage-beta in Table 2. Same for AlexNet on CIFAR-100. Which of the tables are wrong?\n\nAlso, in section 5.1, the authors mention that they use the pre-trained teacher as a starting point for the student network. This is not a fair comparison with the ‘Soft Labels’ approach of Hinton et al, where the student network is not initialized from the teacher.\n\nFurther, it is unclear why the augmentation plot in the student in Figure 3(c) differs wildly from the augmentation plot of the teacher. There is no augmentation for the first 50 epochs, and then the probability of all the augmentation operations moves in discrete steps together.\n\nThe novelty in the paper seems to be:\na) Applying PBA to a Distillation setting.\nb) Introducing the ‘II-KD’ loss.\n\nAs mentioned (a) is not explained clearly. (b) is explained in a generic way but the authors do not give an example of the inter/intra feature map comparisons.\n\nOverall, while I feel the results are impressive, the method is complex as it is presented. I would be reluctant to accept the paper without further clarification from the authors.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxSscotqB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors hypothesise that, in a Knowledge Distillation (KD) setting, the student network may benefit from learning from different training data than the teacher. The motivation comes from the fact that human teachers \"adapt\" the examples given their students, depending on their individual expertise on the subject, personal and cultural biases, and other factors.\n\nMore concretely, the paper proposes to use an evolutionary algorithm for data augmentation (PBA, already published) to train the teacher and student networks. The key aspect of the proposed method is that the teacher and the student use different augmentation schedules. The augmentation schedules improve results on both the teacher and student networks.\nOrthogonally to this, they also propose to combine different KD objective functions and show that this also improves the results over each loss in isolation. \n\nThe datasets used in the experiments are CIFAR-10 and CIFAR-100. In most of the experiments the student has a (much) lower bit-width than the teacher, but they also apply the method with a student network with less parameters. Two different network architectures are used in the experiments: AlexNet and Resnet18, their method shows consistent improvements over the baselines in all cases (although some improvements may be considered marginal).\n\nOverall the paper is well motivated, clearly written and, in the experiments section, they give enough details, and/or cite previous works that contain them, to reproduce the experiments.\n\nMost importantly, the experiments are well designed to test the initial hypothesis: \n1. The authors show that the student (and teacher) trained with the PBA data augmentation achieves a higher accuracy than the baseline method (Table 2). However, this is not enough to confirm/refute the hypothesis itself, since it is known that data augmentation generally helps.\n2. They show that the two augmentation strategies are different, by using the teacher’s augmentation to train the student network (Table 3). The results show that it’s better to use a specific data augmentation schedule for the student network.\n\nHowever, (the main criticism is that) the paper only partially confirms the hypothesis. For instance, it is not clear whether the hypothesis is true for other forms of KD, or it only applies when training a student with lower bit-width. In Section 5.6, they train a student with fewer parameters (less layers) than the teacher, instead of fewer bits/parameter. However, the improvements of their method over the baseline (II-KD) seem marginal there. And most importantly, as pointed out earlier, this alone is not enough to confirm the hypothesis. A similar table to Table 3 should be included in this section.\n\nSecondly, the proposed KD loss, which is just a combination of previously published works, is an orthogonal improvement to the main method (as the authors admit), and it is not related at all to the subject of the study. It is obviously good to introduce more than one contribution in a paper, but this second (and minor) contribution should be well motivated in its own, in order to avoid \"distracting\" the reader from the main contribution.\n\nIn addition, I would suggest to perform statistical tests to give additional robustness to the conclusions drawn from the experiments. They perform experiments on different architectures (AlexNet and Resnet18) and different KD losses, and the conclusions are always consistent with the hypothesis, but it’s not clear whether the improvements are statistically significant or not. In this regard, it would also be appreciated to include results with other datasets, since only CIFAR-10 and CIFAR-100 (which are very similar datasets) were used. \n\nBeyond statistical significance, it’s also not clear how important are these results for researchers working outside the scope of Knowledge Distillation. \n\nFinally, some minor comments:\n\n- L_original in Eq. (9) is not defined. I'm assuming it's the L_{KD}^{soft} loss.\n- Please, check consistency of pronouns: authors refer to the teacher network as “it” (e.g. “[...] teacher distills knowledge to a narrow/shallow student to improve its performance”, page 2), but to the student as “she” (e.g. “[...] to train the student better from her teacher”, page 2).\n- “our methods clearly outperforms [...]” (page 6)  -> “our method clearly outperforms [...]”.\n\nScore: Borderline accept, but I will increase the score if the authors address my concerns and provide better evidence that the hypothesis is confirmed.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BklZhUMtKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper takes the idea of Population-Based Augmentation (PBA) and extends it to knowledge distillation (KD). The idea is that the ideal augmentation protocol for training-from-scratch (or in this context teacher training) may not be the best for student networks under a KD loss.\n\nI am borderline about this paper and had to pick one, so I landed on Weak Reject. On one hand, I think it’s a really neat idea to apply PBA in this context, and package it as a strage-alpha/stage-beta training procedure. However, the experimental results seem very incremental and I’m not convinced there is a genuine signal there.\n\nExperiments:\n\nTable 1 offers a great comparison of prior work, as well as the combination of prior work (II-KD). Table 2 tells me that PBA seems incremental both for the teacher and the student. Going from vanilla training to student with II-KD gets you most of the way there, and the primary contribution of this paper just gives you a slight benefit above this. It’s great that a more traditional full-precision comparison was also added in Table 4, but this table also confuses me. First, it was unclear if “vanilla training” referred to the teacher or the student. The number 74.31 is not the same as in Table 2 (74.85), so I assume this means it’s the student? If so, the student is already very close to the teacher, and this is not a great starting point for evaluating KD. The student after stage-beta also outperforms the teacher - something that was mentioned in the related work, but I would like more discussion around it specifically for Table 4. Another thing I was wondering was how important PBA is for the teacher’s ability to be a good teacher. It gives a modest boost in Table 2; what if we skip PBA in the teacher but still do it for the student. This would be interesting to add.\n\nOverall, there aren’t that many experiments. The dataset is never more challenging than CIFAR-100. There are also no error bars, which are particularly important when the improvements are small. As for Figure 3, I don’t know if there is anything intuitive we can glean from this. I may just be variation between experiments, as far as I can tell.\n\nI think this paper can be made stronger by making the experimental evidence broader, as well as the analysis of why this works stronger. Without these improvements, the reader is left wondering if there really is any significant benefit. We have to remember that PBA is not cheap (perhaps much cheaper than AutoAugment, but more expensive than fixed augmentation). For most practitioners, the complication and compute costs of PBA would probably not be worth adding on top of KD, if the benefits are too modest.\n\nMinor:\n\nIn Table 3, it says \"Ours\" for AlexNet and \"II-KD\" for ResNet8. Should the both be the same?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies Population-Based Augmentation in the context of knowledge distillation (KD) and proposes a role-wise data augmentation schemes for improved KD. While the reviewers believe that there is some merit in the proposed approach, its incremental nature and inherent complexity require a cleaner exposition and a stronger empirical evaluation on additional data sets. I will hence recommend the rejection of this manuscript in the current state. Nevertheless, applying PBA to KD seems to be an interesting direction and we encourage the authors to add the missing experiments and to carefully incorporate the reviewer feedback to improve the manuscript.
Decision: Reject