Teacher-Student Compression with Generative Adversarial Networks
Authors: Ruishan Liu
Nicolo Fusi
Lester Mackey
Venue:  (2020)

Abstract: More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU- or memory-constrained devices. Teacher-student compression (TSC), also known as distillation, alleviates this burden by training a less expensive student model to mimic the expensive teacher model while maintaining most of the original accuracy. However, when fresh data is unavailable for the compression task, the teacher's training data is typically reused, leading to suboptimal compression. In this work, we propose to augment the compression dataset with synthetic data from a generative adversarial network (GAN) designed to approximate the training data distribution. Our GAN-assisted TSC (GAN-TSC) significantly improves student accuracy for expensive models such as large random forests and deep neural networks on both tabular and image datasets. Building on these results, we propose a comprehensive metric—the TSC Score—to evaluate the quality of synthetic datasets based on their induced TSC performance. The TSC Score captures both data diversity and class affinity, and we illustrate its benefits over the popular Inception Score in the context of image classification.

Under review as a conference paper at ICLR 2020
TEACHER-STUDENT COMPRESSION WITH GENERATIVE ADVERSARIAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU- or memory- constrained devices. Teacher-student compression (TSC), also known distilla- lion, alleviates this burden by training less expensive student model to mimic the expensive teacher model while maintaining most of the original accuracy. How- ever, when fresh data is unavailable for the compression task, the teacher's train- ing data is typically reused, leading to suboptimal compression. In this work, we propose to augment the compression dataset with synthetic data from a generative adversarial network (GAN) designed to approximate the training data distribution. Our GAN-assisted TSC (GAN-TSC) significantly improves student accuracy for expensive models such as large random forests and deep neural networks on both tabular and image datasets. Building on these results, we propose comprehen- sive metric- the TSC Score-to to evaluate the quality of synthetic datasets based on their induced TSC performance. The TSC Score captures both data diversity and class affinity, and we illustrate its benefits over the popular Inception Score in the context of image classification.
INTRODUCTION
Modern machine learning models have achieved remarkable levels of accuracy, but their complexity can make them slow to query, expensive to store, and difficult to deploy for real-world use. Ideally, we would like to replace such cumbersome models with simpler models that perform equally well. One way to address this problem is to perform teacher-student compression (TSC, also known as distillation). which consists of training student model to mimic the outputs of a teacher model (Bucila et al., 2006; Li et al., 2014; Hinton et al., 2015). For example, expensive ensemble and deep neural network (DNN) teachers have been used train inexpensive decision tree (Craven & Shavlik, 1996; Frosst & Hinton, 2017) and shallow neural network (Bucila et al., 2006; Li et al., 2014; Ba & Caruana, 2014; Hinton et al., 2015; Urban et al., 2017) students. While alternative model-specific compression strategies abound (see Section 6), TSC is distinguished by its broad applicability: the same framework can be used to compress any classifier, be it a random forest or a deep neural network.
An important degree of freedom in the TSC problem is the compression set used to train the student. Ideally, fresh (unlabeled) data from the training distribution would fuel this task, but often fresh data remains after the teacher is trained (Bucila et al., 2006; Ba & Caruana, 2014). In this case, one branch of the literature, dating back to the pioneering work of Bucila et al. (2006), recommends gen- erating synthetic data for compression and proposes tailored generation schemes for tabular (Bucila al., 2006) and image (Urban et al., 2017) data. A second branch, rooted in the distillation com- munity (Hinton et al., 2015; Frosst & Hinton, 2017), simply uses the same data to train teacher and student (see also Ba & Caruana, 2014). Here, we show that the latter convention leads to subop- timal compression performance and propose a synthetic data generation strategy for both tabular and image data that improves upon standard augmentation schemes. Specifically, when fresh data is unavailable for TSC, we propose augment the compression set with synthetic data produced by generative adversarial networks (GANs) (Goodfellow et al., 2014). GANs attempt to generate new datapoints from the distribution underlying given dataset and have achieved impressive fidelity for a variety of data types including images (Goodfellow et al., 2014), text (Yu et al., 2017), and electronic health records (Choi al., 2017). Here, identify TSC practical downstream task

---

Under review as a conference paper at ICLR 2020
for which GAN generation is consistently useful across data types and classification tasks and de- velop GAN- assisted TSC (GAN-TSC) to improve the TSC of an arbitrary classifier. Our extensive empirical evaluation demonstrates the effecttivnness of GAN-TSC for tabular data (for which GANs are seldom used), image data, random forest classifiers, and DNN classifiers.
Why should synthetic data improve TSC? Note that there is an important distinction between training a student to mimic a teacher with synthetic data and training a student to solve the original supervised learning problem with synthetic data. The goal of the original supervised learning task is to approximate the ideal mapping f between inputs v and outputs y. This ideal f* is functional of the true but unknown distribution underlying our data, and our information concerning f limited by the real data we have collected. The goal in TSC is to approximate the teacher prediction function g which maps from inputs to predictions 2. Because the teacher is a function of the training data alone, 9 itself is a functional of the training data alone and is otherwise independent of the unknown distribution that generated that data. In addition, because we have access to the teacher, we have the freedom to query the function g at any point, and hence our information concerning g is limited only by the number of queries we can afford. In particular, when we generate a new query point I, we can observe the actual target value of interest, the teacher's 's prediction (zx;; this is not true for the supervised learning task, where no new labels can be observed. The insensitivity to errors in synthetic labels and access to fine-grained teacher predictions make TSC more ideally suited to syn- thetic data augmentation. Indeed, we will see in Sections 3 and 4 that the same GAN data that leads to improved TSC leads to degraded accuracy when used to augment the original supervised learning training set. This is consistent with past work that demonstrates gains from GAN-augmented su- pervised learning in specific data-starved situations but reports degraded accuracy when all training data is used (Bowles et al., 2018, Tab. 4). See (Ba & Caruana, 2014) for further discussion on the distinctions between TSC and the original supervised learning task.
Since the improvement realized by GAN-TSC depends on the synthetic data quality, we further pro- pose to use GAN-TSC to evaluate the quality of synthetic datasets and their generators. essence, we declare synthetic dataset to be of higher quality if compressed model trained on that data achieves higher test accuracy. Synthetic data evaluation is a notoriously difficult problem marked by the lack of universally agreed-upon quality measures (Theis et al., 2015). Some standard quality measures, like multiscale structural similarity (Wang et al., 2003), quantify the diversity of a syn- thetic dataset but do not capture class affinity, the ability of datapoints to be correctly associated with their labels with high confidence. Others, like the popular Inception Score (Salimans et al., 2016), quantify class affinity based on the predicted label distribution of trained neural network. However, these scores do not account for within-class diversity and are easily misled by adversarial datapoints that elicit high confidence predictions but do not resemble real data. To address these shortcomings, we develop a TSC Score that quantifies the true test accuracy of compressed models trained using synthetic data; this offers robust, goal-driven metric for synthetic data quality that accounts for both diversity and class affinity. In summary, we make the following principal contributions:
1. We identify TSC practical downstream task for which GAN data augmentation is con- sistently useful across data types and classification tasks and develop GAN-TSC as a drop-in replacement for standard TSC.
2. For random forest teachers, we demonstrate 25 to 336-fold reductions in execution and storage costs with less than 1.2% loss in test performance across suite of real-world tabular datasets. In each case, GAN-TSC improves over student training without TSC, over TSC without synthetic data augmentation, and over the tabular data augmentation strategy of Bucila et al. (2006).
3. For image classification, we show GAN-TSC consistently improves student test accuracy for a variety of deep neural network teacher-student pairings and two popular compression objectives.
4. We introduce a new TSC Score for evaluating the quality of GAN-generated datasets and, on Caltech- and CIFAR- 10, illustrate its value and advantages over the popular Inception Score.
2 TEACHER-STUDENE COMPRESSION WITH GANS
We begin by reviewing standard approaches to DNN TSC and describing our proposals for random forest TSC and improving TSC with GAN data.
2

---

Under review as a conference paper at ICLR 2020
Deep Neural Network TSC In the standard teacher-student approach to compressing neural net- work classifier, a relatively inexpensive prediction rule, like a shallow neural network, is trained to predict the unnormalized log probability values- the logits 2 assigned to each class by previously trained deep network classifier. The inexpensive model is termed the student, and the expensive deep network is termed the teacher. Given compression set of n. feature vectors paired with teacher logit vectors, {(x(1) 2(1)).... .... (xx(*) )( Ba & Caruana (2014) proposed framing the TSC task multitask regression problem with L loss, L( = [g 0) - 2IB. Here, 8 represents any student model parameters to be learned (e.g., the student network weights), and g(2; 0) is the vector of logits
predicted by the student model for the input feature vector x.
Li et al. (2014) introduced an alternative TSC objective function, and Hinton al. (2015) pa- rameterized this objective by a temperature parameter T > 0. Specifically, the student is trained mimic the annealed teacher class probabilities, (z(z/T) exp(z,/T)/ Es (xpzz/T). for each class j by solving a multitask regression problem with cross-entropy loss, (g) -E, (=(2 (og(()(g((;;))) ()))). Hinton et al. (2015) showed that, under a zero-mean logit as- sumption, cross-entropy regression recovers T.2 logit matching as T + Do; however, the two ap- proaches can differ for small T. In Sec. 4, we will experiment with both of these popular TSC approaches.
Random Forest TSC Random forests (Breiman, 2001) construct highly accurate prediction rules by averaging the predictions of a diverse and often large collection of learned decision trees. Effec- tively mimicking a large random forest with a single decision tree or a small forest has the potential to reduce prediction computation and storage costs by multiple orders of magnitude (Bucila et al., 2006; Joly et al., 2012; Begon et al., 2017; Painsky & Rosset, 2016; 2018). Focusing on the common setting of binary classification, we propose to train student regression random forest to predict a teacher forest's 's outputted probability p of a datapoint æ having the label 1.
Reducing overfitting with GAN-assisted TSC (GAN-TSC) In typical TSC setting, as much data as possible has been dedicated to training the highly accurate teacher model, leaving little fresh data for training the student model. While one branch of the TSC literature recommends generating synthetic data with customized augmentation algorithms for tabular (Bucila et al., 2006) and im- age (Urban al., 2017) data, the more common solution in the distillation literature is simply reuse the teacher training set as the compression set (Hinton et al., 2015; Frosst & Hinton, 2017). However, will see in Secs. 3 and 4 that compressing with training data alone leads to suboptimal student performance. This suboptimality occurs both due to teacher overfitting (many teachers per- form very well on test data but are still overfit in the sense of having unrealistically small training error or overconfident training logits not representative of its test logits; these are the teachers that benefit most from GAN-TSC) and student overfitting (the student can benefit from observing the teacher's outputs at points other than the original training points). To boost student performance and compression efficiency, we propose a simple solution applicable to tabular and image data alike: augment the compression set with synthetic feature vectors generated by high-quality GAN. These synthetic feature vectors are then labeled with the teacher's 's outputted class probabilities or logits. We call this approach GAN-assisted TSC and release our Python implementation at REDACTED.
AC-GAN To generate high-quality GAN feature vectors which capture the salient features of each class, we use the auxiliary classifier GAN (AC-GAN) of Odena et al. (2017). The AC-GAN generator G produces a synthetic feature vector Afke = G(W, C*) from a random noise vector W and independent target class label C drawn from the real data class distribution. For any given feature vector v, the AC-GAN discriminator D predicts both the probability of each class label P(C æ) and the probability of the data source being real fake, PS x)) for S {real, fake). For a given training set Dreat of labeled feature vectors, two components contribute to the AC-GAN training objective,
Lsource
(x.c)eD log P(S real x)) + Ew, [log P(S fake G(W, C))] and log P(C = c x)) + Ew Dog P(C G(W,
D
real
Teglase
Draat
D
(rie)EDreat
(1)
representing the expected conditional log- likeeihokhood of the correct source and the correct class of feature vector, respectively. Training proceeds as an adversarial game with the generator G trained
to maximize Tralass Lesource and the discriminator D trained to maximize Tadass Tesource-
3

---

Under review as a conference paper at ICLR 2020
3 RANDOM FOREST GAN-TSC
We now explore how GAN-TSC performs when used to compress large random forests for binary classification. We employ three real-world tabular datasets. The MAGIC Gamma Telescope dataset (Dheeru & Karra Taniskidou, 2017) task is to distinguish hadronic showers from primary gamma signals recorded by a gamma telescope; 64.8% of datapoints have the label 0 (signal). We select a uniformly random subset of 200,000
datapoints from the Higgs dataset (Dheeru & Karra Taniskidou, 2017) to predict whether a given observation was produced by a Higgs boson. Following the feature extraction protocol of (Liu et al., 2017). we extract 29 continuous features from the StumbleUpon Evergreen dataset Eve to predict whether a given web page is evergreen; 48.7% of datapoints have the label 1. We split each dataset into training and test sets uniformly random, with training split sizes given in Figs. la-ld.
In our experiments. the teacher is a random forest classifier with 500 trees, and the student is regression random forest with one to 20 trees; both are trained using scikit-learn (Pedregosa et al., 2011) with default values for all the hyperparameters. For the AC-GAN implementation in Keras, both the generator and the discriminator are one layer fully-connected neural networks with 50 neurons and ReLU activation. We employed noise vectors w E R 100 and an Adam optimizer with learning rate 0.0002 and momentum term 81 = 0.5.
We study three scenarios: TSC using training data only, GAN data only or a mixture of training and GAN data. We generate 11 fake Rreal GAN datapoints for the compression set, where Rreat is the number real training datapoints. The mixture compression set is generated by pooling the Treal training datapoints and the 1 fake GAN datapoints together. We also report the performance of a student trained directly on the original training set without TSC ("Student Only') )) since the student regression forest, the class labels (0 and 1) are treated real value targets.
The results of compressing a random forest with 500 trees into one or more decisions trees are given in Figs. ld. We experiment with variety of training dataset sizes, ranging from n 1k to n = 100k to demonstrate the versatility of GAN-MC. In each case, the trees trained by the teacher and students have similar depth after training. We use test accuracy as our performance metric for the balanced Higgs dataset and test AUC for the unbalanced MAGIC and Evergreen datasets. For all datasets, TSC into a single tree with GAN data outperforms TSC with training data and substantially outperforms the student model trained without TSC. Moreover, for the Higgs dataset, the accuracy boost from GAN-TSC (62.1% 69.6% on Higgs 100k) is 10 times the accuracy boost achieved using training data TSC (62. % to 62.7%)
0.85
0.70
0.85
0.04
0.00
0.82
0.68
Student Only
8 0.00
0.75
GAN
0.61
0.76
GAN
Triknin
0.5
Training GAN Date
0.82
0.64
Tratting and GAN Date
5
10
15
20
10
15
20
Number Treas (btubent)
Number Tress (Student)
Number Timss (Studertt)
GAN-TSC Higgs 1k
(b) GAN-TSC on Higgs 100k
(c) GAN-TSC on Evergreen Sk
Local Var HUNCE
0.320
15.0
Studeat Only
0.90
3.415
12.5
IAAN-15C
0.88
MUNCE Datar
0.06
0.84
Student Only
Traiiing Data
3.530
MUnIc bata pwo)
7.5
3.405.
5.0
0.0
2.5
Number of Thees (Stuuert)
10
3,400
2.0
0.2
Frobsbilty Dryp of MUNGE
24
2.0
200 speedap
(d) GAN-TSC on MAGIC 10k
(c) GAN-TSC vs. MUNGE
(f) Speed-up vs. excess error
Figure 1: (a-e) Student test accuracy (Higgs) and test AUC (Evergreen and MAGIC) when com- pressing a 500 tree random forest into a compact forest. (1) For all datasets, GAN-TSC students increase test time throughput 25 to 336-fold over teacher with less than 1.2% loss of accuracy.
4

---

Under review as a conference paper at ICLR 2020
The example of the Evergreen dataset is also enlightening. Compression into single tree with training data increases student test AUC from 0.731 to 0.856, and compression with only GAN data yields further improvement to 0.882, nearly matching the 0.889 test AUC of the teacher. Remark- ably, this is achieved with a single decision tree which demands 336 times less computation and storage space than the teacher at prediction time. The figure 336 comes from assessment of stu- dent test-time speed-ups summarized in Fig. If. For each dataset, we identified the highest accuracy and most compressed students trained with and without GAN-TSC and measured throughput as the time needed to compute predictions for 30, 000 test examples using one core of an Intel Xeon 6152 processor. At a cutoff of 1.2% excess test error, we observe speed-ups ranging from 25 to 336-fold.
For each dataset save Higgs 1k, TSC with GAN data offers the best (or nearly the best) performance for all forest sizes. For the Evergreen and MAGIC datasets, near-maximal performance is achieved by single GAN-TSC decision tree, with additional trees yielding relatively minor AUC gains. For Higgs 1K, the combination of training and GAN data offers the best performance for all multi-tree forests, with an accuracy boost consistently 2-4 times that of TSC with training data alone.
GAN-TSC vs. MUNGE.
For tabular data, an alternative to GAN-TSC is to augment the compres- sion set in precisely the same way with data generated from the state-of-the-art tabular augmentation strategy, MUNGE, of (Bucila et al., 2006). We find that, for all datasets (Fig. 1e displays a represen- tative comparison for single decision-tree student on MAGIC; additional comparisons are made in Appendix A.2) and all settings of the MUNGE hyperparameters (the local variance 8 and the probability p optimally tuned the test set in Fig. 1e), MUNGE underperforms GAN-TSC.
4 DEEP NEURAL NETWORK GAN-TSC
We now investigate how GAN-TSC performs when used compress convolutional DNN (CNN) classifiers trained on the CIFAR- 10 dataset of (Krizhevsky & Hinton, 2009). CIFAR-10 consists of 32 x 32 RGB images from 10 classes, divided into 50,000 training and 10,000 test images. The test images are randomly divided into a validation set with size 5000 and a test set with size 5000. The AC-GAN is implemented in Keras (Chollet et al., 2015) and trained for 1000 epochs (Tuya, 2017). The discriminator D is CNN with 6 convolution layers and Leaky ReLU nonlinearity. The generator G consists of 'deconvolution' layers which transform the class e and noise vector E ig into 32 x 32 image with 3 color channels. We use the Adam optimizer with learning rate 0.0002 and momentum term 31 = 0.5, as suggested by (Radford et al., 2015).
We employ both of the TSC objectives introduced in Sec. 2 using 200 TSC training epochs. For L logit matching, the teacher and the student are NIN (Lin et al., 2014) and LeNet (LeCun et al., 1998) models. The uncompressed networks are pre-trained by Caffe (Chan, 2016; Jia et al., 2014). Similar to (Chan, 2016), for TSC training, we use the Adam optimizer in Tensorflow (Abadi et al., 2015) with I.2 loss and learning rate 10-4 For cross-entropy regression, we examine three additional networks: WideResNet-28-I1 10 (Zagoruyko & Komodakis, 2016), ResNet-18 (He et al., 2016), and 5-layer CNN with 3 con- volution layers. Network training both with and without compression is carried out in Pytorch (Li, 2018: Paszke al., 2017). For TSC, we use the student objective L(0) aLr + (1 Lo (9), where a E (0, 1] and (0) = EE, {j = c} (og(z)(((z) 0))) is the cross-entropy classification loss for datapoint æ with class label c. For each teacher-student pair, we set T', a, and all optimizer hyperparameters to the default values recommended in (Li, 2018). For the teacher-student pairs 1, 2, and 3 in Table 1, this yields the respective T values 5, 20, and 6 and a values 0.9, 0.9, and 0.95. The Adam optimizer with learning rate 10 3 is used for macher-student pairs and 2, and stochastic gradient descent with learning rate decayed from 0.1 is used for pair 3.
We compare the standard approach of TSC using only the teacher's training dataset to two versions of GAN-TSC: compression using only GAN data and compression using a mixture of training and GAN data. The GAN data is produced in real time during the stochastic optimization training. The mixture of training and GAN data is realized by generating GAN data with probability Pfake and by sampling from the training set with probability 1-P/ake- For each teacher- value of Pfake in {0.0, 0.1, 0.2. ... 1.0) that yields the highest validation set accuracy and report performance on the held-out test set. This results in the choice Pfake = 0.8 for the NIN-LeNet teacher-student pair and 0.2 for the other pairings.
pair, we select the
5

---

Under review as a conference paper at ICLR 2020
Fig. 2a displays student test accuracy following each epoch of TSC training with the L logit- matching objective. In the end, both versions of GAN-TSC significantly outperform TSC on training data alone and training without TSC ('Student Only'). The results are particularly striking for the mixture of GAN and training data which doubles the impact of training data TSC. In this case, student accuracy increases by 10.5 percentage points (from 66.2% to 76.7%) with GAN-TSC as opposed to 5.3 percentage points (from 66.2% to 71.5%) with training data alone. Table I reports comparable improvements for the NIN-LeNet teacher-student pairing when the cross-entropy TSC objective is used. Indeed, the mixture of GAN and training data improves upon training data TSC for all teacher-student pairings investigated.
At the start of the TSC training in Fig. 2, TSC with training data is most effective, presumably because the real training data provide more faithful reflection of the test data distribution, and the overfitting effect is not yet severe. Correspondingly, quicker increase in test accuracy is observed the start Fig. 2a. After approximately 10 epochs, the influence of overfitting gradually increases and becomes dominant over the advantage of fidelity to the test data distribution. The compression set loss for real training data becomes significantly smaller than the loss with either version of GAN-TSC in Fig. 2b, and the test accuracy stops increasing in Fig. 2a. Moreover, the teachers in our experiments yield 100% accuracy on the training set but significantly lower accuracy on test datapoints, indicating significant difference between the distributions of training and test set logit values and a disadvantage to relying wholly on training points. This dynamic illustrates the trade-off between GAN faithfulness to the real data distribution and the influence of overfitting and suggests that GAN-TSC improves accuracy by mitigating overfitting to the compression set using a plentiful source of fresh and realistic (albeit imperfect) data.
Training Data GAN Dota Training dAN Data
0.76
0.74
0.75
0.72
0.70
Training Data
Took
GAN Data
Loss 15
0.73
0.58
Thwniin CAB Date
Stucwnt Only
25
100
125
150
175
Compress Eaoch
Probulally TTxining GAN Dalla
(a) GAN-TSC student test accu- racy as training progresses
(b) GAN-TSC student 2 train- ing loss as training progresses
(c) Effect of Pfake on GAN-TSC
student test accuracy
0.67
0.630
0.77
0.25
0.76
0.75
0.4
0575
8.50
GAN-1SC
0.72
GMN-13C will Stardard
0.2
0.500
Phosubiiyy of Training CAN Dula,
SAN
Probabillty My of (/Traiing GAN Dito,
(d) GAN-TSC complements standard image augmentation
(e) GAN quality matters for
(f) GAN- assisted supervised learning impairs accuracy
GAN-TSC student test accuracy
Figure 2: Student performance for L logit-matching compression on CIFAR-10 (see Sec. 4), aver- aged over independent student training runs. The teacher and student are NIN and LeNet with test accuracies of 78.1% and 66.2% (red dashed curve in (a)) when trained without compression.
Table I: CIFAR-10 image classification test accuracies for cross-entropy compression with various teacher and student neural net architectures. GAN-TSC outperforms standard TSC training data alone and training without compression ("Student Only') all cases. See Sec. 4 for more details.
Teacher Only
Student Only
Student after Compression with Training Data Training & GAN
Teacher
Student
NIN
LeNet
78.1% 94.2%
66.2%
71.0%
75.3%
2
ResNet-18
5-layer CNN
78.8%
84.4%
86.6 %
3
WideResNet-28-10
ResNet- 18
95.8%
94.2%
94.3%
95.0 %

---

Under review as a conference paper at ICLR 2020
Effect of the GAN training proportion parameter Pfake- Adopting the experimental setup of Fig. 2, we next examine how Pfake- the probability of selecting a GAN datapoint over a real dat- apoint when training the student, affects compression performance. We plot the dependence of trained student test accuracy on Pfake in Fig. 2c. When Pfake O, only training data is used for compression; when Pfake 1, only GAN data is used. Notably, every non-zero setting of Pfake leads to improved accuracy over compression with the real training data alone, underscoring the value of GAN augmentation. Beyond this, we observe a non-monotonic but unimodal dependence on Pfake with a combination of GAN and real datapoints providing significantly higher accuracy than GAN or real datapoints alone. This is consistent with a trade-off between the overfitting caused by training data reuse and the inability of GAN to perfectly approximate the true data distribution.
GAN-TSC complements standard augmentation. Our next experiment explores the impact of standard image augmentation on compression with and without GAN-TSC. We adopt the exper- imental setup of Fig. 2 but, during teacher and student training, we introduce the random image augmentations, in the form of left-right image flips and hue-saturation-aat value (HSV) shifts and scal- ings, responsible for the state-of-the-art TSC performance in (Urban et al., 2017). In Fig. 2d, we see that student compressed with standard augmentation alone has 74.5% test accuracy (versus 76.7% for GAN-TSC without standard augmentation); however, the greatest gain is realized when GAN and standard augmentation are combined, yielding a maximum accuracy of 78.3%.
GAN quality matters. To investigate the degree to which synthetic data quality affects TSC im- provement, repeat the experiment of Fig. 2 using GAN data of varying quality and Pfake 1. We use the number of GAN training epochs as a proxy for GAN quality. In Fig. 2e, we see student test accuracy is greatly impaired by using a low-quality GAN trained for too few epochs. Fortunately, student accuracy monotonically improves as the number of epochs and GAN fidelity increase.
GAN-TSC vs. G-N-sssisted supervised learning. In Sec. we discussed the significant differ- ences between GAN-TSC and using GAN data to augment the training set for the original supervised learning problem. Fig. 2f shows that the same mixtures of GAN and training data that improve student compression performance in Fig. 2c actually impair accuracy when the student is trained without compression for the original supervised learning task. We observe the same phenomenon in random forest compression (see Fig. 4 in the supplement).
5 A SEACHER-STUDDER COMPRESSION SCORE FOR EVALUATING GANs
The evaluation of synthetic datasets is an important but challenging task. Two criteria commonly considered essential for a high-quality synthetic dataset are datapoint diversity and class affinity. The most widely used GAN quality measure, the Inception Score (IS) of Salimans al. (2016), measures across-class diversity but does not account for within class diversity. In addition, the IS measures form of class affinity based on the predictions of a pre-trained neural network but is easily misled by datapoints that elicit high confidence predictions without resembling real data. For example, if the classification loss Lelass is heavily upweighted relative the source loss Leource while training an AC-GAN, the generator will be more likely to produce feature vectors classified with high confidence by neural networks. As we will see in Sec. 5, such feature vectors need not resemble real data but will nevertheless receive high ISs (which should be reserved for high-quality datasets). To account for both class affinity and diversity a more robust and holistic manner, we propose to use the performance of a student trained on GAN data as a measure of GAN dataset quality. For reproducibility, Python code to compute the TSC Score is available at REDACTED.
The TSC Score To evaluate the quality of a generated dataset D relative to a real dataset Dreat define a Teacher-Student Compression Score (TSCS) based on the test accuracy acc(D) of a student trained with compression set 'D to mimic pre-trained teacher:
TSCScore (Diicea) =
BDo(D) (oo(D)aacCmod aCCmode (oce(Drett) arC
where acCmode is the accuracy obtained by always predicting the most common class in the test set. In our experiments, we choose Dreat to be the teacher's 's training data, but any choice is equally valid, as the ranking induced by the TSCS is not affected by the choice of D Dreat-
The TSCS declares a synthetic dataset to be of higher quality if a compressed model trained only on that data achieves higher accuracy real test data. The score takes values in [0, co) and tends
7

---

Under review as a conference paper at ICLR 2020
to 0 as the synthetic data distribution diverges from the real data distribution. Appealingly, the TSCS tends to increase in response to increases in within-class diversity, across-class diversity, and class affinity, as each can enable the student to more accurately mimic the teacher's output across all classes. This makes the TSCS a more holistic measure of synthetic data quality than the IS or multiscale structural similarity. However, crucially, the TSCS is only impacted by aspects of class affinity and diversity that matter for performance on real test data. Hence, unlike the IS which is completely determined by the idiosyncratic output of an imperfect network, the TSCS is robust to the idiosyncratic preferences of an imperfect teacher or student. In particular, we would not expect a student trained on unrealistic or adversarial synthetic data to perform well on real test data even it very accurately mimics the teacher's predictions on such data. A potential inconvenience of the TSCS is the need to train an inexpensive student model. To ensure that the TSCS can be computed efficiently, we train each student for only one epoch; our experiments suggest that this is sufficient to effectively capture GAN data quality and can be less expensive then evaluating the IS.
As a first simple illustration of TSCS behavior on real and synthetic data, we use the Caltech-256 dataset (Griffin et al.. 2007) with 256 object catogories and 30,607 total images, Because of the few samples per class and high within-class variability, Caltech-255 is challenging dataset for GANS, and we would expect an AC-GAN trained to be of relatively low quality. To train our AC-GAN and perform evaluations, we randomly split the data into 100 training and 20 test images per category; notably, the fake images produced by the AC-GAN (see Fig. 3) are visually quite distinct from real Caltech- 256 photos (Fig. 5 in the supplement). To compute the TSCS, choose Xception (Chollet, 2017) as the teacher and SqueezeNet (landola et al., 2016) as the student. In line with our expectations, the TSCS drops from 0.96 + 0.05 (real images) to 0.01 + 0.01 (fake images), indicating that little useful information could be learned from the low-quality GAN data. Comfortingly, in this example, the standard IS also decreases (from 21 + 1 for real images to 8.0 + 0.1 for fake images), but we will see next that the IS does not always behave as expected.
Scoring GANs: An Illustration with Caltech-256
Figure 3: GAN data for Caltech-256.
TSC vs. Inception: An Illustration with CIFAR-10 To illustrate the benefit of the TSCS over the commonly-used IS, we reinstate the CIFAR-10 experimental setup of Fig. 2. We evaluate the TSCS on 50K CIFAR-10 images (the teacher's ' training data), 50K well-trained GAN images (i.e., data from the AC-GAN described in Sec. 4), and 50K inferior images which have high confidence classifications under the teacher network but do not resemble real data. The inferior data is generated by training the well-trained AC-GAN for 10 additional epochs using only the classification objective Leiass (1). That is, both the generator G' and discriminator D are trained to maximize Lelass, while ignoring the traditional GAN objective component Lsource- We report means and standard errors across 3 independent runs. In Table 2, the GAN data quality degrades noticeably after the additional training with only Lelass: and the TSCS decreases accordance with our expectations. However, the IS increases for the inferior GAN images despite the evident unrealistic artifacts.
Table 2: Inception and TSC Scores for CIFAR- 10 images; larger scores should signify higher quality. TSCS decreases for inferior images, but IS increases despite evident unrealistic artifacts.
Real Data
Well-trained GAN
Inferior GAN
Inception: 11.2 + 0.1
Inception: 5.80 + 0.06
Inception: 5.93 + 0.06
Compression: 0.994 + 0.003
Compression: 0.778 + 0.002
Compression: 0.702 + 0.002
8

---

Under review as a conference paper at ICLR 2020
To highlight the practicality of the TSCS, also report timing comparison of the IS and TSCS evaluations. To compute the IS, we perform one Inception network forward pass on 50K GAN images. To compute the TSCS, first perform one forward pass the same 50K images to get the NIN teacher's logits. We then train the LeNet student for one epoch with one forward and one backward pass. We finally perform one forward pass 10K real test images compute student test accuracy. Using the IS code of (Salimans et al., 2016) and an NVIDIA Tesla V100 GPU, the IS required 1436.6s 6s and the TSCS 350.1s.
6 RELATED AND FUTURE WORK
To reduce the deployment costs of expensive machine learning classifiers, introduced GAN- assisted TSC as a straightforward way to improve teacher-student compression. We demonstrated the benefits of GAN-TSC for both tabular and image data classifiers and developed a new TSC Score for evaluating the quality of synthetic datasets. While we have focused on improving the popular geacher-studen paradigm of compression, we would be remiss to not mention alternative, model-specific approaches to reducing deployment costs, including parameter sharing (Chen et al., 2015), network pruning (Han et al., 2015), and network parameter prediction (Denil et al., 2013) for DNNs and indicator function selection (Joly et al., 2012), pre-pruning (Begon et al., 2017), and probabilistic modeling and clustering (Painsky Rosset, 2016; 2018) for random forests.
A number of exciting opportunities for future work remain. For example, GAN-TSC is readily integrated into more complex TSC approaches that currently reuse the teacher's training data for compression. Prime examples are the recent approaches of (Wang al., 20188 Xu et al., 2018; Wang et al., 8a). These differ from standard TSC by employing non-standard GAN-type com- pression losses, in which the student et al., 2018; Wang et al., 201 8a); Wang et al. (2018)) also train the teacher and student together. In addition, GAN development for tabular data has received much less attention than GAN develop- ment for image data, and we anticipate that significant improvements over the AC-GANs used in our experiments will result in significant performance benefits for GAN-TSC.
as the discriminator (Wang al., 2018b) generator (Xu
REFERENCES
Stumbleupon evergreen dataset, http::/s/www.kggl..cco.m
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing heterogeneous systems, 2015. URL https: /www///www////////////////ww/ tensorflow org/. Software available from tensorflow.org.
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in neural informa- tion processing systems, pp. 2654-2662, 2014.
Bean-Miched Begon, Arnaud Joly, and Pierre Geurts. Globally induced forest: A prepruning com- pression scheme. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp- 420-428, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger Gunn, Alexander Ham- mers, David Alexander Dickie, Maria Valdés Hernández, Joanna Wardlaw, and Daniel Rueck- ert. Gan augmentation: Augmenting training data using generative adversarial networks. arXiv 2018.
preprint 1810.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.

---

Under review as a conference paper at ICLR 2020
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In Proceed- ings of the Tweifth ACM SICKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, August 20-23, 2006, pp. 535-5415 2006. doi: 10. 10.1145/1150002 1150464.
James Chan.
2016.
Wenlin Chen, James Wilson, Stephen Tyree. Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International Conference on Machine Learning, PP- 2285-2294 2015.
Edward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F Stewart, and Jimeng Sun. Gen- erating multi-label discrete patient records using generative adversarial networks. In Proceedings of the 2nd Machine Learning for Healthcare Conference, volume 68 of Proceedings of Machine Learning Research, pp. 286-305. PMI.R, 2017.
François Chollet et al. Keras. https: //kpss://kers... io, 2015.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Mark Craven and Jude W Shavlik. Extracting strusttrct representations of trained networks. In Advances neural information processing systems, pp. 24-30, 1996.
Misha Denil, Babak Shakibi, Laurent Dinh, Nando De Freitas, et al. Predicting parameters in deep learning. Advances in neural information processing systems, pp. 2148 2156, 2013.
Dua Dheeru and Ef Karra Taniskidou. UCI machine learning repository, 2017. URL http:
//archive ics. uci. edd/ml.
Nicholas Frosst and Geoffrey Hinton. Distilling a neural network into soft decision tree. arXiv preprint arXiv: / /71.//711....4... 2017.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint S50.::55000053111 2015.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint 162200360..07600 2016.
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embed- ding. arXiv preprint arXiv: 1408.5093, 2014.
Arnaud Joly, François Schnitzler, Pierre Geurts, and Louis Wehenkel. Ll-based compression of random forest models. In 20th European Symposium on Artificial Neural Networks, 2012.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
10

---

Under review as a conference paper at ICLR 2020
Yann LeCun, Léon Bottou, Yoshua Bengio. and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(1))22222 2324, 1998.
Li. htt.:///w..ttc.....e.eeeee/e//aa 2018.
Jinyu Li, Rui Zhao, Jui-Ting Huang. and Yifan Gong. Learning small-size dnn with output- distribution-batt based criteria. In Fifteenth annual conference of the international speech commu- nication association, 2014.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In ICLR, 2014.
Yu Liu, Hantian Zhang, Luyuan Zeng, Wentao Wu, and Ce Zhang. MIhench: How good are machine learning clouds for binary classification tasks on structured data. ArXiv e-prints, 2017.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier GANs. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2642-2651 2017.
Amichai Painsky and Saharon Rosset. Compressing random forests. In IEEE 16th International Conference on Data Mining, ICDM 2016, December 12-15, 2016, Barcelona, Spain, pp. 1131- 1136, 2016. doi: 10. N0.1100//CDMM22011..11 URL https:: /tt.///oiiio/ www.doo.o....oooooooo wrg/10..///..oo.......oo 1109/ICDM 2016.01 0148.
Amichai Painsky and Saharon Rosset. Lossless (and lossy) compression of random forests. arXiv preprint arXiv: 197,, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Pedregosa, G. Varoquaux, Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournaeeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn; Machine learning Python. Journal of Machine Learning Research, 12:2825-2888 2011.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:151 10:::111116660 1.06434, 2015.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pp.
2016.
Lucas Theis, Aäron van den Oord, and Mathhis Bethge. A note on the evaluation of generative models. arXiv preprint arXiv:1511.01844 2015.
Tuya, https:: /ttp://iittn httpgtttthhthgge/gg///ggeebee wwt.coo////... of http://iinngo-ninnttteesoeesssee
Gregor Urban, Krzysztof Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang, Rich Caruana, Abdelrahman Mohamed, Matthai Philipose, and Matt Richardson. Do deep convo- lutional nets really need to be deep and convolutional? International conference on learning representations, 2017.
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Kdgan: Knowledge distillation with genera- tive adversarial networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 775-786. Cur- ran Associates, Inc., 2018 8a.
Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. Adversarial learning of portable student net- works. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI- 18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Sym- posium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 pp. 4260 4267, 2018b

---

Under review as a conference paper at ICLR 2020
Z. Wang, E. P. Simoncelli, and A. C. Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, 2003, volume 2. pp. 1398-1402 Vol.2, Nov 2003. doi: 10.1109AACSS.2203.11222
Zheng Xu, Yen-Chang Hsu, and Jiawei Huang. Training student networks for acceleration with conditional adversarial networks. In Brilish Machine Vision Conference (BMVC), 2018.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI, pp. 2852 2258, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
12

---

Under review as a conference paper at ICLR 2020
A SUPPLEMENT
A. RANDOM FOREST GAN-TSC Vs. GAN-ASSISTED SUPERVISED LEARNING
Consistent with our discussion in Sec. and our findings in Fig. 2f, Fig. shows that the same GAN data that substantially improves student compression performance in Fig. 1d harms or scarcely improves test AUC when the random forest student is trained without compression for the original MAGIC supervised learning task.
0.92
0.90
0.88
Test AUC
0.86
0.84
Student Only
0.82
GAN Data
Training and GAN Data
0.80
0.78
1
5
10
15
20
Number of Trees (Student)
Figure 4: Without compression, a random forest student trained for the original supervised learn- ing task on MAGIC using only training data ('Student Only"), only GAN data, or mixture of training and GAN data.
A.2 RANDOM FOREST GAN-TSC VS. MUNGE
On the task compressing a 500-tree random forest into a single tree (see Sec. 3), GAN-TSC outperforms TSC with MUNGE data every dataset ((Hggs 100k: MUNGE 64.9%, GAN-TSC 69.6%), (Higgs 1k: MUNGE 58.3%, GAN-TSC 59.0%) (MAGIC: MUNGE 0.909, GAN-TSC 0.918) (Evergreen: MUNGE 0.878, GAN-TSC 0.882) )
A.3 REAL IMAGES FOR CALTECH-256
The real images for Caltech-256 is shown in Fig. 5.
13

---

Under review as a conference paper at ICLR 2020
Figure 5: Real images for Caltech-256.
14


Reviews:
[{"review_id": "Sye1MwBJ9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an algorithm to generate images for training a student network through distillation. The paper claims the use of the same training data is not necessarily beneficial when it comes to improving the accuracy of the student being trained. \nThe paper sits on the empirically driven side with sufficient experiments in the context of random forest and CNN.\nAs a second contribution, the paper proposes a scoring process to evaluate the quality of datasets generated by GAN methods. There are little experiments on this side. \n\n\nPros:\n- I like the paper and the idea behind being able to improve or even train a student network when the original data is not present.\n- Metrics tailored to the problem are relevant. \n\nNegs:\n\n- While there are plenty of experiments, there is a lack of detailed descriptions. \n- The scoring for GANS seems to be barely tested. In the scoring GAN, The fact that TSCS drops is enough to be a valid metric (or better than existing?). In the TSC vs inception, it is hard for me to see the unrealistic artifacts and, according to the text, that is not what TSCS is measuring, right? What is the influence of using different student-teacher configuration? (on the time to produce the scores the paper claims NiN and LeNet, is there any difference if using other architectures (ResNet family for instance)? At least, in that case, the time changes. It would be nice to see the stability of this metric to demonstrate that the need for training an inexpensive model is correct. \n\n\nFor the TSC vs Inception, the GANs are subjectively assessed, isn't it (as to select well-trained to Inferior). Would it be possible to see exactly the same images between the two of them? Seems like the difference in quality for IS is significantly larger than for the proposed metric (even in the last gan there is a slight increase in the metric). \n\nIt seems to me that IS is just a quality metric based on how a single image looks like. Would it be possible to disentangle the training and the scoring in the proposed metric? What if my hyperparameters for doing the single epoch are totally wrong?\n\n\nIn the general idea of the GAN, while I like it, there is little about how the GAN is actually trained. I guess this GAN is trained using some sort of real data and therefore, the comparison is not totally fair. How many images were used to train this GAN? What would happen if those images are used directly in the distillation framework? \n\nHow many images are generated in section 4? Is the influence of pfake related to the dataset? If I have to train using another dataset, how do i set that parameter? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gog4vAYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this manuscript, authors adopt GAN for data augmentation to improve the performance of knowledge distillation. My concerns are as follows.\n1.\tThe novelty is limited. Using GAN for data augmentation is not new and authors only introduce it for KD, which didn’t address the essential problem of KD itself.\n2.\tThe experiments are not sufficient. For DNNs, authors only compare the performance on CIFAR-10 with the conventional KD. More data sets and benchmark algorithms are helpful to illustrate the effectiveness of GAN data.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxtFPgnFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach for improving teacher-student compression by introducing the assistant of GANs. A conditional GAN is trained for generating synthetic data. Then, the generated data combined with training data is used for knowledge distillation. Experiments on large random forests and deep neural networks demonstrate the effectiveness of the proposed method on data-augmentation. Moreover, an evaluation metric is proposed to evaluate s across-class diversity and intra-class diversity for generative models.\n\nPros:\n\n+ While using synthetic data of GANs to assist supervised learning has been shown to be failed in pervious works (e.g. [1] and also shown in this paper, this paper presents a new perspective to utilize GAN as a successful data-augmentation technique in teacher student paradigm.\n+ Experiments are conducted in several settings including different models (random forests, deep neural networks) and different datasets (images and tabular) to show the effectiveness of proposed method in various settings.\n+ The proposed GAN-TSC can be combined with standard augmentation to achieve higher performance as shown in the experiments.\n+ This paper is well written and easy to follow.\n\nCons:\n\n- Knowledge distillation, as a classical model compression technique, has been applied in deep convolutional models for several years. The CIFAR-10 dataset is too simple to evaluate this kind of methods. The author should try to conduct experiments on large scale datasets such as ImageNet, unless the reliability of the proposed algorithm would be very limited.\n- The proposed TSCScore seems to be similar with [1], especially when its novelty mainly lies in intra-class diversity compared with IS. It’s necessary to discuss difference between TSCScore and [1].\n\n[1] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my GAN? ECCV 2018.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper uses GAN for data augmentation to improve the performance of knowledge distillation.

Reviewers and AC commonly think the paper suffers from limited novelty and insufficient experimental supports/details.

Hence, I recommend rejection.
Decision: Reject