Teacher-Student Compression with Generative Adversarial Networks

1 INTRODUCTION . Modern machine learning models have achieved remarkable levels of accuracy , but their complexity can make them slow to query , expensive to store , and difficult to deploy for real-world use . Ideally , we would like to replace such cumbersome models with simpler models that perform equally well . One way to address this problem is to perform teacher-student compression ( TSC , also known as distillation ) , which consists of training a student model to mimic the outputs of a teacher model ( Bucila et al. , 2006 ; Li et al. , 2014 ; Hinton et al. , 2015 ) . For example , expensive ensemble and deep neural network ( DNN ) teachers have been used to train inexpensive decision tree ( Craven & Shavlik , 1996 ; Frosst & Hinton , 2017 ) and shallow neural network ( Bucila et al. , 2006 ; Li et al. , 2014 ; Ba & Caruana , 2014 ; Hinton et al. , 2015 ; Urban et al. , 2017 ) students . While alternative model-specific compression strategies abound ( see Section 6 ) , TSC is distinguished by its broad applicability : the same framework can be used to compress any classifier , be it a random forest or a deep neural network . An important degree of freedom in the TSC problem is the compression set used to train the student . Ideally , fresh ( unlabeled ) data from the training distribution would fuel this task , but often no fresh data remains after the teacher is trained ( Bucila et al. , 2006 ; Ba & Caruana , 2014 ) . In this case , one branch of the literature , dating back to the pioneering work of Bucila et al . ( 2006 ) , recommends generating synthetic data for compression and proposes tailored generation schemes for tabular ( Bucila et al. , 2006 ) and image ( Urban et al. , 2017 ) data . A second branch , rooted in the distillation community ( Hinton et al. , 2015 ; Frosst & Hinton , 2017 ) , simply uses the same data to train teacher and student ( see also Ba & Caruana , 2014 ) . Here , we show that the latter convention leads to suboptimal compression performance and propose a synthetic data generation strategy for both tabular and image data that improves upon standard augmentation schemes . Specifically , when fresh data is unavailable for TSC , we propose to augment the compression set with synthetic data produced by generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . GANs attempt to generate new datapoints from the distribution underlying a given dataset and have achieved impressive fidelity for a variety of data types including images ( Goodfellow et al. , 2014 ) , text ( Yu et al. , 2017 ) , and electronic health records ( Choi et al. , 2017 ) . Here , we identify TSC as a practical downstream task for which GAN generation is consistently useful across data types and classification tasks and develop GAN-assisted TSC ( GAN-TSC ) to improve the TSC of an arbitrary classifier . Our extensive empirical evaluation demonstrates the effectiveness of GAN-TSC for tabular data ( for which GANs are seldom used ) , image data , random forest classifiers , and DNN classifiers . Why should synthetic data improve TSC ? Note that there is an important distinction between training a student to mimic a teacher with synthetic data and training a student to solve the original supervised learning problem with synthetic data . The goal of the original supervised learning task is to approximate the ideal mapping f∗ between inputs x and outputs y . This ideal f∗ is a functional of the true but unknown distribution underlying our data , and our information concerning f∗ is limited by the real data we have collected . The goal in TSC is to approximate the teacher prediction function g which maps from inputs to predictions z . Because the teacher is a function of the training data alone , g itself is a functional of the training data alone and is otherwise independent of the unknown distribution that generated that data . In addition , because we have access to the teacher , we have the freedom to query the function g at any point , and hence our information concerning g is limited only by the number of queries we can afford . In particular , when we generate a new query point x , we can observe the actual target value of interest , the teacher ’ s prediction g ( x ) ; this is not true for the supervised learning task , where no new labels can be observed . The insensitivity to errors in synthetic labels and access to fine-grained teacher predictions make TSC more ideally suited to synthetic data augmentation . Indeed , we will see in Sections 3 and 4 that the same GAN data that leads to improved TSC leads to degraded accuracy when used to augment the original supervised learning training set . This is consistent with past work that demonstrates gains from GAN-augmented supervised learning in specific data-starved situations but reports degraded accuracy when all training data is used ( Bowles et al. , 2018 , Tab . 4 ) . See ( Ba & Caruana , 2014 ) for further discussion on the distinctions between TSC and the original supervised learning task . Since the improvement realized by GAN-TSC depends on the synthetic data quality , we further propose to use GAN-TSC to evaluate the quality of synthetic datasets and their generators . In essence , we declare a synthetic dataset to be of higher quality if a compressed model trained on that data achieves higher test accuracy . Synthetic data evaluation is a notoriously difficult problem marked by the lack of universally agreed-upon quality measures ( Theis et al. , 2015 ) . Some standard quality measures , like multiscale structural similarity ( Wang et al. , 2003 ) , quantify the diversity of a synthetic dataset but do not capture class affinity , the ability of datapoints to be correctly associated with their labels with high confidence . Others , like the popular Inception Score ( Salimans et al. , 2016 ) , quantify class affinity based on the predicted label distribution of a trained neural network . However , these scores do not account for within-class diversity and are easily misled by adversarial datapoints that elicit high confidence predictions but do not resemble real data . To address these shortcomings , we develop a TSC Score that quantifies the true test accuracy of compressed models trained using synthetic data ; this offers a robust , goal-driven metric for synthetic data quality that accounts for both diversity and class affinity . In summary , we make the following principal contributions : 1 . We identify TSC as a practical downstream task for which GAN data augmentation is consistently useful across data types and classification tasks and develop GAN-TSC as a drop-in replacement for standard TSC . 2 . For random forest teachers , we demonstrate 25 to 336-fold reductions in execution and storage costs with less than 1.2 % loss in test performance across a suite of real-world tabular datasets . In each case , GAN-TSC improves over student training without TSC , over TSC without synthetic data augmentation , and over the tabular data augmentation strategy of Bucila et al . ( 2006 ) . 3 . For image classification , we show GAN-TSC consistently improves student test accuracy for a variety of deep neural network teacher-student pairings and two popular compression objectives . 4 . We introduce a new TSC Score for evaluating the quality of GAN-generated datasets and , on Caltech-256 and CIFAR-10 , illustrate its value and advantages over the popular Inception Score . 2 TEACHER-STUDENT COMPRESSION WITH GANS . We begin by reviewing standard approaches to DNN TSC and describing our proposals for random forest TSC and improving TSC with GAN data . Deep Neural Network TSC In the standard teacher-student approach to compressing a neural network classifier , a relatively inexpensive prediction rule , like a shallow neural network , is trained to predict the unnormalized log probability values—the logits z—assigned to each class by a previously trained deep network classifier . The inexpensive model is termed the student , and the expensive deep network is termed the teacher . Given a compression set of n feature vectors paired with teacher logit vectors , { ( x ( 1 ) , z ( 1 ) ) , ... , ( x ( n ) , z ( n ) ) } , Ba & Caruana ( 2014 ) proposed framing the TSC task as a multitask regression problem with L2 loss , L ( θ ) = ||g ( x ; θ ) − z||22 . Here , θ represents any student model parameters to be learned ( e.g. , the student network weights ) , and g ( x ; θ ) is the vector of logits predicted by the student model for the input feature vector x. Li et al . ( 2014 ) introduced an alternative TSC objective function , and Hinton et al . ( 2015 ) parameterized this objective by a temperature parameter T > 0 . Specifically , the student is trained to mimic the annealed teacher class probabilities , qj ( z/T ) = exp ( zj/T ) / ∑ k exp ( zk/T ) , for each class j by solving a multitask regression problem with cross-entropy loss , LT ( θ ) = − ∑ j qj ( z/T ) log ( qj ( g ( x ; θ ) /T ) ) . Hinton et al . ( 2015 ) showed that , under a zero-mean logit assumption , cross-entropy regression recovers L2 logit matching as T → ∞ ; however , the two approaches can differ for small T . In Sec . 4 , we will experiment with both of these popular TSC approaches . Random Forest TSC Random forests ( Breiman , 2001 ) construct highly accurate prediction rules by averaging the predictions of a diverse and often large collection of learned decision trees . Effectively mimicking a large random forest with a single decision tree or a small forest has the potential to reduce prediction computation and storage costs by multiple orders of magnitude ( Bucila et al. , 2006 ; Joly et al. , 2012 ; Begon et al. , 2017 ; Painsky & Rosset , 2016 ; 2018 ) . Focusing on the common setting of binary classification , we propose to train a student regression random forest to predict a teacher forest ’ s outputted probability p of a datapoint x having the label 1 . Reducing overfitting with GAN-assisted TSC ( GAN-TSC ) In a typical TSC setting , as much data as possible has been dedicated to training the highly accurate teacher model , leaving little fresh data for training the student model . While one branch of the TSC literature recommends generating synthetic data with customized augmentation algorithms for tabular ( Bucila et al. , 2006 ) and image ( Urban et al. , 2017 ) data , the more common solution in the distillation literature is to simply reuse the teacher training set as the compression set ( Hinton et al. , 2015 ; Frosst & Hinton , 2017 ) . However , we will see in Secs . 3 and 4 that compressing with training data alone leads to suboptimal student performance . This suboptimality occurs both due to teacher overfitting ( many teachers perform very well on test data but are still overfit in the sense of having unrealistically small training error or overconfident training logits not representative of its test logits ; these are the teachers that benefit most from GAN-TSC ) and student overfitting ( the student can benefit from observing the teacher ’ s outputs at points other than the original training points ) . To boost student performance and compression efficiency , we propose a simple solution applicable to tabular and image data alike : augment the compression set with synthetic feature vectors generated by a high-quality GAN . These synthetic feature vectors are then labeled with the teacher ’ s outputted class probabilities or logits . We call this approach GAN-assisted TSC and release our Python implementation at REDACTED . AC-GAN To generate high-quality GAN feature vectors which capture the salient features of each class , we use the auxiliary classifier GAN ( AC-GAN ) of Odena et al . ( 2017 ) . The AC-GAN generator G produces a synthetic feature vector Xfake = G ( W , C ) from a random noise vector W and an independent target class label C drawn from the real data class distribution . For any given feature vector x , the AC-GAN discriminator D predicts both the probability of each class label P ( C | x ) and the probability of the data source being real or fake , P ( S | x ) for S ∈ { real , fake } . For a given training setDreal of labeled feature vectors , two components contribute to the AC-GAN training objective , Lsource = 1 |Dreal| ∑ ( x , c ) ∈Dreal logP ( S = real | x ) + EW , C∼pc [ logP ( S = fake | G ( W , C ) ) ] and Lclass = 1 |Dreal| ∑ ( x , c ) ∈Dreal logP ( C = c | x ) + EW , C∼pc [ logP ( C | G ( W , C ) ) ] , ( 1 ) representing the expected conditional log-likelihood of the correct source and the correct class of a feature vector , respectively . Training proceeds as an adversarial game with the generator G trained to maximize Lclass − Lsource and the discriminator D trained to maximize Lclass + Lsource .

Summary: This paper proposes an approach for improving teacher-student compression by introducing the assistant of GANs. A conditional GAN is trained for generating synthetic data. Then, the generated data combined with training data is used for knowledge distillation. Experiments on large random forests and deep neural networks demonstrate the effectiveness of the proposed method on data-augmentation. Moreover, an evaluation metric is proposed to evaluate s across-class diversity and intra-class diversity for generative models.