Model Compression with Generative Adversarial Networks

1 INTRODUCTION . Modern machine learning models have achieved remarkable levels of accuracy , but their complexity can make them slow to query , expensive to store , and difficult to deploy for real-world use . Ideally , we would like to replace such cumbersome models with simpler models that perform equally well . One way to address this problem is to perform model compression ( also known as distillation ) , which consists of training a student model to mimic the outputs of a teacher model ( Bucila et al. , 2006 ; Hinton et al. , 2015 ) . For example , expensive ensemble and deep neural network ( DNN ) teachers have been used to train inexpensive decision tree ( Craven & Shavlik , 1996 ; Frosst & Hinton , 2017 ) and shallow neural network ( Bucila et al. , 2006 ; Ba & Caruana , 2014 ; Hinton et al. , 2015 ; Urban et al. , 2017 ) students . An important degree of freedom in the model compression problem is the compression set1 used to train the student . Ideally , fresh ( unlabeled ) data from the training distribution would fuel this task , but often no fresh data remains after the teacher is trained ( Bucila et al. , 2006 ; Ba & Caruana , 2014 ) . In this case , one branch of the literature , dating back to the pioneering work of Bucila et al . ( 2006 ) , recommends generating synthetic data for compression and proposes tailored generation schemes for tabular ( Bucila et al. , 2006 ) and image ( Urban et al. , 2017 ) data . A second branch , rooted in the distillation community ( Hinton et al. , 2015 ; Frosst & Hinton , 2017 ) , simply uses the same data to train teacher and student ( see also Ba & Caruana , 2014 ) . Here , we show that the latter convention leads to suboptimal compression performance and propose a synthetic data generation strategy that is suitable for image and tabular data alike . Specifically , when fresh data is unavailable for model compression , we propose to augment the compression set with synthetic data produced by generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . GANs attempt to generate new datapoints from the distribution underlying a given dataset and have achieved impressive performance for a variety of data types including images 1To avoid ambiguity , we will refer to the dataset used for compression as “ compression set ” and reserve the name “ training set ” for the data used to train the teacher . ( Goodfellow et al. , 2014 ) , text ( Yu et al. , 2017 ) , and electronic health records ( Choi et al. , 2017 ) . Here , we develop GAN-assisted model compression ( GAN-MC ) to improve the compression of expensive machine learning classifiers and demonstrate its effectiveness for both image and tabular data . We further propose to use GAN-MC to evaluate the quality of synthetic datasets and their generators . In essence , we declare a synthetic dataset to be of higher quality if a compressed model trained on that data achieves higher test accuracy . Synthetic data evaluation is a notoriously difficult problem marked by the lack of universally agreed-upon quality measures ( Theis et al. , 2015 ) . Some standard quality measures , like multiscale structural similarity ( Wang et al. , 2003 ) , quantify the diversity of a synthetic dataset but do not capture discriminability , the ability of datapoints to be correctly associated with their labels with high confidence . Others , like the popular Inception Score ( Salimans et al. , 2016 ) , quantify discriminability based on the predicted label distribution of a trained neural network . However , these scores do not account for within-class diversity and are easily misled by adversarial datapoints that elicit high confidence predictions but do not resemble real data . To address these shortcomings , we develop a Compression Score that quantifies the true test accuracy of compressed models trained using synthetic data ; this offers a robust , goal-driven metric for synthetic data quality that accounts for both diversity and discriminability . In summary , we make the following principal contributions in this paper : 1 . We propose GAN-assisted model compression ( GAN-MC ) , a simple approach to improving teacher-student compression by augmenting the compression set with GAN data . 2 . On CIFAR-10 image classification , we show GAN-MC consistently improves student test accuracy for a variety of deep neural network teacher-student pairings and two popular compression objectives . 3 . For random forest teachers , we demonstrate 25 to 336-fold reductions in execution and storage costs with less than 1.2 % loss in test performance across a suite of real-world tabular datasets . 4 . We introduce a new Compression Score for evaluating the quality of GAN-generated datasets and illustrate its advantages over the popular Inception Score on CIFAR-10 . 2 MODEL COMPRESSION WITH GANS . In this section , we review standard approaches to model compression for DNNs and describe our proposals for compressing random forests and improving model compression with GAN data . 2.1 DEEP NEURAL NETWORK COMPRESSION . In the standard teacher-student approach to compressing a neural network classifier , a relatively inexpensive prediction rule , like a shallow neural network , is trained to predict the unnormalized log probability values—the logits z—assigned to each class by a previously trained deep network classifier . The inexpensive model is termed the student , and the expensive deep network is termed the teacher . Given a compression set of n feature vectors paired with teacher logit vectors , { ( x ( 1 ) , z ( 1 ) ) , ... , ( x ( n ) , z ( n ) ) } , Ba & Caruana ( 2014 ) proposed framing the compression task as a multitask regression problem with L2 loss , L ( θ ) = ||g ( x ; θ ) − z||22 . ( 1 ) Here , θ represents any student model parameters to be learned ( e.g. , the student network weights ) , and g ( x ; θ ) is the vector of logits predicted by the student model for the input feature vector x. Hinton et al . ( 2015 ) introduced an alternative compression objective function , indexed by a temperature parameter T > 0 . Specifically , the student is trained to mimic the annealed teacher class probabilities , qj ( z/T ) = exp ( zj/T ) ∑ k exp ( zk/T ) , for each class j by solving a multitask regression problem with cross-entropy loss , LT ( θ ) = − ∑ j qj ( z/T ) log ( qj ( g ( x ; θ ) /T ) ) . Hinton et al . ( 2015 ) showed that , under a zero-mean logit assumption , cross-entropy regression recovers L2 logit matching as T → ∞ ; however , the two approaches can differ for small T . In Sec . 3 , we will experiment with both of these popular compression approaches . 2.2 RANDOM FOREST COMPRESSION . Random forests ( Breiman , 2001 ) construct highly accurate prediction rules by averaging the predictions of a diverse and often large collection of learned decision trees . Effectively mimicking a large random forest with a single decision tree or a small forest has the potential to reduce prediction computation and storage costs by multiple orders of magnitude ( Bucila et al. , 2006 ; Joly et al. , 2012 ; Begon et al. , 2017 ; Painsky & Rosset , 2016 ; 2018 ) . Focusing on the common setting of binary classification with labels in { 0 , 1 } , we propose to train a student regression random forest to predict a teacher forest ’ s outputted probability p of a datapoint x having the label 1 . 2.3 GAN-ASSISTED MODEL COMPRESSION ( GAN-MC ) . In a typical compression setting , as much data as possible has been dedicated to training the highly accurate teacher model , leaving little fresh data for training the student model . While one branch of the model compression literature recommends generating synthetic data with customized augmentation algorithms for tabular Bucila et al . ( 2006 ) and image Urban et al . ( 2017 ) data , the more common solution in the distillation literature is to simply reuse the teacher training set as the compression set ( Hinton et al. , 2015 ; Frosst & Hinton , 2017 ) . However , we will see in Secs . 3 and 4 that compressing with training data alone leads to suboptimal student performance . To boost student performance and compression efficiency , we propose a simple solution applicable to tabular and image data alike : augment the compression set with synthetic feature vectors generated by a high-quality GAN . These synthetic feature vectors are then labeled with the true outputted teacher class probabilities or logits , as described in Secs . 2.1 and 2.2 . We call this approach GAN-assisted model compression ( GAN-MC ) . Intuition . Ideally , the student would be trained to mimic the predictions of the teacher on fresh feature vectors drawn from the true data distribution . However , synthetic data with a similar distribution can provide an effective surrogate for training an accurate student . The generator of a GAN for instance can produce an auxiliary stream of fake feature vectors by transforming independent noise vectors drawn from a simple distribution . The distributions of the synthetic and real data are encouraged to align via an adversarial game between a generator and a discriminator . To generate high-quality GAN feature vectors which capture the salient features of each class , we use the auxiliary classifier GAN ( AC-GAN ) of Odena et al . ( 2017 ) . The AC-GAN generator G produces a synthetic feature vector Xfake = G ( W , C ) given a random noise vector W and an independent target class label C drawn from the real data class distribution . For any given feature vector x , the AC-GAN discriminator D predicts both the probability of each class label P ( C | x ) and the probability of the data source being real or fake , P ( S | x ) for S ∈ { real , fake } . Given a training datasetDreal of labeled feature vectors , two components contribute to the AC-GAN training objective : Lsource = 1 |Dreal| ∑ ( x , c ) ∈Dreal logP ( S = real | x ) +EW , C∼pc [ logP ( S = fake | G ( W , C ) ) ] and Lclass = 1 |Dreal| ∑ ( x , c ) ∈Dreal logP ( C = c | x ) + EW , C∼pc [ logP ( C | G ( W , C ) ) ] , ( 2a ) representing the expected conditional log-likelihood of the correct source and the correct class of a feature vector , respectively . In the adversarial game , the generatorG is trained to maximize Lclass− Lsource , and the discriminator D is trained to maximize Lclass + Lsource . It should be noted that there is an important distinction between training a student to mimic a teacher with GAN data and training a student to solve the original supervised learning problem with GAN data . The goal of the original supervised learning task is to approximate the ideal mapping f∗ between inputs x and outputs y . This ideal f∗ is a functional of the true but unknown distribution underlying our data , and our information concerning f∗ is limited by the real data we have collected . The goal in model compression is to approximate the teacher prediction function g which maps from inputs to predictions z . Because the teacher is a function of the training data alone , g itself is a functional of the training data alone and is otherwise independent of the unknown distribution that generated that data . In addition , because we have access to the teacher , we have the freedom to query the function g at any point , and hence our information concerning g is limited only by number of queries we can afford . In particular , when we generate a new query point x , we can observe the actual target value of interest , the teacher ’ s prediction g ( x ) ; this is not true however for the supervised learning task , where no new labels can be observed . We believe these properties make the model compression task a much more tractable one and one that is ideal for data augmentation with generative models . For further discussion on the distinctions between model compression and the original supervised learning task , we refer the reader to ( Ba & Caruana , 2014 ) .

Summary: The paper proposes an approach for improving standard techniques for model compression, i.e. compressing a big model (teacher) in a smaller and more computationally efficient one (student), using data generated by a conditional GAN (cGAN). The paper suggests that the standard practice of training the student to imitate the behavior of the teacher *on the same training data* that the teacher was trained on is problematic and can lead to overfitting. Instead, the paper proposes learning a conditional GAN, which can potentially generate large amounts of realistic synthetic data, and use this data (in addition to original training data) for model compression.