Generative Teaching Networks: Accelerating Neural Architecture Search by Learning  to Generate Synthetic Training Data

1 INTRODUCTION AND RELATED WORK . Access to vast training data is now common in machine learning . However , to effectively train neural networks ( NNs ) does not require using all available data . For example , recent work in curriculum learning ( Graves et al. , 2017 ) , active learning ( Konyushkova et al. , 2017 ; Settles , 2010 ) and core-set selection ( Sener & Savarese , 2018 ; Tsang et al. , 2005 ) demonstrates that a surrogate dataset can be created by intelligently sampling a subset of training data , and that such surrogates enable competitive test performance with less training effort . Being able to more rapidly determine the performance of an architecture in this way could particularly benefit architecture search , where training thousands or millions of candidate NN architectures on full datasets can become prohibitively expensive . From this lens , related work in learning-to-teach has shown promise . For example , the learning to teach ( L2T ) ( Fan et al. , 2018 ) method accelerates learning for a NN learner ( hereafter , just learner ) through reinforcement learning , by learning how to subsample mini-batches of data . A key insight in this paper is that the surrogate data need not be drawn from the original data distribution ( i.e . they may not need to resemble the original data ) . For example , humans can learn new skills from reading a book or can prepare for a team game like soccer by practicing skills , such as passing , dribbling , juggling , and shooting . This paper investigates the question of whether we can train a data-generating network that can produce synthetic data that effectively and efficiently teaches a target task to a learner . Related to the idea of generating data , Generative Adversarial Networks ( GANs ) can produce impressive high-resolution images ( Goodfellow et al. , 2014 ; Brock et al. , 2018 ) , but they are incentivized to mimic real data ( Goodfellow et al. , 2014 ) , instead of being optimized to teach learners more efficiently than real data . Another approach for creating surrogate training data is to treat the training data itself as a hyperparameter of the training process and learn it directly . Such learning can be done through metagradients ( also called hyper-gradients ) , i.e . differentiating through the training process to optimize a meta-objective . This approach was described in Maclaurin et al . ( 2015 ) , where 10 synthetic training images were learned using meta-gradients such that when a network is trained on these images , the network ’ s performance on the MNIST validation dataset is maximized . In recent work concurrent with our own , Wang et al . ( 2019b ) scaled this idea to learn 100 synthetic training examples . While the 100 synthetic examples were more effective for training than 100 original ( real ) MNIST training examples , we show that it is difficult to scale this approach much further without the regularity across samples provided by a generative architecture ( Figure 2b , green line ) . Being able to very quickly train learners is particularly valuable for neural architecture search ( NAS ) , which is exciting for its potential to automatically discover high-performing architectures , which otherwise must be undertaken through time-consuming manual experimentation for new domains . Many advances in NAS involve accelerating the evaluation of candidate architectures by training a predictor of how well a trained learner would perform , by extrapolating from previously trained architectures ( Luo et al. , 2018 ; Liu et al. , 2018a ; Baker et al. , 2017 ) . This approach is still expensive because it requires many architectures to be trained and evaluated to train the predictor . Other approaches accelerate training by sharing training across architectures , either through shared weights ( e.g . as in ENAS ; Pham et al . ( 2018 ) ) , or Graph HyperNetworks ( Zhang et al. , 2018 ) . We propose a scalable , novel , meta-learning approach for creating synthetic data called Generative Teaching Networks ( GTNs ) . GTN training has two nested training loops : an inner loop to train a learner network , and an outer-loop to train a generator network that produces synthetic training data for the learner network . Experiments presented in Section 3 demonstrate that the GTN approach produces synthetic data that enables much faster learning , speeding up the training of a NN by a factor of 9 . Importantly , the synthetic data in GTNs is not only agnostic to the weight initialization of the learner network ( as in Wang et al . ( 2019b ) ) , but is also agnostic to the learner ’ s architecture . As a result , GTNs are a viable method for accelerating evaluation of candidate architectures in NAS . Indeed , controlling for the search algorithm ( i.e . using GTN-produced synthetic data as a drop-in replacement for real data when evaluating a candidate architecture ’ s performance ) , GTN-NAS improves the NAS state of the art by finding higher-performing architectures than comparable methods like weight sharing ( Pham et al. , 2018 ) and Graph HyperNetworks ( Zhang et al. , 2018 ) ; it also is competitive with methods using more sophisticated search algorithms and orders of magnitude more computation . It could also be combined with those methods to provide further gains . One promising aspect of GTNs is that they make very few assumptions about the learner . In contrast , NAS techniques based on shared training are viable only if the parameterizations of the learners are similar . For example , it is unclear how weight-sharing or HyperNetworks could be applied to architectural search spaces wherein layers could be either convolutional or fully-connected , as there is no obvious way for weights learned for one layer type to inform those of the other . In contrast , GTNs are able to create training data that can generalize between such diverse types of architectures . GTNs also open up interesting new research questions and applications to be explored by future work . Because they can rapidly train new architectures , GTNs could be used to create NNs ondemand that meet specific design constraints ( e.g . a given balance of performance , speed , and energy usage ) and/or have a specific subset of skills ( e.g . perhaps one needs to rapidly create a compact network capable of three particular skills ) . Because GTNs can generate virtually any learning environment , they also one day could be a key to creating AI-generating algorithms , which seek to bootstrap themselves from simple initial conditions to powerful forms of AI by creating an openended stream of challenges ( learning opportunities ) while learning to solve them ( Clune , 2019 ) . 2 METHODS . The main idea in GTNs is to train a data-generating network such that a learner network trained on data it rapidly produces high accuracy in a target task . Unlike a GAN , here the two networks cooperate ( rather than compete ) because their interests are aligned towards having the learner perform well on the target task when trained on data produced by the GTN . The generator and the learner networks are trained with meta-learning via nested optimization that consists of inner and outer training loops ( Figure 1a ) . In the inner-loop , the generator G ( z , y ) takes Gaussian noise ( z ) and a label ( y ) as input and outputs synthetic data ( x ) . Optionally , the generator could take only noise as input and produce both data and labels as output ( Appendix F ) . The learner is then trained on this synthetic data for a fixed number of inner-loop training steps with any optimizer , such as SGD or Adam ( Kingma & Ba , 2014 ) : we use SGD with momentum in this paper . SI Equation 1 defines the inner-loop SGD with momentum update for the learner parameters θt . We sample zt ( noise vectors input to the generator ) from a unit-variance Gaussian and yt labels for each generated sample ) uniformly from all available class labels . Note that both zt and yt are batches of samples . We can also learn a curriculum directly by additionally optimizing zt directly ( instead of sampling it randomly ) and keeping yt fixed throughout all of training . The inner-loop loss function ` inner can be cross-entropy for classification problems or mean squared error for regression problems . Note that the inner-loop objective does not depend on the outerloop objective and could even be parameterized and learned through meta-gradients with the rest of the system ( Houthooft et al. , 2018 ) . In the outer-loop , the learner θT ( i.e . the learner parameters trained on synthetic data after the T inner-loop steps ) is evaluated on the real training data , which is used to compute the outer-loop loss ( aka meta-training loss ) . The gradient of the meta-training loss with respect to the generator is computed by backpropagating through the entire inner-loop learning process . While computing the gradients for the generator we also compute the gradients of hyperparameters of the inner-loop SGD update rule ( its learning rate and momentum ) , which are updated after each outer-loop at no additional cost . To reduce memory requirements , we leverage gradientcheckpointing ( Griewank & Walther , 2000 ) when computing meta-gradients . The computation and memory complexity of our approach can be found in Appendix D. A key motivation for this work is to generate synthetic data that is learner agnostic , i.e . that generalizes across different potential learner architectures and initializations . To achieve this objective , at the beginning of each new outer-loop training , we choose a new learner architecture according to a predefined set and randomly initialize it ( details in Appendix A ) . Meta-learning with Weight Normalization . Optimization through meta-gradients is often unstable ( Maclaurin et al. , 2015 ) . We observed that this instability greatly complicates training because of its hyperparameter sensitivity , and training quickly diverges if they are not well-set . Combining the gradients from Evolution Strategies ( Salimans et al. , 2017 ) and backpropagation using inverse variance weighting ( Fleiss , 1993 ; Metz et al. , 2019 ) improved stability in our experiments , but optimization still consistently diverged whenever we increased the number of inner-loop optimization steps . To mitigate this issue , we introduce applying weight normalization ( Salimans & Kingma , 2016 ) to stabilize meta-gradient training by normalizing the generator and learner weights . Instead of updating the weights ( W ) directly , we parameterize them as W = g · V/‖V ‖ and instead update the scalar g and vector V . Weight normalization eliminates the need for ( and cost of ) calculating ES gradients and combining them with backprop gradients , simplifying and speeding up the algorithm . We hypothesize that weight normalization will help stabilize meta-gradient training more broadly , although future work is required to test this hypothesis in meta-learning contexts besides GTNs . The idea is that applying weight normalization to meta-learning techniques is analogous to batch normalization for deep networks ( Ioffe & Szegedy , 2015 ) . Batch normalization normalizes the forward propagation of activations in a long sequence of parameterized operations ( a deep NN ) . In meta-gradient training both the activations and weights result from a long sequence of parameterized operations and thus both should be normalized . Results in section 3.1 support this hypothesis . Learning a Curriculum with Generative Teaching Networks . Previous work has shown that a learned curriculum can be more effective than training from uniformly sampled data ( Graves et al. , 2017 ) . A curriculum is usually encoded with indexes to samples from a given dataset , rendering it non-differentiable and thereby complicating the curriculum ’ s optimization . With GTNs however , a curriculum can be encoded as a series of input vectors to the generator ( i.e . instead of sampling the zt inputs to the generator from a Gaussian distribution , a sequence of zt inputs can be learned ) . A curriculum can thus be learned by differentiating through the generator to optimize this sequence ( in addition to the generator ’ s parameters ) . Experiments confirm that GTNs more effectively teach learners when optimizing such a curriculum ( Section 3.2 ) . Accelerating NAS with Generative Teaching Networks . Since GTNs can accelerate learner training , we propose harnessing GTNs to accelerate NAS . Rather than evaluating each architecture in a target task with a standard training procedure , we propose evaluating architectures with a metaoptimized training process ( that generates synthetic data in addition to optimizing inner-loop hyperparameters ) . We show that doing so significantly reduces the cost of running NAS ( Section 3.4 ) . The goal of these experiments is to find a high-performing CNN architecture for the CIFAR10 image-classification task ( Krizhevsky et al. , 2009 ) with limited compute costs . We use the same architecture search-space , training procedure , hyperparameters , and code from Neural Architecture Optimization ( Luo et al. , 2018 ) , a state-of-the-art NAS method . The search space consists of the topology of two cells : a reduction cell and a convolutional cell . Multiple copies of such cells are stacked according to a predefined blueprint to form a full CNN architecture ( see Luo et al . ( 2018 ) for more details ) . The blueprint has two hyperparameters N and F that control how many times the convolutional cell is repeated ( depth ) and the width of each layer , respectively . Each cell contains B = 5 nodes . For each node within a cell , the search algorithm has to choose two inputs as well as two operations to apply to those inputs . The inputs to a node can be previous nodes or the outputs of the last two layers . There are 11 operations to choose from ( Appendix C ) . Following Luo et al . ( 2018 ) , we report the performance of our best cell instantiated with N = 6 , F = 36 after the resulting architecture is trained for a significant amount of time ( 600 epochs ) . Since evaluating each architecture in those settings ( named final evaluation from now on ) is time consuming , Luo et al . ( 2018 ) uses a surrogate evaluation ( named search evaluation ) to estimate the performance of a given cell wherein a smaller version of the architecture ( N = 3 , F = 32 ) is trained for less epochs ( 100 ) on real data . We further reduce the evaluation time of each cell by replacing the training data in the search evaluation with GTN synthetic data , thus reducing the training time per evaluation by 300x ( which we call GTN evaluation ) . While we were able to train GTNs directly on the complex architectures from the NAS search space , training was prohibitively slow . Instead , for these experiments , we optimize our GTN ahead of time using proxy learners described in Appendix A.2 , which are smaller fully-convolutional networks ( this meta-training took 8h on one p6000 GPU ) . Interestingly , although we never train our GTN on any NAS architectures , because of generalization , synthetic data from GTNs were still effective for training them .

Summary: This paper proposes a meta-learning algorithm Generative Teaching Networks (GTN) to generate fake training data for models to learn more accurate models. In the inner loop, a generator produces training data and the learner takes gradient steps on this data. In the outer loop, the parameters of the generator are updated by evaluating the learner on real data and differentiating through the gradient steps of the inner loop. The main claim is this method is shown to give improvements in performance on supervised learning for MNIST and CIFAR10. They also suggest weight normalization patches up instability issues with meta-learning and evaluate this in the supervised learning setting, and curriculum learning for GTNs.