Generative Teaching Networks: Accelerating Neural Architecture Search by Learning  to Generate Synthetic Training Data
Authors: Felipe Petroski Such
Aditya Rawal
Joel Lehman
Kenneth Stanley
Jeff Clune
Venue:  (2020)
Keywords: Generative models
generating synthetic data
neural architecture search
learning to teach
meta-learning

Abstract: This paper investigates the intriguing question of whether we can create learning algorithms that automatically generate training data, learning environments, and curricula in order to help AI agents rapidly learn. We show that such algorithms are possible via Generative Teaching Networks (GTNs), a general approach that is applicable to supervised, unsupervised, and reinforcement learning. GTNs are deep neural networks that generate data and/or training environments that a learner (e.g.\ a freshly initialized neural network) trains on before being tested on a target task. We then differentiate \emph{through the entire learning process} via meta-gradients to update the GTN parameters to improve performance on the target task. GTNs have the beneficial property that they can theoretically generate any type of data or training environment, making their potential impact large. This paper introduces GTNs, discusses their potential, and showcases that they can substantially accelerate learning. We also demonstrate a practical and exciting application of GTNs: accelerating the evaluation of candidate architectures for neural architecture search (NAS), which is rate-limited by such evaluations, enabling massive speed-ups in NAS. GTN-NAS improves the NAS state of the art, finding higher performing architectures when controlling for the search proposal mechanism. GTN-NAS also is competitive with the overall state of the art approaches, which achieve top performance while using orders of magnitude less computation than typical NAS methods. Overall, GTNs represent a first step toward the ambitious goal of algorithms that generate their own training data and, in doing so, open a variety of interesting new research questions and directions.

Under review as a conference paper at ICLR 2020
GENERATIVE TEACHING NETWORKS: ACCELERATING NEURAL ARCHITECTURE SEARCH BY LEARNING TO GENERATE SYNTHETIC TRAINING DATA
Anonymous authors Paper under double-bling review
ABSTRACT
This paper investigates the intriguing question of whether we create learning algorithms that automatically generate training data, learning environments, and curricula in order to help Al agents rapidly learn. We show that such algorithms are possible via Generative Teaching Networks (GTNs), a general approach that in theory, applicable supervised, unsupervised, and reinforcement learning. although our experiments only focus on the supervised case. GTNs are deep neu- ral networks that generate data and/or training environments that a learner (e.g. a freshly initialized neural network) trains on for a few SGD steps before being tested on a target task. We then differentiate through the entire learning process via meta-gradients to update the GTN parameters to improve performance the target task. GTNs have the beneficial property that they can theoretically gener- ate any type of data training environment, making their potential impact large. This paper introduces GTNs, discusses their potential, and showcases that they can substantially accelerate learning. We also demonstrate practical and excit- ing application of GTNs: accelerating the evaluation of candidate architectures for neural architecture search (NAS), which is rate-limited by such evaluations, enabling massive speed-ups in NAS. GTN-NAS improves the NAS state of the art, finding higher performing architectures when controlling for the search pro- posal mechanism. GTN-NAS also is competitive with the overall state of the art approaches, which achieve top performance while using orders of magnitude less computation than typical NAS methods. Speculating forward, GTNS may repre- sent a first step toward the ambitious goal algorithms that generate their own training data and, in doing so, open variety of interesting new research questions and directions.
INTRODUCTION AND RELATED WORK
Access to vast training data is common in machine learning. However, to effectively train neural networks (NNs) does not require using all available data. For example, recent work in cur- riculum learning (Graves al., 2017), active learning (Konyushkova al., 2017: Settles, 2010) and core-set selection (Sener & Savarese, 2018; Tsang et al., 2005) demonstrates that a surrogate dataset can be created by intelligently sampling subset of training data, and that such surrogates enable competitive test performance with less training effort. Being able to more rapidly determine the per- formance of an architecture in this way could particularly benefit architecture search, where training thousands or millions of candidate NN architectures on full datasets can become prohibitively ex- pensive. From this lens, related work in learning-to-teade has shown promise. For example, the learning to teach (I.2T) (Fan et al., 2018) method accelerates learning for a NN learner (hereafter, just learner) through reinforcement learning, by learning how to subsample mini-batches of data.
A key insight in this paper is that the surrogate data need not be drawn from the original data distribution (i.e. they may not need to resemble the original data). For example, humans can learn new skills from reading book can prepare for team game like soccer by practicing skills, such as passing, dribbling, juggling, and shooting. This paper investigates the question of whether we can train geaaraennnrnngggggg network that can produce synthetic data that effectively and efficiently teaches a target task to a learner. Related to the idea of generating data, Generative Adversarial Networks (GANs) can produce impressive high-resolution images (Goodfellow et al., 2014; Brock

---

Under review as a conference paper at ICLR 2020
et al., 2018), but they incentivized to mimic real data (Goodfellow et al., 2014), instead of being optimized to teach learners more efficiently than real data.
Another approach for creating surrogate training data is to treat the training data itself hyper- parameter of the training process and learn it directly. Such learning can be done through meta- gradients (also called h hyper-gradients), i.e. differentiating through the training process to optimize meta-objective. This approach was described in Maclaurin et al. (2015), where 10 synthetic training images were learned using meta- meta-gradients such that when network trained on these images, the network's performance on the MNIST validation dataset is maximized. In recent work concurrent with our own, Wang et al. (2019b) scaled this idea to learn 100 synthetic training examples. While the 100 synthetic examples were more effective for training than 100 original (real) MNIST training examples, we show that difficult to scale this approach much further without the regularity across samples provided by generative architecture (Figure 2h, green line).
Being able to very quickly train learners is particularly valuable for neural architecture search (NAS), which is exciting for its potential to automatically discover high-performing architectures, which otherwise must be undertaken through time-consuming manual experimentation for new domains. Many advances in NAS involve accelerating the evaluation of candidate architectures by training a predictor of how well a trained learner would perform. by extrapolating from previously trained architectures (Luo et al., 2018; Liu et al., 2018a; Baker et al., 2017). This approach is still expensive because it requires many architectures to be trained and evaluated to train the predictor. Other approaches accelerate training by sharing training across architectures, either through shared weights (e.g. as in ENAS; Pham et al. (2018)), or Graph HyperNetworks (Zhang et al., 2018).
We propose scalable, novel, meta-learning approach for creating synthetic data called Generative Teaching Networks (GTNs). GTN training has two nested training loops: an inner loop to train a learner network, and an outer-loop to train a generator network that produces synthetic training data for the learner network. Experiments presented in Section 3 demonstrate that the GTN approach produces synthetic data that enables much faster learning, speeding up the training of NN by a fac- tor of 9. Importantly, the synthetic data in GTNs is not only agnostic to the weight initialization of the learner network (as in Wang et al. (2019b)), but is also agnostic to the learner's 's architecture. As result, GTNs are a viable method for accelerating evaluation of candidate architectures in NAS. Indeed, controlling for the search algorithm (i.e. using GTN-produced synthetic data as a drop-in replacement for real data when evaluating candidate architecture's 's performance). GTN-NAS im- proves the NAS state of the art by finding figher-performing architectures than comparable methods like weight sharing (Pham et al., 2018) and Graph HyperNetworks (Zhang al., 2018); it also is competitive with methods using more sophisticated search algorithms and orders of magnitude more computation. It could also be combined with those methods provide further gains.
One promising aspect of GTNs is that they make very few assumptions about the learner. In contrast, NAS techniques based on shared training are viable only if the parameterizations of the learners are similar. For example, it is unclear how weight-sharing or HyperNetworks could be applied to architectural search spaces wherein layers could be either convolutional or dully-connected as there is no obvious way for weights learned for one layer type to inform those of the other. In contrast, GTNs are able to create training data that can generalize between such diverse types of architectures.
GTNS also open up interesting new research questions and applications to be explored by future work. Because they can rapidly train new architectures, GTNs could be used to create NNs on- demand that meet specific design constraints (e.g. given balance of performance, speed, and en- ergy usage) and/or have a specific subset of skills (e.g. perhaps one needs to rapidly create a compact network capable of three particular skills). Because GTNs can generate virtually any learning en- vironment, they also one day could be a key to creating Al-generating algorithms,, which seek to bootstrap themselves from simple initial conditions to powerful forms of AI by creating open- ended stream of challenges (learning opportunities) while learning to solve them (Clune, 2019).
METHODS
The main idea in GTNs is to train a data- tatta-generatig network such that a learner network trained on data it rapidly produces high accuracy in target task. Unlike GAN, here the two networks cooperate (rather than compete) because their interests are aligned towards having the learner per- form well on the target task when trained data produced by the GTN. The generator and the
2

---

Under review as a conference paper at ICLR 2020
learner networks are trained with meta-learning via nested optimization that consists of inner and outer training loops (Figure la). In the inner-loop, the generator G(z, y) takes Gaussian noise (z) and label (y) as imput and outputs synthetic data (z). Optionally, the generator could take only noise as input and produce both data and labels as output (Appendix F). The learner is then trained on this synthetic data for fixed number of inner-loop training steps with any optimizer, such as SGD or Adam (Kingma & Ba, 2014): we use SGD with momentumn in this paper. SI Equation defines the inner-loop SGD with momentum update for the learner parameters 0r. We sample z (noise vectors input to the generator) from a unit-variance Gaussian and y, labels for each generated sample) uniformly from all available class labels. Note that both z; and y, are batches of samples. We can also learn a curriculum directly by additionally optimizing z, directly (instead of sampling randomly) and keeping y, fixed throughout all of training.
The inner-loop loss function linner can be cross-entropy for classification problems mean squared error for regression problems. Note that the inner-loop objective does not depend on the outer- loop objective and could even be parameterized and learned through meta-gradients with the rest of the system (Houthooft et al., 2018). In the outer-loop, the learner 02 (i.e. the learner parameters trained on synthetic data after the T inner-loop steps) is evaluated on the real training data, which is used to compute the outer-loop loss (aka meta- training loss). The gradient of the meta- veea-training loss with respect to the generator is computed by backpropagating through the entire inner-loop learning process. While computing the gradients for the generator we also compute the gradients of hyper- parameters of the inner-loop SGD update rule (its learning rate and momentum), which are updated after each outer-loop at no additional cost. To reduce memory requirements, we leverage gradient- checkpointing (Griewank & Walther, 2000) when computing meta-gradientt, The computation and memory complexity of our approach can be found in Appendix D.
Outer-lopp
1.500
Inner-loop
(4) Meta-loss
Valitation Lave
0.975
0.550
(2) Data
Real Data
Generator
lerrner
0.935
0.000
Shuftea
Shuff ed twich
(3) SED Sxep
0.875
Full Carriculum
0.550
(5) Gradient Meta-loss w.r.t. Generator
0.der4oop faaracions
(a) Overview of Generative Teaching Networks (b) GTN stability with WN (c) GTN curricula comparison
Figure I: (a) Generative Teaching Network (GTN) Method. The numbers in the figure reflect the order in which GTN is executed. Noise is fed as an input to the Generator (I), which uses it to gen- erate new data (2). The learner is trained (e.g. using SGD or Adam) to perform well on the generated data (3). The trained learner is then evaluated on the real training data in the outer-loop to compute the outer-loop meta-loss (4). The gradients of the generator parameters are computed w.r.t. to the meta-loss to update the generator (5). Both learned curriculum and weight normalization sub- stantially improve GTN performance. (b) Weight normalization improves peeta-gradient training of GTNs, and makes the method much more robust to differentt hyperparameter settings. Each boxplot reports the final loss of 20 runs obtained during hyperparameter optimization with Bayesian Opti- mization (lower is better). (c) shows a comparison between GTNs with different types of curricula. The GTN method with the most control over how samples are presented performs the best.
A key motivation for this work is to generate synthetic data that is learner agnostic, i.e. that gener- alizes across different potential learner architectures and initializations. To achieve this objective, at the beginning of each new outer-loop training, we choose new learner architecture according to a predefined set and randomly initialize it (details in Appendix A).
Meta-learning with Weight Normalization. Optimization through meta meta-gradients is often unsta- ble (Maclaurin et al., 2015). We observed that this instability greatly complicates training because of its hyperparameter sensitivity, and training quickly diverges they are not well-set. Combining the gradients from Evolution Strategies (Salimans et al., 2017) and backpropagation using inverse variance weighting (Fleiss, 1993; Metz et al., 2019) improved stability in our experiments, but op- timization still consistently diverged whenever we increased the number of inner-loop optimization steps. To mitigate this issue, we introduce applying weight normalization (Salimans & Kingma, 2016) to stabilize meta-gradient training by normalizing the generator and learner weights. Instead
3

---

Under review as a conference paper at ICLR 2020
of updating the weights (W) directly, we parameterize them as W g --VV/ the scalar g and vector V. Weight normalization eliminates the need for (and cost of) calculating ES gradients and combining them with backprop gradients, simplifying and speeding up the algorithm.
V/|IV||
and instead update
We hypothesize that weight normalization will help stabilize meta-gradient training more broadly, although future work is required to test this hypothesis in meta-learning contexts besides GTNs. The idea is that applying weight normalization to meta-learning techniques is analogous to batch normalization for deep networks (Ioffe & Szegedy, 2015). Batch normalization normalizes the forward propagation of activations in a long sequence of parameterized operations (a deep NN). In meta- deta-gradient training both the activations and weights result from long sequence of parameterized operations and thus both should be normalized. Results in section 3.1 support this hypothesis.
Learning a Curriculum with Generative Teaching Networks. Previous work has shown that a learned curriculum can be more effective than training from uniformly sampled data (Graves et al., 2017). A curriculum usually encoded with indexes to samples from a given dataset, rendering non-differentiable and thereby complicating the curriculum's 's optimization. With GTNs however, curriculum can be encoded as series of input vectors to the generator (i.e. instead of sampling the Z inputs to the generator from a Gaussian distribution, sequence of zu inputs can be learned). A curriculum can thus be learned by differentiating through the generator to optimize this sequence (in addition to the generator's 's parameters). Experiments confirm that GTNs more effectively teach learners when optimizing such a curriculum (Section 3.2).
Accelerating NAS with Generative Teaching Networks. Since GT's can accelerate learner train- ing, we propose harnessing GTNS to accelerate NAS. Rather than evaluating each architecture in a target task with a standard training procedure, we propose evaluating architectures with meta- optimized training process (that generates synthetic data in addition to optimizing inner-loop hyper- parameters). We show that doing so significantly reduces the cost of running NAS (Section 3.4).
The goal of these experiments is to find high-performing CNN architecture for the CIFARIO image-llasiffiation task (Krizhevsky et al., 2009) with limited compute costs. We use the same architecture search-space, training procedure, hyperparameters, and code from Neural Architecture Optimization (Luo et al., 2018), a state-of the-eher art NAS method. The search space consists of the topology of two cells: reduction cell and convolutional cell. Multiple copies of such cells are stacked according to a predefined blueprint to form a full CNN architecture (see Luo et al. (2018) for more details). The blueprint has two hyperparameters N and F that control how many times the convolutional cell is repeated (depth) and the width of each layer, respectively. Each cell contains B = 5 nodes. For each node within cell, the search algorithm has to choose two inputs as well as two operations to apply to those inputs. The inputs to a node can be previous nodes or the outputs of the last two layers. There 11 operations choose from (Appendix C).
Following Luo et al. (2018), we report the performance of our best cell instantiated with N = 6, F 36 after the resulting architecture is trained for significant amount of time (600 epochs). Since evaluating each architecture in those settings (named final evaluation from now on) is time consuming, Luo al. (2018) uses surrogate evaluation (named search evaluation) to estimate the performance of given cell wherein a smaller version of the architecture (N = F = 32) trained for less epochs (100) on real data. We further reduce the evaluation time of each cell by replacing the training data in the search evaluation with GTN synthetic data, thus reducing the training time per evaluation by 300x (which we call GTN evaluation). While we were able to train GTNs directly on the complex architectures from the NAS search space, training was prohibitively slow. Instead, for these experiments, we optimize our GTN ahead of time using proxy learners described in Appendix A.2, which are smaller fully-oonvolutional networks (this meta-training took 8h on one p6000 GPU). Interestingly, although we never train our GTN on any NAS architectures, because of generalization, synthetic data from GTNs were still effective for training them.
3 RESULTS
We first demonstrate that weight normalization significantly improves the stability of meta-learning, an independent contribution of this paper (Section 3.1). We then show that training with synthetic data is more effective when learning such data jointly with curriculum that orders its presentation to the learner (Section 3.2). We next show that GTNs can generate a synthetic training set that enables more rapid learning in a few SGD steps than real training data in two supervised learning
4

---

Under review as a conference paper at ICLR 2020
domains (MNIST and CIFARIO) and in a reinforcement learning domain (cart-pole, Appendix H). We then apply GTTN-sythhee training data for neural architecture search to find high performing architectures for CIFARIO with limited compute, outperforming comparable methods like weight sharing (Pham et al., 2018) and Graph HyperNetworks (Zhang et al., 2018) (Section 3.4).
We uniformly split the usual MNIST training set into training (50k) and validation sets (10k). The training set was used for inner-loop training (for the baseline) and to compute meta-grade meta-gradients for all the treatments. We used the validation set for hyperparameter tuning and report accuracy on the usual MNIST test set (10k images). We followed the same procedure for CIFARIO, 10, resulting in training. validation, and test sets with 45k, 5k, and 10k examples, respectively. Unless otherwise specified, we ran each experiment 5 times and plot the mean and its 95% confidence intervals from (n=1,000) bootstrapping. Appendix A describes additional experimental details.
3.1 IMPROVING STABILITY WITH WEIGHT NORMAI.IZATION
To demonstrate the effectiveness of weight normalization for stabilizing and robustifying meta- optimization, we compare the results of running hyperparameter optimization for GTNs with and without weight normalization on MNIST. Figure 1 shows the distribution of the final performance obtained for 20 runs during hyperparameter tuning, which reflects how sensitive the algorithms are to hyperparameter settings. Overall, weight normalization substantially improved robustness to hyperparameters and final learner performance, supporting the initial hypothesis.
3.2 IMPROVING GTNs WITH A CURRICULUM
We experimentally evaluate four different variants of GTNs, each with increasing control over the ordering of the z codes input to the generator, and thus the order of the inputs provided to the learner. The first variant (called GTN No Curriculum), trains generator to output synthetic training data by sampling the noise vector 2 for each sample independently from a Gaussian distribution. In the next three GTN variants, the generator is provided with fixed set of input samples (instead of noise vector). These input samples are learned along with the generator parameters during GTN training. The second GTN variant (called GTN All Shuffled) learns a fixed set of 4,096 input samples that are presented in a random order without replacement (thus learning controls the data, but not the order which they are presented). The third variant (called GTN Shuffled Batch) learns 32 batches of 128 samples each (so learning controls which samples coexist within batch), but the order which the batches are presented randomized (without replacement). Finally, the fourth variant (called GTN Full Curriculum) learns a deterministic sequence of 32 batches of 128 samples, giving learning full control. Learning such a curriculum incurs no additional computational expense, as learning the z, tensor is computationally negligible and avoids the cost of repeatedly sampling new Gaussian z codes. We plot the test accuracy of a learner (with random initial weights and architecture) as function of outer-loop iterations for all four variants in Figure Ic. Although GTNs - No curriculum can seemingly generate endless data (see Appendix G). it performs worse than the other three variants with a fixed set of generator inputs. Overall, training the GTN with exact ordering of input samples (GTN- Full Curriculum) outperforms all other variants.
While curriculum learning usually refers training on easy tasks first and increasing their difficulty over time, our curriculum goes beyond presenting tasks in a certain order. Specifically, GTN- Full Curriculum learns both the order in which present samples and the specific group of samples to present at the same time. The ability to learn a full curriculum improves GTN performance. For that reason, we adopt that approach for all GTN experiments.
3.3 GTNS FOR SUPERVISED LEARNING
To explore whether GTNs can generate training data that helps networks learn rapidly, we compare to treatments for MNIST classification. 1) Real Data -irainin learners with random mini-batches of real data, as is ubiquitous in SGD. 2) Dataset Distillation Training learners with synthetic data, where training examples are directly encoded as tensors optimized by the meta-objective, as in Wang et al. (2019b). 3) GTN Our method where the training data presented to the learner is generated by a neural network. Note that all three methods meta-optimize the inner-loop hyperparameters (i.e. the learning rate and momentum of SGD) as part of the meta-optimization.
We emphasize that producing sfate-of-the. the-art art (SOTA) performance (e.g. on MNIST or CIFAR)
when training with GTN-generated data is not important for GTNs. Because the ultimate aim for
5

---

Under review as a conference paper at ICLR 2020
GTNs is to accelerate NAS (Section 3.4), what matters is how well and inexpensively we can identify architectures that achieve high asymptotic accuracy when later trained on the full (real) training set. A means to that end is being able to train architectures rapidly, ie. with very few SGD steps. because doing so allows NAS to rapidly identify promising architectures. We are thus interested in "few-step accuracy (i.e. accuracy after few-e.g. 32 or 228-GGD steps). Besides, there are many reasons not to expect SOTA performance with GTNs (Appendix B).
0.01) and trains a learner to be much more accurate when in the few-step performance regime. Specifically, for each treatment the figure shows the test performance of a learner following 32 inner-loop training steps with batch size of 128. We would not expect training on synthetic data to produce higher accuracy than unlimited SGD steps on real data, but here the performance gain comes because GTNs can compress the real training data by producing synthetic data that enables learners to learn more quickly than on real data. For example, the original dataset might contain many similar images, where only a few of them would be sufficient for training (and GTN can produce just these few). GTN could also combine many different things that need to be learned about images into one image.
Figure 2a shows that the GTN treatment significantly outperforms the other ones
Figure 2b shows the few-step performance of learner from each treatment after 2000 total outer- loop iterations (~1 hour on p6000 GPU). For reference, Dataset Distillation (Wang et al., 2019b) reported 79.5% accuracy for randomly initialized network (using 100 synthetic images 4,096) and 1.2T (Fan et al., 2018) reported needing 300x more training iterations to achieve > 98% MNIST accuracy. Surprisingly, although recognizable as digits and effective for training, GTN- generated images (Figure 2c) were not visually realistic (see Discussion).
our
1.00
0.96
Training Accuracy
6:98
0.5
0.00
0.9
GTN
634
GTN
0.2
0:22
Real Data Dataset Distilation
Dataset Distillation
0.80
500
1000
1503
2000
3
15 20 Innerloop Iteratione
10
25
30
Duter-loop lterations
(a) Meta-training curves
(b) Training curves
(c) GTN-generated samples
Figure 2: Teaching MNIST with GTN -TN-eeeeetted images. (a) shows MNIST test set few-step ac- curacy across outer-lopp iterations for different sources of inner-loop training data. The inner-loop consists of 32 SGD steps and the outer-loop optimizes MNIST validation accuracy. Our method (GTN) outperforms the controls (dataset distillation and samples from real data). (b) shows, given final meta-training iteration, how iterations of inner-loop training compare between training data sources (measured by MNIST training set accuracy). (c) shows 100 random samples from the trained GTN, Samples are usually recognizable as digits, but are not realistic (see Discussion). Each column contains samples from a different digit class, and each row is taken from different inner-loop iterations (evenly spaced from the 32 total iterations, with early iterations at the top).
3.4 ARCHITECTURE SEARCH WITH GTNS
We next test the benefits of GTN for NAS (GTN-NAS) in CIFARIO, a domain where NAS has previously shown significant improvements over the best architectures produced by armies of hu- man scientists. Figure 3a shows the few-step training accuracy of learner trained with either GTN-synthetic data or real (CIFAR10) data over meta-training iterations. After 8h of meta- metnnnga,nnng training with GTN-generated data was significantly faster than with real data, as in MNIST
To explore the potential for GTN-NAS to accelerate CIFARIO architecture search, we investigated the Spearman rank correlation (across architectures sampled from the NAS search space) between accelerated GTN-trained network performance (GTN evaluation) and the usual more expensive per- formance metric used during NAS (search evaluation). A correlation plot is shown in Figure 3c; note that a strong correlation implies we can train architectures using GTN evaluation as an inexpensive surrogate. We find that GTN evaluation enables predicting the performance of architccture effi- ciently. The wunk-correlatio between 128 steps of training with CCTN-synthen data vs. 100 epochs of real data is 0.3606. The correlation improves to 0.5582 when considering the top 50% of archi-

---

Under review as a conference paper at ICLR 2020
tectures recommended by GTN evaluation scores, which is important because those the ones that search would select. This improved correlation is slightly stronger than that from 3 epochs of training with real data (0.5235), a 9x cost-reduction per trained model.
Real Deta Predicted Pert.
2.05
Training Accurncy
0.5
9:34
0.5
9:99
9:92
0:01
0.3
Roal Data
20
40
C0
00
100
120
0.1
0.2
0.3
044
0.5
Inner-loop Iterations
GTN Prodicted Performance
(a) CIFARIO inner-loop training
(b) CIFARIO GTN samples
(c) CIFARIO correlation
Figure 3: Teaching CIFARIO with GTN-generated images. (a) CIFARIO training set performance of the final learner (after 1,700 meta-optimization steps) across inner-loop learning iterations. (b) Samples generated by GTN to teach CIFAR are unrecognizable, despite being effective for train- ing. Each column contains a different class, and each row taken from the same inner-loop iteration (evenly spaced from all 128 iterations, early iterations at the top). (c) Correlation between perfor- mance prediction using GTN-data vs. Real Data. When considering the top half of architectures (as ranked by GTN evaluation). correlation between GTN evaluation and search evaluation is strong (0.5582 cank-correlation), suggesting that GTN-NAS has potential to uncover high performing ar- chitectures at a significantly lower cost. Architectures shown are uniformly sampled from the NAS search space. The top 10% of architectures according to the GTN evaluation (blue squares)- those likely to be selected by GTN-NAS have high true asymptotic accuracy.
Architecture search methods are composed of several semi-independerr components, such as the choice of search space, search algorithm, and proxy evaluation of candidate architectures. GTNs are proposed as an improvement to this last component, i.e. as a new way to quickly evaluate new architecture. Thus we test our method under the standard search space for CIFAR1Ó. using a simple form of search (random search) for which there previous benchmark results. In particular, an architecture search experiment where we evaluated 800 randomly generated architectures trained with GTN-synthetic data. We present the performance after final evaluation of the best architecture found in Table 1. This experimental setting is similar to that of Zhang et al. (2018). Highlighting achieve mate-of-the-ane
the potential of GTNs an improved proxy evaluation for architectures, results when controlling for search algorithm (the choice of which is orthogonal to our contribution). While it is an po-oes-to comparison, GTN-NAS is competitive even with methods that use more ad advanced search techniques than random search to propose architectures (Appendix E). GTN compatible with such techniques, and would likely improve their performance, an interesting area of future work. Furthermore, because of the NAS search space, the modules GTN found can be used to create even larger networks. further test of whether GTNS predictions generalize is such larger networks would continue performing better than architectures generated by the real- data control, similarly scaled, We tried F=128 and show it indeed does perform better (Table I), suggesting additional gains can be had by searching post-hoc for the correct F and N settings.
DISCUSSION, FUTURE WORK, AND CONCLUSION
The results presented here suggest potential future applications and extensions of GTNs. Given the ability of GTNs to rapidly train new models, they are particularly useful when training many independent models is required (as we showed for NAS). Another such application would be to teach networks on demand to realize particular trade-offs between e.g. accuracy, inference time, and memory requirements, While to address range of such trade-off's would ordinarily require training many models ahead of time and selecting amongst them (Elsken et al., 2019), GTNs could instead rapidly train network only when a particular trade-off is needed. Similarly, agents with unique combinations of skills could be created on demand when needed.
Interesting questions raised by the lack of similarity between the synthetic GTN data and real MNIST and CIFARIO data. That unrealistic and/or unrecognizable images can meaningfully affect NNs is reminiscent of the finding that deep neural networks are easily fooled by unrecognizable

---

Under review as a conference paper at ICLR 2020
Table 1: Performance of different architecture search methods. Our results report mean + SD of evaluations of the same architecture with different initializations. It is common to report scores with and without Cutout (DeVries & Taylor, 2017), a data augmentation technique used during training. We found better architectures compared to other methods that reduce architecture evaluation speed and were tested with random search (Random Search+WS and Random Search+GHN), Increasing the width of the architecture found (F=128) further improves performance. Because each NAS method finds a different architecture, the number of parameters differs. Each method ran once.
Model
Error( (%)
#params
GPU Days
Random Search GHN (Zhang al., 2018)
4.3 0.1
5.1M
0.42
Random Search + Weight Sharing (Luo et al., 2018)
3.92
3.9M
0.25
Random Search Real Data (baseline)
3.88 + 0.08
12.4M
10
Random Search + GTN (ours)
3.84 0.06
8.2M
0.67
Random Search Real Data + Cutout (baseline)
0.03
12.4M
10
Random Search GTN Cutout (ours)
2.92 + 0.06
8.2M
0.67
Random Search + Real Data + Cutout (F=128) (baseline)
2.51
151.7M
10
Random Search GTN Cutout (F=128) (ours)
2.42 0.03
97.9M
0.67
images (Nguyen et al., 2015). It is possible that if neural network architectures were functionally more similar to human brains, GTNs' synthetic data might more resemble real data. However, an alternate (speculative) hypothesis is that the human brain might also be able to rapidly learn an arbitrary skill by being shown unnatural, unrecognizable data (recalling the novel Snow Crash).
The improved stability of training GTNs from weight normalization naturally suggests the hypoth- esis that weight normalization might similarly stabilize, and thus meaningfully improve, any tech- niques based on meta- geaaiggrdssets (c.g. MAML (Finn et al., 2017), learned optimizers (Metz et al., 2019), and learned update rules (Metz et al., 2018)). In future work, we will more deeply investigate how consistently, and to what degree, this hypothesis holds.
Both weight sharing and GHNs can be combined with GTNs by using the shared weights Hyper- Network for initialization of proposed learners and then fine-tuning on GTN-produced data. GTNs could also be combined with more intelligent ways propose which architecture sample next such as NAO (Luo et al., 2018). Many other extensions would also be interesting to consider. GTNs could be trained for unsupervised learning, for example by training a useful embedding function. Additionally, they could be used to stabilize GAN training and prevent mode collapse (Appendix I shows encouraging initial results). One particularly promising extension is to introduce closed- loop curriculum (i.e. one that responds dynamically to the performance of the learner throughout training), which we believe could significantly improve performance. For example, a recurrent GTN that is conditioned on previous learner outputs could adapt its samples to be appropriately easier or more difficult depending on an agent's learning progress, similar in spirit to the approach a human tutor. Such closed-loop teaching can improve learning (Fan et al., 2018).
An additional interesting direction is having GTNs generate training environments for RL agents. Appendix H shows this works for the simple RL task of CartPole. That could be either for pre- defined target task, or could be combined with more open-ended algorithms that attempt to con- tinuously generate new, different, interesting tasks that foster learning (Clune, 2019; Wang et al., 2019a). Because GTNs can encode any possible environment, they (or something similar) may be necessary to have truly unconstrained, open-ended algorithms (Stanley et al., 2017). If techniques could be invented to coax GTNs to produce recognizable, human- momaan-maninnfull training environments, the technique could also produce interesting virtual worlds for us to learn in, play in, or explore.
This paper introduces a new method called Generative Teaching Networks, wherein data genera- tors are trained to produce effective training data through meta-learning. We have shown that such an approach can produce supervised datasets that yield better few-step accuracy than an equivalent amount of real training data, and generalize across architectures and random initializations. We leverage such efficient training data to create fast NAS method that generates -atee-of-the-at ar- chitectures (controlling for the search algorithm). While GTNs may be of particular interest to the field of architecture search (where the computational cost evaluate candidate architectures often limits the scope of its application). we believe that GTNs open up an intriguing and challenging line of research into a variety of algorithms that learn generate their training data.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. arXiv preprint arXiv:1705.10823, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1xXv::6666.0140 2016.
Jeff Clune. Ai-gas:
Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv preprint arXiv: /905. 10985, 2019.
Terrance Vriese and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 2017.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. In International Conference on Learning Representations, 2019.
Yang Fan, Fei Tian, Tao Qin, Xiang- Xiang-Yang Li, and Tie Liu. Learning teach. arXiv preprint arXiv:1805.03643, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
JL Fleiss. Review papers: The statistical basis of meta-analysis. Statistical methods in medical research, 2(2):121-145, 1993.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Alex Graves, Mare G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1311-1320, 2017.
Andreas Griewank and Andrea Walther. Algorithm 799: revolve: an implementation of check- pointing for the reverse adjoint mode of computational differentiation. ACM Transactions Mathematical Software (TOMS), 26(1): 2000.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Geoffrey F. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.02531, 2015.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAl Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Resa-Bianchi, and R. Garnetr (eds.), Advances in Neural Information Processing Systems 31, pp. 5405-5414. Curran Associates, Inc., 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint /00.:::50000016000 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69880 2014.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. In NIPS, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.

---

Under review as a conference paper at ICLR 2020
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. Proceed- ings of the European Conference Computer Vision (ECCV), pp. 19-34, 2018a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprinl arXiv: 1xxiv:800.09000 2018b.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. In Advances in neural information processing systems, pp. 7816-7827,200
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In Proc. ieml, volume 30, pp. 3, 2013.
Dougal Maclaurin, David Duvenaud, and Ryan P. Adams. Gradient-based hyperparameter opti- mization through reversible learning. In Proceedings of the 32Nd International Conference on lernational Conference on Machine Learning Volume 37, ICML 15, pp. 2113-2122. JMLR.org, 2015.
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Meta-learning up- date rules for unsupervised representation learning. arXiv preprint (0to:/80000022 2018.
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-dickstein. Learned optimizers that outperform on wall-clock and validation loss, 2019.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937, 2016.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High con- fidence predictions for unrecognizable images. In Computer Vision and Pattern Recognition (CVPR 15), 2015.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 4095-4104, Stockholmsmssan Stockholm Sweden, 10-15 Jul 2018. PMLR.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4780-4789, 2019.
Tim Salimans and Durk P Kingma. Weight normalization: simple reparametctizatioo to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp- 901-909, 2016.
Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, 2016.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative reinforcement learning. arXiv preprint arXiv:1703.03880 2017.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference Learning Representations, 2018.
Burr Settles. Active learning literature survey. Technical report, 2010.
Hava T Siegelmann and Eduardo D Sontag. On the computational power of neural nets. Journal of computer and system sciences, 50(1): 152-150, 1995.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3308-3318 Curran Associates, Inc., 2017.
10

---

Under review as a conference paper at ICLR 2020
Kenneth O. Stanley, Joel Lehman, and Lisa Soros. Open-endedness:: The last grand challenge wwwp:eiiiiicccooiaaaaae.e.ea
youve never heard of. O'Reilly Online, 2017. htt..atttee///w...rrilllyyycyo htt.:/acc...tttt..aaaa.aaroaaoaa
Ivor Tsang, James Kwok, and Pak-Ming Cheung. Core vector machines: Fast svm training on very large data sets. Journal of Machine Learning Research, 6:363-392, 04 2005.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv: 2019a.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation, 2019b.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. arXiv preprint arXiv:1810.05749, 2018.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. 2017. URL https://arxiv. wttp://wwww...//////....... 01578.

---

Under review as a conference paper at ICLR 2020
APPENDIX A ADDITIONAL EXPERIMENTAL DETAILS
The outer loop loss function is domain specific. In the supervised experiments on MNIST and CIFAR, the outer loop loss was cross-entropy for logistic regression on real MNIST CIFAR data. The inner-loop loss matches the outer-loop loss, but with synthetic data instead of real data. Appendix H describes the losses for the RL experiments.
The following equation defines the inner-loop SGD with momentum update for the learner parame- ters 0,.
41+1 = =0 a Σ gi
Ne).
(1)
where a and / are the learning rate and momentum hyperparameters, respectively. 2, is a batch of noise vectors that are input to the generator and are sampled from a unit-variance Gaussian. Y, are batch of labels for each generated sample/input and are sampled uniformly from all available class labels. Instead of randomly sampling 2, we can also learn curriculum by additionally optimizing z directly and keeping y, fixed throughout all of training. Results for both approaches (and additional curriculum ablations) are reported in Section 3.2.
A.1 MNIST EXPERIMENTS:
For the GTN training for MNIST we sampled architectures from distribution that produces ar- chitectures with convolutional (conv) and fully-connectd (FC) layers. All architectures had conv layers, but the number of filters for each layer was sampled uniformly from the ranges U ([3,, 128)) and U((4, 256)). respectively. After each conv layer there is max pooling layer for dimensionality reduction. After the last conv layer, there is fully-connected layer with number of filters sampled uniformly from the range U U(64, 256]). We used Kaiming Normal initialization (He al., 2015) and LeakyReLUs (Maas et al., 2013) (with a = 0.1). We use BatchNorm (Ioffe & Szegedy, 2015) for both the generator and the learners. The BatchNorm momentum for the learner was set imeta-training consistently converged to small values and we saw no significant gain from learning the value).
The generator consisted of 2 FC layers (1024 and 128 v H/4 * H/4 filters, respectively, where H is the final width of the synthetic image). After the last FC layer there are 2 conv layers. The first conv has 64 filters. The second conv has filter followed by a Tanh. We found it particularly important to normalize (mean of zero and variance of one) all datasets. Hyperparameters are shown in Table 2.
Hyperparameter
Value
Learning Rate
0.01
Initial LR
0.02
Initial Momentum
0.5
Adam Beta_1
0.9
Adam Beta 2
0.999
Size latent variable
128
Inner-Loop Batch Size
128
Outer-Loop Batch Size
128
Table 2: Hyperparameters for MNIST experiments
A.2 CIFARIO EXPERIMENTS:
For GTN training for CIFAR-10, the template architecture is a small learner with 5 convolutional layers followed by global average pooling and an FC layer. The second and fourth convolution had stride=2 for dimensionality reduction. The number of filters of the first conv layer was sam- pled uniformly from the range U[32, 128]) while all others were sampled uniformly from the range U ([64, 256)). Other details including the generator architecture were the same as the MNIST exper- iments, except the CIFAR generator's ss second cony layer had 3 filters instead of 1. Hyperparameters
12

---

Under review as a conference paper at ICLR 2020
used can be found in Table 3. For CIFARIO we augmented the real training when training GTNs with random crops and horizontal flips. We do not add weight normalization to the final architectures found during architecture search, but we do when train architectures with GTN -TNeeeenet data during architecture search to provide an estimate of their asymptotic performance.
Hyperparameter
Value
Learning Rate
0.002
Initial LR
0.02
Initial Momentum
0.5
Adam Beta_1
0.9
Adam Beta 2
0.9
Adam e
le-5
Size latent variable
128
Inner-loop Batch Size
128
Outer-loop Batch Size
256
Table Hyperparameters for CIFARIO 10 experiments
APPENDIX B REASONS GTNS ARE NOT EXPECTED TO PRODUCE SOTA ACCURACY VS. ASYMPTOTIC PERFORMANCE WHEN TRAINING ON REAL DATA
There three reasons not to expect SOTA accuracy levels for the learners trained synthetic data: (1) we train for very few SGD steps (32 or 128 vs. tens of thousands), (2) SOTA performance results from architectures explicitly designed (with much human effort) to achieve record accuracy, whereas GTN produces compressed training data optimized to generalize across diverse architectures with the aim of quickly evaluating new architecture's potential, and (3) SOTA methods often use data outside of the benchmark dataset and complex data-augmentation schemes.
APPENDIX C
CELL SEARCH SPACE
When searching for the operations in CNN cell, the 11 possible operations are listed below.
identity
1 X convolution
3 x 3 convolution
1 x 3 + 3 x convolution
1 x 7 7 x convolution
2 x 2 max pooling
3 x max pooling
5 x max pooling
2 x average pooling
3 X average pooling
5 average pooling
APPENDIX D COMPUTATION AND MEMORY COMPLEXITY
With the traditional training of DNNs with back proppppppggtiinn the memory requirements are pro- portional to the size of the network because activations during the forward propagation have to be stored for the backward propagation step. With meta-gradions meta-graddents, the memory requirement also grows with the number of inner-loop steps because all activations and weights have to be stored for the
13

---

Under review as a conference paper at ICLR 2020
2nd order gradient to be computed. This becomes impractical for large networks and/or many inner- loop steps. To reduce the memory requirements, we utilize maudient-checpoiniiee Walther, 2000) by only storing the computed weights of learner after each inner-loop step and computing the activations during the backward pass. This trick allows us to compute meta-gradients for networks with 10s of millions of parameters over hundreds of inner-loop steps in single GPU. While in theory the computational cost of computing meta-gradients with 4x larger than computing gradients (and 12x larger than forward propagation), in our experiments is about 2.5x slower than gradients through backpropagation due to parallelism. We could further reduce the memory requirements by utilizzig reversable hypergradients (Maclaurin et al., 2015), but, in our case, we were not constrained by the number of inner-loop steps we can store in memory.
(Griewank &
is
APPENDIX E EXTENDED NAS RESULTS
In the limited computation regime (less than day of computation), the best methods were, in order, GHN, ENAS, GTN, and NAONet with a mean error 2.84%, 2.89%, 2.92%, and 2.93%, respectively. A 0.08% difference on CIFARIO 10 represents 8 out of the 10k test samples. For that reason, we consider all of these methods as state of the art. Note that out of the four, GTN the only one relying on Random Search for architecture proposal.
Table 4: Performance of different architecture search methods. Search with our method required 16h total, of which were spent training the GTN and 8h were spent evaluating 800 architectures with GTN-produced synthetic data. Our results report mean + SD of evaluations of the same architec- ture with different initializations. is common to report scores with and without Cutout (DeVries & Taylor, 2017), data augmentation technique used during training. We found better architectures compared to other methods using random search (Random-We - and GHN-Top) and are competitive with algorithms that benefit from more advanced search methods (e.g. NAONet and ENAS employ non-random architecture proposals for performance gains; GTNs could be combined with such non- random proposals, which would likely further improve performance). Increasing the width of the
architecture found (F=128) further improves performance.
Model
Error %))
#params
Random
GPU Days
NASNet-A (Zoph Le, 2017)
3.41
3.3M
2000
AmoebaNet-B + Cutout (Real et al., 2019)
2.13
34.9M
X
3150
DARTS Cutout (Liu et al., 2018b)
2.83
4.6M
NAONet Cutout (Luo et al., 2018)
2.48
10.6M
X
200
NAONet- WS (Luo et al., 2018)
3.53
2.5M
X
0.3
NAONet-Wi WS Cutout (Luo et al., 2018)
2.93
2.5M
0.3
ENAS (Pham et al., 2018)
3.54
4.6M
X
0,45
ENAS Cutout (Pham et al., 2018)
2.89
4.6M
0.45
GHN Top-Best Cutout (Zhang et al., 2018)
2.84 + 0.07
5.7M
X
0.84
GHN Top (Zhang et al., 2018)
4.3 + 0.1
5.1M
0.42
Random-WS (Luo et al., 2018)
3.92
3.9M
0.25
Random Search Real Data (baseline)
3.88 0.08
12.4M
10
RS + Real Data Cutout (baseline)
3.02
12.4M
10
RS + Real Data + Cutout (F=128) (baseline)
2.11
151.7M
10
Random Search GTN (ours)
3.84
8.2M
0.67
Random Search GTN Cutout (ours)
2.92
8.2M
0.67
RS GTN Cutout (F=128) (ours)
2.42 + 0.03
97.9M
0.67
APPENDIX F CONDITIONED GENERATOR VS. XY-GENERATOR
Our experiments the main paper conditioned the generator to create data with given labels, by concatenating one-hot encoded label to the input vector. We also explored an alternative approach where the generator itself produced a target probability distribution to label the data generates. Because more information is encoded into soft label than one-hot encoded one, expected an improved training set to be generated by this variant. Indeed, such a "dark knowledge" dis- tillation setup has been shown to perform better than learning from labels (Hinton al., 2015).
14

---

Under review as a conference paper at ICLR 2020
However, the results in Figure 4 indicate that jointly generating both images and their soft labels under-per mnder-performs generating only images, although the result could change with different hyperpa- rameter values and/or innovations that improve the stability of training.
Real Data
0.975
GTN
0.950
0.925
Validation
0.900
0.875
0.850
0.825
0
500
1000
1500
2000
2500
3000
3500
Outer-loop Steps
Figure 4: Comparison between conditional generator and generator that outputs an image/label pair. We expected the latter "dark knowledge" approach to outperform the conditional generator, but that does not seem to be the case. Because initialization and training of the dark knowledge variant were more sensitive, we believe a more rigorous tuning of the process could lead to different result.
APPENDIX G GTN GENERATES (SEEMINGLY) ENDLESS DATA
While optimizing images directly (i.e. optimizing a fixed tensor of images) would result a fixed number of samples, optimizing generator can potentially result in unlimited amount of new samples. We tested this generative capability by generating more data during evaluation (i.e. with no change to the meta-optimization procedure) in two ways. In the first experiment, we increase the amount of data in each imner-lopp optimization step by increasing the batch size (which results in lower variance gradients). fixed, but increase the number of inner-loop optimization steps for which new network is trained. Both cases result in an increased amount of training data. If the GTN generator has overfit to the number of inner-loop optimization steps during meta-training and/or the batch size, then we would not expect performance to improve when we have the generator produce more data. However, an alternate hypothesis is that the GTN is producing a healthy distribution of training data, irrespective of exactly how is being used. Such a hypothesis would be supported by performance increase in these experiments.
the second experiment, we keep the number of samples per batch
Figure 5a shows performance as a function of increasing batch size (beyond the batch size used during meta-optimization,, i.e. 128). The increase in performance of GTN means that larger training sets from our generator (with diminishing returns) and that we are not limited by the choice of batch size during training (which constrained due both memory and computation requirements).
sample
Figure 5b shows the results of generating more data by increasing the number of inner-loop opti- mization steps. Generalization to more inner-loop optimization steps is important when the number of inner-loop optimization steps used during meta-optimizationn is not enough achieve maximum performance. This experiment also tests the generalization of the optimizer hyperparameters be- cause they were optimized maximize learner performance after fixed number of steps. There is an increase in performance of the learner trained on GTN-generated data as the number of inner- loop optimization steps is increased, demonstrating that the GTN is producing generally useful data instead of overfitting to the number of inner-loop optimization steps during training (Figure 5b). Extending the conclusion from Figure 2b, in the very low data regime, GTN is significantly better than training real data (p 0.05). However, more inner-loop optimization steps taken and thus more unique data is available to the learner, training on the real data becomes more effective than learning from synthetic data (p < 0.05) (see Figure 5b).
15

---

Under review as a conference paper at ICLR 2020
Beal Date GT%
0.60
Swiddeton Acc.20
3.955
3.66
2.946
Via //lidaooon Acc.
3.955
2.000
3.64
0.00
Real Data
0.0
153
200
203
200
352
400
430
500
120
Inner loop
Innertoop
Increasing inner-loop batch size
(b) Increasing inner-loop optimization steps
Figure 5: (a) The left figure shows that even though GTN was meta-trained to generate synthetic data of batch size 128, sampling increasingly larger batches results improved learner performance (the inner-loop optimization steps are fixed to 16). (b) The right figure shows that increasing the number of inner-loop optimization steps (beyond the steps used during meta-training) improves learner performance. The performance gain with real data is larger in this setting. This improvement shows that GTNs do not overfit to a specific number of inner-loop optimization steps.
9
Figure 6: GTN samples w/o curriculum.
Another interesting test for our generative model is to test the distribution of learners after they have trained on the synthetic data. We want to know, for instance, if training synthetic samples from one GTN results in a functionally similar set of learner weights regardless of learner initialization (this phenomena can be called learner mode collapse). Learner mode collapse would prevent the performance gains that can be achieved through ensembling diverse learners. We tested for learner mode collapse by evaluating the performance (on held-out data and held-out architecture) of an en- semble of 32 randomly initialized learners that are trained on independent batches from the same GTN. To construct the ensemble, we average the predicted probability distributions across the learn- ers to compute combined prediction and accuracy. The results of this experiment can be seen in Figure 7, which shows that the combined performance an ensemble is better (on average) than an individual learner, providing additional evidence that the distribution of synthetic data is healthy and allows ensembles to be harnessed to improve performance, as is standard with networks trained on real data.
APPENDIX H GTN FOR RL
To demonstrate the potential of GTNs for RL, we tested our approach with a small experiment on the classic CartPole test problem (see Brockman et al. (2016) for details on the domain. We conducted this experiment before the discovery that weight normalization improves GTN training, so these experiments do not feature it; it might further improve performance. For this experiment, the meta-objective the GTN is trained with is the advantage actor-critic formulation: log π (a)e,) ) Vss; 9.)) (Mnih al., 2016). The state-value V is provided by separate neural network trained to estimate the average state-value for the learners produced so far during meta-training. The learners train synthetic data via single-step of SGD with batch size of 512 and mean squared error
16

---

Under review as a conference paper at ICLR 2020
1.00
0.98
Validation Accuracy
0.94
0.92
Real Data single
0.88
GTN single
0.26
GTN ensemble
1000 1500
2000
2500
3000
3500
Outer-loop Steps
Figure 7: Performance of ensemble of GTN learners vs, individual GTN learners. Ensembling set of neural networks that each had different weight initializations, but were trained on data from the same GTN substantially improves performance. This result provides more evidence that GTNs generate a healthy distribution of training data and are not somehow forcing the learners to all learn a functionally equivalent solution.
regression loss, meaning the inner loop is supervised learning. The outer-loop is reinforced because the simulator non-differentiable. We could have also used an RL algorithm in the inner loop. In that scenario the GTN would have to learn to produce an entire synthetic world an RL agent would learn in. Thus, would create the initial state and then iteratively receive actions and generate the next state and optionally reward. For example, GTN could learn to produce an entire MDP that an agent trains on, with the meta-objective being that the trained agent then performs well on a target task. We consider such synthetic (PO)MDPs an exciting direction for future research.
The score on CartPole the number of frames out of 200 for which the pole elevated. Both GTN and an A2C (Mnih et al., 2016) control effectively solve the problem (Figure 8). Interestingly, training GTNs takes the same number of simulator steps as training a single learner with policy- gradients (Figure 8). Incredibly, however, once trained, the synthetic data from GTN be used to train a learner to maximum performance in a single SGD step! While that is unlikely to be true for harder target RL tasks, these results suggest that the speed-up for architecture search from using GTNs in the RL domain can be even greater than in supervised domain.
The CartPole experiments feature single-layer neural network with 64 hidden units and tanh activation function for both the policy and the value network. The inner-loop batch size was 512 and the number of inner-loop training iterations was 1. The observation space this environment consists of a real-valued vector of size 4 (Cart position, Cart velocity, Pole position, Pole velocity). The action space consists of discrete actions (move left or move right). The outer loop loss the reward function for the target domain (here, pole-balancing). The inner loop loss is mean squared error (i.e. the network is doing supervised learning on the state-action mapping pairs provided by the GTN).
APPENDIX SOLVING MODE COLLAPSE IN GANS WITH GTNS
We created an implementation of generative adversarial networks (GANs) (Goodfellow al., 2014) and found they tend to generate the same class of images (e.g. only 1s, Figure 9), which is a common training pathology in GANs known as mode collapse (Srivastava al., 2017). While there are tech- niques to prevent mode collapse (e.g. minibatch discrimination and historical averaging (Salimans al., 2016)). we hypothesized that combining the ideas behind GTNs and GANs might provide a different, additional technique to help combat mode collapse. The idea is to add a discriminator to the GTN forcing the data it generates to both be realistic and help a learner perform well on the meta-objective of classifying MNIST. The reason this approach should help prevent mode collapse that if the generator only produces one class of images, a learner trained on that data will not be able to classify all classes of images. This algorithm (GTN-GAN) was able to produce realistic images with no identifiable mode collapse (Figure 10). GTNs offer a different type of solution to the issue of mode collapse than the many that have been proposed, adding new tool toolbox
17

---

Under review as a conference paper at ICLR 2020
200
A2C Agent
175
A2C GTN Agent
150
Reward
125
100
40000
80000 100000
Environment Steps
Figure 8: An A2C Agent control trains a single policy throughout all of training, while the GTN method starts with new, randomly initialized network at each ileration and produces the plotted performance after a single step of SGD. This plot is difficult to parse because of that difference: compares the accumulated performance of A2C across all environment steps up to that point vs. the performance achieved with GTN data in single step of SGD from a single batch of synthetic data. Thus, at the 100,000th step of training. GTNs enable training a newly initialized network to the given performance (of around 190) 100,000 times faster with GTN synthetic data than with A2C from scratch. With GTNs, we can therefore train many new, high- high-performing agents quickly. That would be useful in many ways, such as greatly accelerating architecture search algorithms for RL. Of course, these results are on a simple problem, and (unlike our supervised learning experiments) have not yet shown that the GTN data works with different architectures, but these results demonstrate the intriguing potential GTNs for RI. One reason we might expect even larger speedups for RI. supervised learning is because major reason RL is sample inefficient is because it requires exploration to figure out how to solve the problem. However, once that exploration has been done, the GTN can produce data to efficiently teach that solution a new architecture. RL thus represents an exciting area of future research for GTNs. Performing that research is beyond the scope of this paper, but we highlight the intriguing potential here to inspire such future work.
for solving that problem. Note we not claim this approach is better than other techniques to prevent mode collapse, only that it is interesting new type of option, perhaps one that could be productively combined with other techniques.
6
0
0
3
3
8
5
0
4
4
86
6
4
//1///1
4
8
9
7
0
0
7 8 607
Figure 9: Images generated by basic GAN on MNIST before and after mode collapse. The left image shows GAN Croduuudd images early in GAN training and the right image shows GAN samples later in training after mode collapse has occurred due to training instabilities.
APPENDIX ADDITIONAL MOTIVATION
There is an additional motivation for GTNs that involves long-term, ambitious research goals: GTN is step towards algorithms that generate their own training environments, such that agents trained in them eventually solve tasks we otherwise do not know how to train agents to solve (Clune, 2019). It is important to pursue such algorithms because capacity conceive of effective training vironments on our own as humans is limited, yet for our learning algorithms to achieve their full potential they will ultimately need to consume and complex curricula of learning challenges
18

---

Under review as a conference paper at ICLR 2020
2
0
6
6
D
2
5
8
6
6
0
7
9
9
8
0
4
2
4
7
6
3 4 2 2
6
0
7
7
9
9
0
0
5
7
1
7
9
0
9
2
[
7
8
1
3
1
5
4
3
0
5
5
3
4
8
/
3
0
2
4
5
7
7
9
7
4
0
9
8
7
17446 5
19
3 9
8
6
Figure 10: Images generated by GTN with an auxiliary GAN loss. Combining GTNs with GANs produces far more realistic images than GTNs alone (which produced alien, unrecognizable images, Figure 6). The combination also stabilizes GAN training, preventing mode collapse.
and data. Algorithms for generating curricula, such the the paired open-ended trailblazer (POET) algorithm (Wang et al., 2019a), have proven effective for achieving behaviors that would otherwise be out of reach, but algorithm yet can generate completely unconstrained training conditions. For example, POET searches for training environments within a highly restricted preconceived space of problems. GTNs are exciting because they can encode rich of possible environments with min- imal assumptions, ranging from labeled data for supervised learning to (in theory) entire complex virtual RI. domains (with their own learned internal physics). Because RNNs are Turing-complete (Siegelmann & Sontag, 1995), GTNs should be able to theoretically encode all possible learning environments. Of course, while what is theoretically possible is different from what is achievable in practice, GTNs give
expressive environmental encoding to begin exploring what potential unlocked when we can learn to generate sophisticated learning environments. The initial results presented here show that GTNs can be trained end-to-end with gradient descent through the entire learning process; such end-to-end learning has proven highly scalable before, and may similarly in the future enable learning expressive GTNs that encode complex learning environments.
19


Reviews:
[{"review_id": "S1x3_iukqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm for generating training data to help other machine learning agents learn faster. The proposed Generative Teaching Networks (GTNs) are networks that are trained to generate training data for other networks and are trained jointly with these other networks by back propagating through the entire learning problems via meta-gradients. They also show how weight normalization can help stabilize the training of GTNs. The paper is well-written overall and easy to follow, except for a few typos that can be fixed for the camera ready. The main idea of the paper is quite simple and it’s nice to see it works well. I’m actually surprised it has not been proposed before, but I am also not very familiar with this research area. For these reasons, I lean towards accepting this paper, although I have a few comments that I would like to see addressed for the camera ready version.\n\nThe authors present experiments where they apply GTNs on image classification and neural architecture search. GTN does indeed seem to do better than the baselines for these problems. However, for MNIST and CIFAR it looks like the models being used may not be that good, as it’s quite easy to obtain better performance than the results shown in the paper. I would be curious to know why the authors did not use a better architecture and also what the architecture they used actually is. I am unfortunately not familiar with neural architecture search to be able to evaluate their experimental setup in that case.\n\nRegarding the curriculum used in Section 3.2, my understanding is that the full curriculum approach has an additional computational cost vs the no curriculum approach, as you have to learn that curriculum. Thus, even though the result the authors present is interesting and verifies that GTNs learn a useful curriculum, I would also like to see curves of how accuracy improves per computational unit (e.g., the horizontal axis could be CPU training time). This would allow us to see whether learning a curriculum this way is in fact practically useful. It may just as well be that it is too expensive and training without it is faster.\n\nThe authors show example images generated by GTNs and, as they also mention, these images do not look very realistic. It would be good to have some explanation/analysis around this. Could it be that these are images that are “hard” for the classifier? (e.g., thinking in terms of support vector machines, do these images lie in or close to the margin of the classifier?). I would love to seem an analysis around this and a couple of proposed explanations.\n\nI would also like to see more details on the actual architecture used for the experiments as I feel that the paper does not provide enough information to reproduce the results.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJxKcbi-5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a meta-learning algorithm Generative Teaching Networks (GTN) to generate fake training data for models to learn more accurate models. In the inner loop, a generator produces training data and the learner takes gradient steps on this data. In the outer loop, the parameters of the generator are updated by evaluating the learner on real data and differentiating through the gradient steps of the inner loop. The main claim is this method is shown to give improvements in performance on supervised learning for MNIST and CIFAR10. They also suggest weight normalization patches up instability issues with meta-learning and evaluate this in the supervised learning setting, and curriculum learning for GTNs.\n\nTo me, the main claim is very surprising and counter-intuitive - it is not clear where the extra juice is coming from, as the algorithm does not assume any extra information. The actual results I believe do not bear out this claim because the actual results on MNIST and CIFAR10 are significantly below state of the art. On MNIST, GTN achieves about 98% accuracy and the baseline “Real Data” achieves  <97% accuracy, while the state of the art is about 99.7% and well-tuned convnets without any pre-processing or fancy extras achieve about 99% according to Yann LeCunn’s website. The disparity on CIFAR seems to be less egregious but the state of the art stands at 99% while the best GTN model (without cutout) achieves about 96.2% which matches good convnets and is slightly worse than neural architecture search according to https://paperswithcode.com/sota/image-classification-on-cifar-10.\n\nThis does not negate the potential of GTNs which I feel are an interesting approach, but I believe the paper should be more straightforward with the presentation of these results. The current results basically  show that GTNs improve the performance of learners with bad hyper-parameters. On problems that are not as well-studied as MNIST or CIFAR10 this could still be very valuable (as we do not know what performance is good or bad in advance). Based on the results, GTN does seem to be a significant step forward in synthetic data generation for learning compared to prior work (Zhang 2018, Luo 2018).\n\nThe paper proposes two other contributions: using weight normalization for meta-learning and curriculum learning for GTNs. Weight normalization is shown to stabilize GTNs on MNIST. I think the paper oversteps in the relevant method section, hypothesizing it may stabilize meta-learning more broadly. The paper should present a wider set of experiments to make this claim convincing. But the point for GTNs on MNIST nevertheless stands. For curriculum learning: the description of the method is done across section 2 and section 3.2 and does not really describe it completely. How exactly are the samples chosen in GTN - All Shuffled? How does GTN - Full Curriculum and Shuffled Batch parametrize the order of the samples so that it can be learned? I suggest that this information is all included as a subsection in the method (section 2). The results seem to show the learned curriculum is superior to no curriculum.\n\nAt a high level it would be very surprising to me if the way forward for better discriminative models was to learn good generative models and use them again for training discriminative models, simply because discriminative models have proved thus far significantly easier to train. If this work does eventually show this result, it would be a very interesting result. At the moment, I believe it does not, but I would be happy to change my mind if the authors provide convincing evidence. Alternatively, I feel that the paper could be a valuable contribution to the community if the writing is toned down to focus on the contributions, presents the results comparing to well-tuned hyperparameters and not over-claim.\n\nMore comments:\n\nWhat is the outer loop loss function? Is it assumed to be the same as the inner one (but using real data instead of training data)? I think this should be made explicit in the method section.\n\nThere are some additional experiments in other settings such as RL and unsupervised learning. Both seem like quite interesting directions but seem like preliminary experiments that don’t work convincingly yet. The RL experiment shows that using GTN does not change performance much. There is a claim about optimizing randomly initialized networks at each step, but the baseline which uses randomly initialized networks at each step with A2C is missing. The GAN experiments shows the GAN loss makes GTN realistic (as expected) but there are no quantitative results on mode collapse. (Another interesting experiment would be to show how adding a GAN loss for generating data affects the test performance of the method.) Perhaps it would benefit the paper to narrow in on supervised learning? Given that these final experiments are not polished, the claim in the abstract that the method is “a general approach that is applicable to supervised, unsupervised, and reinforcement learning” seems to be over-claiming. I understand it can be applicable but the paper has not really done the work to show this outside the supervised learning setting.\n\nMinor comments:\nPg. 4: comperable -> comparable", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlacioyqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes Generative Teaching Networks, which aims to generate synthetic training data\nfor a given prediction problem. The authors demonstrate its use in an MNIST prediction task\nand a neural architecture search task on Cifar10.\nI do not find the idea compelling nor the empirical idea convincing enough to warrant acceptance at\nICLR.\n\n\nDetailed Comments:\n \nAt a high level, the motivation for data generation in order to improve a given prediction problem \nis not clear. From a statistical perspective, one can only do so well given a certain amount of\ntraining data, and being able to generate new data would suggest that one can do arbitrarily better\nby simply creating more data -- this is not true. \n\nWhile data augmentation techniques have improved accuracy in many cases, they have also relied\nheavily on domain knowledge about the problem, such as mirroring, cropping for images. The proposed\nGTN model does not seem to incorporate such priors and I would be surprised that one can do better\nwith such synthetically generated data. \nIndeed, the proposed approach does not do better than the best performing models on MNIST.\n \nThe authors use GTNs in a NAS problem where they use the accuracy on the generated images as a proxy\nfor the validation accuracy. As figure 4c illustrates there actually does not seem to be much\ncorrelation between the accuracies on the synthetic and real datasets. \nWhile Table 1 indicates that they outperform some baselines, I do not find them compelling. This\ncould simply be because random search is a coarse optimization method (and hence the proposed metric\nmay not do well on more sophisticated search techniques). \n    - On a side note, why is evaluating on the synthetic images cheaper than evaluating on the\n      original images? \n    - What is the rank-correlation metric used? Did you try more standard correlation metrics such\n      as Pearson's coefficient? \n\n\n=================\nPost rebuttal\n\nHaving read the rebuttal, the comments from other reviewers, and the updated manuscript, I am more positive about the paper now. I agree that with reviewer 2 that the proposed approach is interesting and could be a method to speed up NAS in new domains. I have upgraded my score to reflect this.\n\nMy only remaining issue is that the authors should have demonstrated this on new datasets (by running other methods on these datasets) instead of sticking to the same old datasets. However, this is the standard practice in the NAS literature today.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overview:
This paper introduces a method to distill a large dataset into a smaller one that allows for faster training. The main application of this technique being studied is neural architecture search, which can be sped up by quickly evaluating architectures on the generated data rather than slowly evaluating them on the original data.

Summary of discussion:
During the discussion period, the authors appear to have updated the paper quite a bit, leading to the reviewers feeling more positive about it now than in the beginning. In particular, in the beginning, it appears to have been unclear that the distillation is merely used as a speedup trick, not to generate additional information out of thin air. The reviewers' scores left the paper below the decision boundary, but closely enough so that I read it myself. 

My own judgement:
I like the idea, which I find very novel. However, I have to push back on the authors' claims about their good performance in NAS. This has several reasons:

1. In contrast to what is claimed by the authors, the comparison to graph hypernetworks (Zhang et al) is not fair, since the authors used a different protocol: Zhang et al sampled 800 networks and reported the performance (mean +/- std) of the 10 judged to be best by the hypernetwork. In contrast, the authors of the current paper sampled 1000 networks and reported the performance of the single one judged to be best. They repeated this procedure 5 times to get mean +/- std. The best architecture of 1000 is of course more likely to be strong than the average of the top 10 of 800.

2. The comparison to random search with weight sharing (here: 3.92% error) does not appear fair. The cited paper in Table 1 is *not* the paper introducing random search + weight sharing, but the neural architecture optimization paper. The original one reported an error of 2.85% +/- 0.08% with 4.3M params. That paper also has the full source code available, so the authors could have performed a true apples-to-apples comparison. 

3. The authors' method requires an additional (one-time) cost for actually creating the 'fake' training data, so their runtimes should be increased by the 8h required for that.

4. The fact that the authors achieve 2.42% error doesn't mean much; that result is just based on scaling the network up to 100M params. (The network obtained by random search also achieves 2.51%.)

As it stands, I cannot judge whether the authors' approach yields strong performance for NAS. In order to allow that conclusion, the authors would have to compare to another method based on the same underlying code base and experimental protocol. Also, the authors do not make code available at this time. Their method has a lot of bells and whistles, and I do not expect that I could reproduce it. They promise code, but it is unclear what this would include: the generated training data, code for training the networks, code for the meta-approach, etc? This would have been much easier to judge had the authors made the code available in anonymized fashion during the review.

Because of these reasons, in terms of making progress on NAS, the paper does not quite clear the bar for me. The authors also evaluated their method in several other scenarios, including reinforcement learning. These results appear to be very promising, but largely preliminary due to lack of time in the rebuttal phase.  

Recommendation:
The paper is very novel and the results appear very promising, but they are also somewhat preliminary. The reviewers' scores leave the paper just below the acceptance threshold and my own borderline judgement is not positive enough to overrule this. I believe that some more time, and one more iteration of reorganization and review, would allow this paper to ripen into a very strong paper. For a resubmission to the next venue, I would recommend to either perform an apples-to-apples comparison for NAS or reorganize and just use NAS as one of several equally-weighted possible applications. In the current form, I believe the paper is not using its full potential.
Decision: Reject