Synthetic Petri Dish: A Novel Surrogate Model for Rapid Architecture Search

1 INTRODUCTION . The architecture of deep neural networks ( NNs ) is critical to their performance . This fact motivates neural architecture search ( NAS ) , wherein the choice of architecture is often framed as an automated search for effective motifs , i.e . the design of a repeating recurrent cell or activation function that is repeated often in a larger NN blueprint . However , evaluating a candidate architecture ’ s ground-truth performance in a task of interest depends upon training the architecture to convergence . Complicating efficient search , the performance of an architectural motif nearly always benefits from increased computation ( i.e . larger NNs trained with more data ) . The implication is that the best architectures often require training near the bounds of what computational resources are available , rendering naive NAS ( i.e . where each candidate architecture is trained to convergence ) exorbitantly expensive . To reduce the cost of NAS , methods often exploit heuristic surrogates of true performance . For example , motif performance can be evaluated after a few epochs of training or with scaled-down architectural blueprints , which is often still expensive ( because maintaining reasonable fidelity between ground-truth and surrogate performance precludes aggressive scaling-down of training ) . Another approach learns models of the search space ( e.g . Gaussian processes models used within Bayesian optimization ) , which improve as more ground-truth models are trained , but can not generalize well beyond the examples seen . This paper explores whether the computational efficiency of NAS can be improved by creating a new kind of surrogate , one that can benefit from miniaturized training and still generalize beyond the observed distribution of ground-truth evaluations . To do so , we take inspiration from an idea in biology , bringing to machine learning the application of a Synthetic Petri Dish microcosm that aims to identify high-performing architectural motifs . The overall motivation behind “ in vitro ” ( test-tube ) experiments in biology is to investigate in a simpler and controlled environment the key factors that explain a phenomenon of interest in a messier and more complex system . For example , to understand causes of atypical mental development , scientists extract individual neuronal cells taken from brains of those demonstrating typical and atypical behavior and study them in a Petri dish ( Adhya et al. , 2018 ) . The approach proposed in this paper attempts to algorithmically recreate this kind of scientific process for the purpose of finding better neural network motifs . The main insight is that biological Petri dish experiments often leverage both ( 1 ) key aspects of a system ’ s dynamics ( e.g . the behavior of a single cell taken from a larger organism ) and ( 2 ) a human-designed intervention ( e.g . a measure of a test imposed on the test-tube ) . In an analogy to NAS , ( 1 ) the dynamics of learning through backpropagation are likely important to understanding the potential of a new architectural motif , and ( 2 ) compact synthetic datasets can illuminate an architecture ’ s response to learning . That is , we can use machine learning to learn data such that training an architectural motif on the learned data results in performance indicative of the motif ’ s ground-truth performance . In the proposed approach , motifs are extracted from their ground-truth evaluation setting ( i.e . from large-scale NNs trained on the full dataset of the underlying domain of interest , e.g . MNIST ) , instantiated into very small networks ( called motif-networks ) , and evaluated on learned synthetic data samples . These synthetic data samples are trained such that the performance ordering of motifs in this Petri dish setting ( i.e . a miniaturized network trained on a few synthetic data samples ) matches their ground-truth performance ordering . Because the relative performance of motifs is sufficient to distinguish good motifs from bad ones , the Petri dish evaluations of motifs can be a surrogate for ground-truth evaluations in NAS . Training the Synthetic Petri Dish is also computationally inexpensive , requiring only a few ground-truth evaluations , and once trained it enables extremely rapid evaluations of new motifs . A key motivating hypothesis is that because the Synthetic Petri Dish evaluates the motif by actually using it in a simple experiment ( e.g . training it with SGD and then evaluating it ) , its predictions can generalize better than other neural network ( NN ) based models that predict motif performance based on only observing the motif ’ s structure and resulting performance ( Liu et al. , 2018a ; Luo et al. , 2018 ) . For example , consider the demonstration problem of predicting the ground-truth performance of a two-layer feedforward MNIST network with sigmoidal non-linearity . The blue points in Figure 1 shows how the ground-truth performance of the MNIST network varies when the slope of its sigmoid activations ( the term c in the sigmoid formula 1/ ( 1 + e−cx ) ) is varied in the range of 0.01− 2.01 . The MNIST network performance peaks near a slope-value of 0.23 . Similarly to the NN-based model previously developed in Liu et al . ( 2018a ) ; Luo et al . ( 2018 ) , one can try to train a neural network that predicts the performance of the corresponding MNIST network given the sigmoid slope value as input ( Section 4.1 provides full details ) . When training points ( tuples of sigmoid slope value and its corresponding MNIST network performance ) are restricted to an area to the right of the peak ( Figure 1 , blue-shaded region ) , the NN-based prediction model ( Figure 1 , red diamonds ) generalizes poorly to the test points on the left side of the peak ( c < 0.23 ) . However , unlike such a conventional prediction model , the prediction of the Synthetic Petri Dish generalizes to test points left of the peak ( despite their behavior being drastically different than what would be expected solely based on the points in the blue shaded region ) . That occurs because the Synthetic Petri Dish trains and evaluates the actual candidate motifs , rather than just making predictions about their performance based on data from past trials . Beyond this explanatory experiment , the promise of the Synthetic Petri Dish is further demonstrated on a challenging and compute-intensive language modelling task that serves as a popular NAS benchmark . The main result is that Petri dish obtains highly competitive results even in a limitedcompute setting . Interestingly , these results suggest that it is indeed possible to extract a motif from a larger setting and create a controlled setting ( through learning synthetic data ) where the instrumental factor in the performance of the motif can be isolated and tested quickly , just as scientists use Petri dishes to test specific hypothesis to isolate and understand causal factors in biological systems . 2 RELATED WORK . NAS methods have discovered novel architectures that significantly outperform hand-designed solutions ( Zoph and Le , 2017 ; Elsken et al. , 2018 ; Real et al. , 2017 ) . These methods commonly explore the architecture search space with either evolutionary algorithms ( Suganuma et al. , 2017 ; Miikkulainen et al. , 2018 ; Real et al. , 2019 ; Elsken et al. , 2019 ) or reinforcement learning ( Baker et al. , 2016 ; Zoph and Le , 2017 ) . Because running NAS with full ground-truth evaluations can be extremely expensive ( i.e . requiring many thousands of GPU hours ) , more efficient methods have been proposed . For example , instead of evaluating new architectures with full-scale training , heuristic evaluation can leverage training with reduced data ( e.g . sub-sampled from the domain of interest ) or for fewer epochs ( Baker et al. , 2017 ; Klein et al. , 2017 ) . More recent NAS methods such as DARTS ( Liu et al. , 2018b ) and ENAS ( Pham et al. , 2018 ) exploit sharing weights across architectures during training to circumvent full ground-truth evaluations . However , a significant drawback of such weight sharing approaches is that they constrain the architecture search space and therefore limit the discovery of novel architectures . Another approach to accelerate NAS is to train a NN-based performance prediction model that estimates architecture performance based on its structure ( Liu et al. , 2018a ) . Building on this idea , Neural Architecture Optimization ( NAO ) trains a LSTM model to simultaneously predict architecture performance as well as to learn an embedding of architectures . Search is then performed by taking gradient ascent steps in the embedding space to generate better architectures . NAO is used as a baseline for comparison in Experiment 4.2 . Bayesian optimization ( BO ) based NAS methods have also shown promising results ( Kandasamy et al. , 2018 ; Cao et al. , 2019 ) . BO models the architecture space using a Gaussian process ( GP ) , although its behavior is sensitive to the choice of a kernel function that models the similarity between any two architectures . Another recent NAS method presents a technique to progressively grow an architecture by adding skip connections , and is named similarly ( “ Petridish ” ) to the method proposed here ( Hu et al. , 2019 ) . However unlike the Synthetic Petri Dish introduced here , which is a learned surrogate for NAS , Petridish ( Hu et al. , 2019 ) is instead an incremental growth method . Generative teaching networks ( GTNs ) also learn synthetic data to accelerate NAS ( Such et al. , 2020 ) . However , learned data in GTNs helps to more quickly train full-scale networks to evaluate their potential on real validation data . In the Petri dish , synthetic training and validation instead enables a surrogate microcosm training environment for much smaller extracted motif-networks . Additionally , GTNs are not explicitly trained to differentiate between different networks ( or network motifs ) . In contrast , the Synthetic Petri Dish is optimized to find synthetic input data on which the performance of various architectural motifs is different . 3 METHODS . Recall that the aim of the Synthetic Petri Dish is to create a microcosm training environment such that the performance of a small-scale motif trained within it well-predicts performance of the fullyexpanded motif in the ground-truth evaluation . First , a few initial ground-truth evaluations of motifs are needed to create training data for the Petri dish . In particular , consider N motifs for which ground-truth validation loss values ( Litrue , where i ∈ 1 , 2 , ... N ) have already been pre-computed by training each motif in the ground-truth setting . The next section details how these initial evaluations are leveraged to train the Synthetic Petri Dish . 3.1 TRAINING THE SYNTHETIC PETRI DISH . To train the Synthetic Petri Dish first requires extracting the N motifs from their ground-truth setting and instantiating each of them in miniature as separate motif-networks . For the experiments performed in this paper , the ground-truth network and the motif-network have the same overall blueprint and differ only in the width of their layers . For example , Figure 2a shows a ground-truth network ’ s size reduced from a 2-layer , 100-neuron wide MLP to a motif-network that is a 2-layer MLP with a single neuron per layer . Given such a collection of extracted motif-networks , a small number of synthetic training and validation data samples are then learned that can respectively be used to train and evaluate the motifnetworks . The learning objective is that the validation loss of motifs trained in the Petri dish resemble the validation loss of the motif ’ s ground-truth evaluation ( Litrue ) . Note that this training process requires two nested optimization loops : an inner-loop that trains and evaluates the motif-networks on the synthetic data and an outer-loop that trains the synthetic data itself . Initializing the Synthetic Petri Dish : Before training the Petri dish , the motif-networks and synthetic data must be initialized . Once the motifs have been extracted into separate motif-networks , each motif-network is assigned the same initial random weights ( θinit ) . This constraint reduces confound- ing factors by ensuring that the motif-networks differ from each other only in their instantiated motifs . At the start of Synthetic Petri Dish training , synthetic training data ( Strain = ( xtrain , ytrain ) ) and validation data samples ( Svalid = ( xvalid , yvalid ) ) are randomly initialized . Note that these learned training and validation data can play distinct and complementary roles , e.g . the validation data can learn to test out-of-distribution generalization from a learned training set . Empirically , setting the training and validation data to be the same initially ( i.e . Strain = Svalid ) benefited optimization at the beginning of outer-loop training ; over iterations of outer-loop training , the synthetic training and validation data then diverge . The size of the motif-network and the number of synthetic data samples are chosen through the hyperparameter selection procedure described in Appendix A.2 . Inner-loop training : The inner optimization loop is where the performance of motif-networks is evaluated by training each such network independently with synthetic data . This training reveals a sense of the quality of the motifs themselves . In each inner-loop , the motif-networks are independently trained with SGD using the synthetic training data ( Strain ) . The motif-networks take synthetic training inputs ( xtrain ) and produce their respective output predictions ( ŷtrain ) . For each motif-network , a binary cross-entropy ( BCE ) loss is computed between the output predictions ( ŷtrain ) and the synthetic training labels ( ytrain ) . Because the Petri dish is an artificial setting , the choice of BCE as the inner-loop loss ( Linner ) is independent of the actual domain loss ( used for ground-truth training ) , and other losses like regression loss could instead be used . The gradients of the BCE loss w.r.t . the motif-network weights inform weight updates ( as in regular SGD ) . θit+1 = θ i t − α∇Liinner_train ( Strain , θit ) i ∈ 1 , 2 , .. , N ( 1 ) where α is the inner-loop learning rate and θi0 = θinit . Inner-loop training proceeds until individual BCE losses converge . Once trained , each motif-network is independently evaluated using the synthetic validation data ( Svalid ) to obtain individual validation loss values ( Liinner_valid ) . These inner-loop validation losses then enable calculating an outer-loop loss to optimize the synthetic data , which is described next . Outer-loop training : Recall that an initial sampling of candidate motifs evaluated in the ground-truth setting serve as a training signal for crafting the Petri dish ’ s synthetic data . That is , in the outer loop , synthetic training data is optimized to encourage motif-networks trained upon it to become accurate surrogates for the performance of full networks built with that motif evaluated in the ground-truth setting . The idea is that training motif-networks on the right ( small ) set of synthetic training data can potentially isolate the key properties of candidate motifs that makes them effective . To frame the outer-loop loss function , what is desired is for the validation loss of the motif-network to induce the same relative ordering as the validation loss of the ground-truth networks ; such relative ordering is all that is needed to decide which new motif is likely to be best . One way to design such an outer-loop loss with this property is to penalize differences between normalized loss values in the Petri dish and ground-truth setting1 . To this end , the motif-network ( inner-loop ) loss values and their respective ground-truth loss values are first independently normalized to have zero-mean and unit-variance . Then , for each motif , a mean squared error ( MSE ) loss is computed between the normalized inner-loop validation loss ( L̂iinner_valid ) and the normalized ground-truth validation loss ( L̂itrue ) . The MSE loss is averaged over all the motifs and used to compute a gradient step to improve the synthetic training and validation data . Louter = 1 N N∑ i=1 ( L̂iinner_valid − L̂itrue ) 2 ( 2 ) Straint+1 = S train t − β∇Louter and Svalidt+1 = Svalidt − β∇Louter ( 3 ) where β is the outer-loop learning rate . For simplicity , only the synthetic training ( xtrain ) and validation ( xvalid ) inputs are learned and the corresponding labels ( ytrain , yvalid ) are kept fixed to 1We tried an explicit rank-loss as well , but the normalized regression loss performed slightly better empirically . their initial random values throughout training . Minimizing the outer-loop MSE loss ( Louter ) modifies the synthetic training and validation inputs to maximize the similarity between the motif-networks ’ performance ordering and motifs ’ ground-truth ordering . After each outer-loop training step , the motif-networks are reset to their original initial weights ( θinit ) and the inner-loop training and evaluation procedure ( equation 1 ) is carried out again . The outer-loop training proceeds until the MSE loss converges , resulting in optimized synthetic data .

Summary: This paper presents an approach to accelerating NAS with 'petri-dish' networks, which hope to mimic the response of original networks at a fraction of training time cost. The key idea is to evaluate an architectural setting on a miniaturized network as opposed to the original network. With this approach computational effort is saved by eschewing expensive 'ground truth' original network evaluations.