Synthetic Petri Dish: A Novel Surrogate Model for Rapid Architecture Search
Authors: Aditya Rawal
Joel Lehman
Felipe Petroski Such
Jeff Clune
Kenneth Stanley
Venue:  (2021)
Keywords: Neural Architecture Search
AutoML
Meta-learning

Abstract: Neural Architecture Search (NAS) explores a large space of architectural motifs --  
a compute-intensive process that often involves ground-truth evaluation of each motif by instantiating it within a large network, and training and evaluating the network with thousands or more data samples. Inspired by how biological motifs such as cells are sometimes extracted from their natural environment and studied in an artificial Petri dish setting, this paper proposes the Synthetic Petri Dish model for evaluating architectural motifs. In the Synthetic Petri Dish, architectural motifs are instantiated in very small networks and evaluated using very few learned synthetic data samples (to effectively approximate performance in the full problem). The relative performance of motifs in the Synthetic Petri Dish can substitute for their ground-truth performance, thus accelerating the most expensive step of NAS. Unlike other neural network-based prediction models that parse the structure of the motif to estimate its performance, the Synthetic Petri Dish predicts motif performance by training the actual motif in an artificial setting, thus deriving predictions from its true intrinsic properties. Experiments in this paper demonstrate that the Synthetic Petri Dish can therefore predict the performance of new motifs with significantly higher accuracy, especially when insufficient ground truth data is available.
Our hope is that this work can inspire a new research direction in studying the performance of extracted components of models in a synthetic diagnostic setting optimized to provide informative evaluations.

Under review as a conference paper at ICLR 2021
SYNTHETIC PETRI DISH: A NOVEL SURROGATE MODEL FOR RAPID ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Neural Architecture Search (NAS) explores large space of architectural motifs a compute-intensive process that often involves ground-truth evaluation of each motif by instantiating within a large network, and training and evaluating the network with thousands or more data samples. Inspired by how biological motifs such as cells are sometimes extracted from their natural environment and studied in artificial Petri dish setting, this paper proposes the Synthetic Petri Dish model for evaluating architectural motifs. In the Synthetic Petri Dish, architectural motifs are instantiated in very small networks and evaluated using very few learned synthetic data samples (to effectively approximate performance in the full problem). The relative performance of motifs in the Synthetic Petri Dish substitute for their ground-truth performance, thus accelerating the most expensive step of NAS. Unlike other neural network-based prediction models that parse the structure of the motif to estimate its performance, the Synthetic Petri Dish predicts motif performance by training the actual motif in an artificial setting, thus deriving predictions from its true intrinsic properties. Experiments in this paper demonstrate that the Synthetic Petri Dish can therefore predict the performance of new motifs with significantly higher accuracy, especially when insufficient ground truth data is available. Our hope that this work can inspire new research direction in studying the performance of extracted components of models in synthetic diagnostic setting optimized to provide informative evaluations.
INTRODUCTION
The architecture of deep neural networks (NNs) is critical to their performance. This fact motivates neural architecture search (NAS), wherein the choice of architecture is often framed as an automated search for effective motifs, i.e. the design of a repeating recurrent cell or activation function that is repeated often in larger NN blueprint. However, evaluating a candidate architecture's ground-truth performance in a task of interest depends upon training the architecture to convergence. Complicating efficient search, the performance of an architectural motif nearly always benefits from increased computation (i.e. larger NNs trained with more data). The implication is that the best architectures often require training near the bounds of what computational resources are available, rendering naive NAS (i.e. where each candidate architecture is trained to convergence) exorbitantly expensive.
To reduce the cost of NAS, methods often exploit heuristic surrogates of true performance. For example, motif performance can be evaluated after few epochs of training or with scaled-down architectural blueprints, which is often still expensive (because maintaining reasonable fidelity between ground-truth and surrogate performance precludes aggressive scaling-down of training). Another approach learns models of the search space (e.g. Gaussian processes models used within Bayesian optimization), which improve as more ground-truth models are trained, but cannot generalize well beyond the examples seen. This paper explores whether the computational efficiency of NAS can be improved by creating a new kind surrogate, one that can benefit from miniaturized training and still generalize beyond the observed distribution of ground-truth evaluations. To do so, take inspiration from an idea in biology, bringing to machine learning the application of a Synthetic Petri Dish microcosm that aims to identify gigh-performin architectural motifs.
The overall motivation behind "in vitro" (test-tube) experiments in biology is to investigate a simpler and controlled environment the key factors that explain phenomenon of interest in messier

---

Under review as a conference paper at ICLR 2021
and more complex system. For example, to understand causes of atypical mental development, scientists extract individual neuronal cells taken from brains of those demonstrating typical and atypical behavior and study them in Petri dish (Adhya et al. (018)) The approach proposed in this paper attempts to algorithmically recreate this kind of scientific process for the purpose of finding better neural network motifs. The main insight is that biological Petri dish experiments often leverage both (1) key aspects of a system's is dynamics (c.g. the behavior of single cell taken from a larger organism) and (2) a human-designed intervention (e.g. a measure of a test imposed on the test-tube). In an analogy to NAS, (1) the dynamics of learning through backpropagation are likely important to understanding the potential of new architectural motif, and (2) compact synthetic datasets can illuminate an architecture's 's response to learning. That is, we can use machine learning to learn data such that training an architectural motif on the learned data results in performance indicative of the motif"s s ground-truth performance.
In the proposed approach, motifs are extracted from their ground-truth evaluation setting (i.e. from large-scale NNs trained on the full dataset of the underlying domain of interest, e.g. MNIST), instantiated into very small networks (called motif-networks), and evaluated on learned synthetic data samples. These synthetic data samples trained such that the performance ordering of motifs in this Petri dish setting (i.e. a miniaturized network trained on a few synthetic data samples) matches their ground-truth performance ordering. Because the relative performance of motifs is sufficient to distinguish good motifs from bad ones, the Petri dish evaluations of motifs can be a surrogate for ground-truth evaluations in NAS. Training the Synthetic Petri Dish is also computationally inexpensive, requiring only a few ground-truth evaluations, and once trained it enables extremely rapid evaluations of new motif'.
Ground truth performance of layer. 100 wide MNIST network Synihwtic Petri Disin Model Predicted Penformance Neural Network Model Predicted Performance
Normalized Accuracy
Ground-truth values used for fraining performance prediction models restricted to this range
Peak Performance
slope 0.23
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Sigmoid Slope Value
Figure 1: Predicting the Optimal Slope of the Sigmoid Activation. Each blue diamond depicts the normalized validation accuracy (ground-truthh of 2-layer, 100-wide feed-forward network with unique sigmoid slope value (mean of 20 runs). The validation accuracy peaks at slope of 0.23. Both the Synthetic Petri Dish and a neural network surrogate model that predicts performance as function of sigmoid slope are trained on limited set of ground- -rutt points, restricted to the blue-shaded region to the right of the peak. The normalized performance predictions for Synthetic Petri Dish are shown with green diamonds and those for the NN surrogate model are shown as red diamonds. The plot shows that the NN model predictions overfits the training data. contrast, because the Synthetic Petri Dish conducts experiments with small neural networks with these sigmoid slope values it is more more accurate at inferring both that there a peak and its approximate location.
A key motivating hypothesis is that because the Synthetic Petri Dish evaluates the motif by actually using it in a simple experiment (e.g. training it with SGD and then evaluating it), its predictions can generalize better than other neural network (NN) based models that predict motif performance based on only observing the motif's s structure and resulting performance (Liu et al.) 2018a| Luo al.) 2018). For example, consider the demonstration problem of predicting the ground-truth performance of two-layer feedforward MNIST network with sigmoidal non-linearity. The blue points in Figure [ shows how the ground-truth performance of the MNIST network varies when the slope of its sigmoid activations (the term e in the sigmoid formula 1/(1 + e cx)) is varied in the range of 0.01 2.01. The MNIST network performance peaks near slope-value of 0.23. Similarly to the NN-based model previously developed in [iu et al. (2018a); Luo et al. (2018), one can try to train a neural network that predicts the performance of the corresponding MNIST network given the sigmoid slope value
2

---

Under review as a conference paper at ICLR 2021
as input (Section 4.1 provides full details). When training points (tuples of sigmoid slope value and its corresponding MNIST network performance) are restricted to an area to the right of the peak (Figure [] blue-shaded region), the NN-based prediction model (Figure || red diamonds) generalizes poorly to the test points on the left side of the peak (c < 0.23). However, unlike such a conventional prediction model, the prediction of the Synthetic Petri Dish generalizes test points left of the peak (despite their behavior being drastically different than what would be expected solely based on the points in the blue shaded region). That occurs because the Synthetic Petri Dish trains and evaluates the actual candidate motifs, rather than just making predictions about their performance based on data from past trials.
Beyond this explanatory experiment, the promise of the Synthetic Petri Dish is further demonstrated on challenging and compute-intensive language modelling task that serves as a popular NAS benchmark. The main result is that Petri dish obtains highly competitive results even in a limited- compute setting. Interestingly, these results suggest that it indeed possible to extract a motif from a larger setting and create controlled setting (through learning synthetic data) where the instrumental factor in the performance the motif can be isolated and tested quickly, just as scientists use Petri dishes test specific hypothesis to isolate and understand causal factors in biological systems.
2 RELATED WORK
NAS methods have discovered novel architectures that significantly outperform hand-designed solutions (Zoph and Le| 2017 Elsken et al. 2018 Real et al., (017). These methods commonly
explore the architecture search space with either evolutionary algorithms (Suganuma et al., 201 7
Miikkulainen et al. 2018 Real et al. 2019 Elsken et al. 2019) reinforcement learning (Baker 2017). Because running NAS with full ground-truth evaluations can be extremely expensive (i.c. requiring many thousands of GPU hours), more efficient methods have been proposed. For example, instead of evaluating new architectures with full-scale training, heuristic evaluation can leverage training with reduced data (e.g. sub-sampled from the domain of interest) or for fewer epochs (Baker et al.) 2017 Klein et al. 2017).
et Zoph and Le
More recent NAS methods such DARTS (Liu et al. 2018b) and ENAS (Pham et al. 2018 exploit sharing weights across architectures during training to circumvent full ground-truth evaluations. However, a significant drawback of such weight sharing approaches is that they constrain the architecture search space and therefore limit the discovery of novel architectures.
Another approach accelerate NAS is train NN-hased performance prediction model that estimates architecture performance based on its structure (Liu et al. (2018)). Building on this idea, Neural Architecture Optimization (NAO) trains LSTM model to simultaneously predict architecture performance as well as to learn an embedding of architectures. Search is then performed by taking gradient ascent steps in the embedding space to generate better architectures. NAO is used a baseline for comparison in Experiment
Bayesian optimization (BO) based NAS methods have also shown promising results (Kandasamy et al. 2018 Cao et al. 2019). BO models the architecture space using a Gaussian process (GP), although its behavior is sensitive to the choice of kernel function that models the similarity between any two architectures. Another recent NAS method presents technique to progressively grow an architecture by adding skip connections, and is named similarly ("Petridish")) to the method proposed here (Hu et al. 2019). However unlike the Synthetic Petri Dish introduced here, which is learned surrogate for NAS, Petridish (Hu et al. (219)) is instead an incremental growth method.
Generative teaching networks (GTNs) also learn synthetic data to accelerate NAS (Such et al. 2020). However, learned data in GTNs helps to more quickly train full-scale networks to evaluate their potential on real validation data. In the Petri dish, synthetic training and validation instead enables surrogate microcosm training environment for much smaller extracted motif-networks. Additionally, GTNS are not explicitly trained to differentiate between different networks (or network motifs). In contrast, the Synthetic Petri Dish is optimized to find synthetic input data on which the performance of various architectural motifs is different.
3

---

Under review as a conference paper at ICLR 2021
MNIST
Velidation
Synthetic Petri Dish Inforence
Ground-truth training and evallation
(2x) Trained Petri Dish
(2b) New
(3) Top Predicted
Validation
Valizaion
Volidazion
Lowa
Lown
Extracting
motif:
Average Mean Square Error between normalized losses normalized ground
Synthetic Diah
NAS method for
Reducing
2-4nywr, 100
wide MLP
to Z-ayer,
Vaidation Loas
Valdetion Loan
Voldaaton
single-
1
(fa) Moeif ground-truth
(1b) Mottf ground-truth performance
wide MLP
training similarity ground-truth performance
Inner-loop Training and Validation of mottf networks
performance
Ground-truth
Synthotic Training and Validation Data (Data Size 40 10)
Modif sot
Synthetic Petri Dish Training Procedure
Combining Architecture Search with the Petri Dish
Figure 2: (a) Synthetic Petri Dish Training. The left figure illustrates the inner-loop and outer-loop training procedure. The motifs (in this example, activation functions) are extracted from the full network (e.g 2-layer, 100 wide MLP) and instantiated in separate, much smaller motif-networks (e.g. two-layer, single-neuron MLP). The motif-networks are trained in the inner-loop with the synthetic training data and evaluated using synthetic validation data. In the outer-loop, an average mean squared error loss is computed between the normalized Petri dish validation losses and the corresponding normalized ground-truth losses. Synthetic training and validation data are optimized by taking gradient steps w.r.t the outer-loop loss. (b) Combining Architecture Search with the Petri Dish. Functions are depicted inside rectangles and function outputs are depicted as arrows with their descriptions adjacent them.
3 METHODS
Recall that the aim of the Synthetic Petri Dish is to create microcosm training environment such that the performance a small-scale motif trained within it well-predicts performance of the fully- expanded motif in the ground-truth evaluation. First, few initial ground-truth evaluations of motifs are needed to create training data for the Petri dish. In particular, consider N motifss for which ground-truth validation loss values (Cirue where i 1, 2,2, ...NN have already been pre-computed by training each motif in the ground-trute setting. The next section details how these initial evaluations leveraged train the Synthetic Petri Dish.
3.1 TRAINING THE SYNTHETIC PETRI DISH
To train the Synthetic Petri Dish first requires extracting the N motifs from their ground-truth setting and instantiating each of them in miniature as separate motif-networks. For the experiments performed in this paper, the ground-truth network and the motif-network have the same overall blueprint and differ only in the width of their layers. For example, Figure 2 shows a ground-trui network's size reduced from a layer, 100-neuron wide MLP to motif netif- that is 2-layer MLP with single neuron per layer.
Given such collection of extracted motif-networks a small number of synthetic training and validation data samples are then learned that can respectively used to train and evaluate the motif- networks. The learning objective is that the validation loss of motifs trained in the Petri dish resemble the validation loss of the motif's ground-trun -tuth evaluation (Cirue). Note that this training process requires two nested optimization loops: inner-loop that trains and evaluates the motif-networks the synthetic data and an outer-loop that trains the synthetic data itself.
Initializing the Synthetic Petri Dish: Before training the Petri dish, the motif- networks and syn- thetic data must be initialized. Once the motifs have been extracted into separate motif-networkk networks, each motif-network is assigned the same initial random weights (0.nt). This constraint reduces confound-
4

---

Under review as a conference paper at ICLR 2021
ing factors by ensuring that the motif-networrks differ from each other only in their instantiated motifs. At the start of Synthetic Petri Dish training. synthetic training data (Strain (x'rain, " grrae))) 3) and validation data samples (Swalid = (x )) are randomly initialized. Note that these learned training and validation data can play distinct and complementary roles, e.g. the validation data can learn to test training and validation data to be the same initially (i.e. Straán = Swalid) benefited optimization at the beginning of outer-loop training: over iterations of outer-loop training, the synthetic training and validation data then diverge. The size of the motif- network and the number of synthetic data samples are chosen through the hyperparameter selection procedure described in Appendix [A.2]
generalization from learned training set. Empirically, setting the
Inner-loop training: The inner optimization loop is where the performance of motif-networts is evaluated by training each such network independently with synthetic data. This training reveals a sense of the quality of the motifs themselves.
In each inner-loop, the motif-networks are independently trained with SGD using the synthetic training data (Strann). The motif-networks take synthetic training inputs (x"roin) and produce their respective output predictions (j*rain). For each motif-network, a binary cross-entropy (BCE) loss is computed between the output predictions (jffrann) and the synthetic training labels (y)rain). Because the Petri dish is an artificial setting, the choice of BCE as the inner-loop loss (Cinner) is independent of the actual domain loss (used for ground truth training), and other losses like regression loss could instead be used. The gradients of the BCE loss w.r.t. the motif-network weights inform weight updates (as in regular SGD).
8/_1 = 8
(Strain . i 1,
where a is the inner-loop learning rate and 0j Oinit Inner-loop training proceeds until individual BCE losses converge. Once trained, each motif-network is independently evaluated using the synthetic validation data (Swalid) to obtain individual validation loss values (Linner potier___aaa...a These inner-loop validation losses then enable calculating an outer-loop loss to optimize the synthetic data, which is described next.
Outer-loop training: Recall that an initial sampling of candidate motifs evaluated in the ground- truth setting serve training signal for crafting the Petri dish's synthetic data. That is, in the outer loop, synthetic training data is optimized to encourage motif-networks trained upon it to become accurate surrogates for the performance of full networks built with that motif evaluated in the ground-truth setting. The idea is that training motif-networks on the right (small) set of synthetic training data can potentially isolate the key properties of candidate motif's that makes them effective.
To frame the outer-loop loss function, what is desired is for the validation loss of the motif-netword induce the same relative ordering as the validation loss of the ground networks; such relative ordering is all that is needed to decide which new motif is likely to be best. One way to design such an outer-loop loss with this property is to penalize differences between normalized loss values in the Petri dish and ground-truth setting]] To this end, the motif-network (inner-loop) loss values and their respective ground- truth loss values are first independently normalized to have zero-mean and unit- variancee Then, for each motif, mean squared error (MSE) loss is computed between the normalized inner-loop validation loss (Cimmer__vae vakaa) and the normalized ground-truth validation loss (Cirue). The MSE loss is averaged over all the motifs and used to compute a gradient step to improve the synthetic training and validation data.
Louter
1 N
> (Ciimor_yvaidd Liruc)?
(2)
strain = Strain Lonter and gvalid = spalid BV Conter
(3)
where 3 is the outer-loop learning rate. For simplicity, only the synthetic training (atrain) and validation (alid) inputs are learned and the corresponding labels rrinn y yoalid) are kept fixed to
We tried explicit ross- as well, but the normalized regression loss performed slightly better empiri- cally.
5

---

Under review as a conference paper at ICLR 2021
their initial random values throughout training. Minimizing the outer-loop MSE loss (Couter) modifies the synthetic training and validation imputs to maximize the similarity between the motif-netword performance ordering and motifs' ground-trute ordering.
After each outer-loop training step, the notif-n are reset to their original initial weights (0ima) and the inner-loop training and evaluation procedure (equation I is carried out again. The outer-loop training proceeds until the MSE loss converges, resulting in optimized synthetic data.
3.2 PREDICTING PERFORMANCE WITH THE TRAINED PETRI DISH
The Synthetic Petri Dish training procedure deseribed so far results in synthetic training and validation data optimized to sort notif-networks similarly to the ground-trute setting. This section describes how the trained Petri dish can predict the relative performance of unseen motifs, which we call the Synthetic Petri Dish inference procedure. In this procedure, new motifs are instantiated in their individual motif-networks, and the motif-networks are trained and evaluated using the optimized synthetic data (with the same hyperparameter settings as in the inner-loop training and evaluation). The relative inner-loop validation loss for the motif-networks then serves surrogate for the motifs' relative ground-trutt truth validation loss; as stated earlier, such relative loss values are sufficient to compare the potential of new candidate motifs. Such Petri dish inference is computationally inexpensive because it involves the training and evaluation of very small notif-networks with very few synthetic examples. Accurately predicting the performance ordering of unseen motifs contingent on the generalization capabilities of the Synthetic Petri Dish (this aspect is further investigated in section
3.3 COMBINING ARCHITECTURE SEARCH WITH THE SYNTHETIC PETRI DISH
Interestingly, the theap-to-evaluue surrogate performance prediction given by the trained Petri dish is complementary to most NAS methods that search for motifs, meaning that they can easily be combined. Algorithm ii Appendix A. shows one possible hybridization of Petri dish and NAS, which is the one we experimentally investigate in this work.
First, the Petri dish model is warm-started by training (inner-loop and outer-loop) using the ground- truth evaluation data (Pevat) of small set of -eenerttde motifs (X Then in each iteration of NAS, the NAS method generates M new motifs and the Petri dish inference procedure inexpensively predicts their relative performance. The top K motifs (where K << M) with highest predicted performance are then selected for ground-truth evaluation. The ground-truth performance of motifs both guides the NAS method as well as provides further data to re-train the Petri dish model. The steps outlined above are repeated until convergence and then the motif with the best ground-truth performance selected for the final test evaluation.
Synthetic Petri Dish training and inference is orders of magnitude faster than ground-truth evaluations, thus making NAS computationally more efficient and faster to run, which can enable finding higher- performing architectures given limited compute budget.
4 EXPERIMENTS
4.1 SEARCHING FOR THE OPTIMAL SLOPE FOR SIGMOIDAL ACTIVATION FUNCTIONS
Preliminary experiments demonstrated that when a 2-layer, 100-wide feed-forward network with sigmoidal activation functions is trained on MNIST data, its validation accuracy (holding all else constant) depends on the slope of the sigmoid. The points on the blue curve in Figure Saarredddmmosstrie this fact, where the empirical peak performance is a slope of 0.23. This simple dependence provides a way to clearly illustrate the benefits of the Synthetic Petri Dish model.
Both the Synthetic Petri Dish and the NN-based surrogate model are trained using 30 ground-truth points that are randomly selected from a restricted interval of sigmoid slope values (the shuees region in Figure D]. The remaining ground -ruth points (outside the blue-shaded region) are used only for testing. The NN-based surrogate control a 2-layer, 1--neuron-wide feedforward network that takes the sigmoid value as input and predicts the corresponding MNIST network validation accuracy as its output. A mean-squared error loss is computed between the predicted accuracy and the ground-truth validation accuracy, and the network is trained with the Adam optimizer.

---

Under review as a conference paper at ICLR 2021
For training the Synthetie Petri Dish model, each training motif (i.e. sigmoid with unique slope value) is extracted from the MNIST network and instantiated in a 2-layer, single-neuron-ve widde motif- network (0miat) The setup is similar to the one shown in Figure 2b). The motif-networks are trained in the inner-loop for 200 SGD steps and subsequently their performance on the synthetic validation valid) is used to compute outer-loop MSE loss w.r.t the ground-truth performance (as described in section 3.1). A total of 20 outer-loop training steps are performed. Hyperparameter selection details for the two models are described in Appendix A.
data (Linner_ _
Results demonstrate that the NN-based model overfits the training points. (red-curve in Figure I In contrast, the Synthetie Petri Dish predictions accurately infer that there is peak (including the falloff to its left) and also its approximate location (green-curve in Figure
NAO with Reduced Data Synthetic Petri Dish with RS Synthetic Petri Dish with NAO RS with Partial Evaluation
62.2
Test Perplexity
4.612
60.5
59.8
59.0
50.5
58.7
57.1
60 Ground-truth evaluations
50
100
Figure 3: RNN Cell Search for PTB: best found cell for four NAS methods across variable numbers of NAS iterations, All the methods are warmed up at the beginning (Step 0 in Figure 2b) with 40 ground-truth evaluations notice the top-left point with best test perplexity of 63. For Synthetic Petri Dish with RS (blue curve) and Synthetic Petri Dish with NAO (green curve), the top 20 motifs with the best predicted performance are selected for ground-truth evaluations in each NAS iteration. Original NAO (Luo et al. 2018) (not shown here) requires 1,000 ground-truth evaluations to achieve a test perplexity of 56.0. NAO with Reduced Data (red-curve) shows the results obtained by rumning original NAO, but with fewer ground-truth evaluations (the same number the Synthetic Petri Dish variants get). With such limited data, Synthetic Petri Dish with NAO outperforms other NAS methods and achieves test perplexity of 57.1 after 100 ground-trutt truth evaluations.
This graph plots the test perplexity (mean of five runs) of the
4.2 ARCHITECTURE SEARCH FOR RECURRENT CELLS
The previous experiment demonstrated that Synthetic Petri Dish model trained with limited ground- truth data can successfully generalize to unseen out-of-distributioon motifs. This next experiment tests whether the Synthetic Petri Dish can be applied to more realistic and challenging setting, that of NAS for a NN language model that is applied to the Penn Tree Bank (PTB) dataset- a popular language modeling and NAS benchmark (Marcus al. 1993). In this experiment, the architectural motif is the design of a recurrent cell. The recurrent cell search space and its ground-truth evaluation setup is the same as NAO (Luo et al., 2018). This NAS problem is challenging because the search space is expansive and few solutions perform well (Pham et al. 2018). Each cell in the search space composed of 12 layers (each with the same layer width) that are connected in unique pattern. An input embedding layer and an output soft-max layer is added to the cell (each of layer width 850) to obtain a full network (27 Million parameters) for ground tround-truth evaluation. Each ground- evaluation requires training on PTB data-set for 600 epochs and takes 10 hours on Nvidia 1080T..
NAO is one of the sfate-of-the-an NAS methods for this problem and is therefore used as a baseline in this experiment (called here original NAO). In the published results (Luo et al. 2018), original NAO requires 1,000 ground-truth evaluations (300 GPU days) over three successive NAS iterations to discover cell with test perplexity of 56.0. These are good results, but the compute cost even to reproduce them is prohibitively large for many researchers. Because the Synthetic Petri Dish offers a potential low-compute option, in this experiment, different NAS methods are compared instead in setting where only limited ground-trum truth evaluation data is available (< 100 samples), giving a sense of how far different methods can get with more reasonable compute resources.
7

---

Under review as a conference paper at ICLR 2021
Each NAS iteration can be accelerated if the number of costly ground-trute -outh evaluations is reduced by instead cheaply evaluating the majority of candidate motifs (i.e. new cells) in the Petri dish. For the purpose of training the Synthetic Petri Dish, each cell is extracted from its ground-truth setting (850 neurons per layer) and is instantiated in a motif-network with three neurons per layer (its internal cell connectivity, including its depth, remains unchanged). Thus, the ground-truth network that has 27 million parameters is reduced to a motif-network with only 140 parameters. To train motif-networkk synthetic training and validation data each of size 20 x 10 x 10 (batch size x time steps mootif-network imput size) is learned (thus replacing the 923k training and 73k validation words of PTB). The Petri dish training and inference procedure is very similar to the one described in Experiment |..1 and CPU).
adds negligible compute cost (2 extra hours for training, and a few minutes for inference
Following the steps outlined in algorithm T and Figure 2h the Petri dish surrogate can be combined with two existing NAS methods: (1) Random Search (RS) or (2) NAO itself, resulting in two new methods called Synthetic Petri Dish-RS and Synthetic Petri Dish-NAO. Also, the Random Search NAS method can be combined with partial evaluations resulting in another baseline (Appendix A.4)
For the Synthetic Petri Dish variants, at the beginning of search, both the Petri dish surrogate and the NAS method (RS/NAO) used within the Petri dish variant are warm-started with the ground- truth data of an initial motif set (size 40). In each NAS iteration, 100 newly generated motifs (variable M in algorithm| D are evaluated using the Petri dish inference procedure and only the top 20 predicted motifs (variable K in algorithm|] [T) are evaluated for their ground-truth performance. The test perplexity of the best found motif at the end of each NAS iteration is plotted in Figure - the blue curve depicts the result for Synthetic Petri Dish-RS and green depicts the result for Synthetic Petri Dish-NAO, For a fair comparison, original NAO re-run in this limited ground-truth setting and the resulting performance is depicted by the red-curve in Figure 3 The results show that Synthetic Petri Dish-NAO outperforms both Synthetic Petri Dish-RS and NAO when keeping the amount of ground-trus truth data points the same, suggesting that the Synthetic Petri Dish and NAO complement each other well. The hybridization of Synthetic Petri Dish and NAO finds a cell that competitive in its performance (test perplexity 57.1) with original NAO 156.0), using only 1 1/10th of original NAO's compute (and exceeds the performance of original NAO when both are given equivalent compute).
5 DISCUSSION AND CONCLUSIONS
In the general practice of science, often the question arises of what factor accounts for an observed phenomenon. In the real world, with all its intricacy and complexity, it can be difficult to test or even formulate a clear hypothesis on the relevant factor involved. For that reason, often a hypothesis is formulated and tested in a simplified environment where the relevant factor can be isolated from the confounding complexity of the world around it. Then, in that simplified setting becomes possible to run rapid and exhaustive tests, as long as there is expectation that their outcome might correlate to the real world. In this way, the Synthetic Petri Dish is a kind of microcosm of a facet of the scientific method, and its synthetic data is the treatment whose optimization tethers the dynamics within the simplified setting to their relevant counterparts in the real world.
By approaching architecture search in this way as a kind of question-answeriin problem how certain motif's or factors impact final results, we gain the intriguing advantage that the prediction model is longer a black box. Instead, it actually contains within it a critical piece of the larger world that it seeks to predict. This piece, a motif cut from the ground-truie network (and its corresponding learning dynamics), carries with it from the start a set of priors that no black box learned model could carry on its own. These priors pop out dramatically in the simple sigmoid slope experiment - the notion that there is an optimal slope for training and roughly where it lies emerges automatically from the fact that the sigmoid slope itself is part of the Petri Dish prediction model. In the later NAS for recurrent cells, the benefit in a more complex domain also becomes apparent, where the advantage of the intrinsic prior enables the Petri Dish to have better performance than a leading NAS method when holding the number of ground-truth evaluations constant, and achieves roughly the same performance with th the compute when allowing differing numbers of ground-truth evaluations.
is also possible that other methods can be built in the future on the idea of extracting a component of a candidate architecture and testing it in another setting. The opportunity to tease out the underlying causal factors of performance is a novel research direction that may ultimately teach us new lessons on architecture by exposing the most important dimensions of variation through principled empirical process that could capture the spirit and power of the scientific process itself.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
D. Adhya, E. Annuario, M. A. Lancaster, J. Price, S. Baron-Cohen, and D. P. Srivastava. Understand- ing the role of stervids in typical and atypical brain development: Advantages of using a "brain in dish" approach. Journal of Neuroendocrrnology, 30(2) 2018. doi: 10.1111/jne.12444 12547
URL
com www.wwwwwo......a/aa//oaoaa
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European Conference on Computer Vision (ECCV), pages 19-34, 2018a.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimization. In Advances in neural information processing systems, pages 7816-7827, 2018.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. 2017. URL attps://arxiv: org/ haagg//bs/iin /1611. 01578
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, CoRR, 0tss/1808.0377,.00 2018.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In ICML, 2017.
Masanori Suganuma, Shinichi Shirakawa, and Tomoharu Nagao. A genetic programming approach to designing convolutional neural network architectures. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 17, pages 497-504, New York, NY, USA, 2017. doi: 10.1 00.1145/0771178.3072222 URL http:://oi..... acm 3071229
ACM. ISBN
org/10.
Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. Evolving deep neural networks. In Robert Kozma, Cesare Alippi, Yoonsuck Choe, and Francesco Carlo Morabito, editors, Artificial Intelligence in the Age of Neural Networks and Brain Computing. Amsterdam: Elsevier, 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4780 4789- 2019.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. In International Conference on Learning Representations, 2019.
B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neural network architectures using reinforce- ment learning. 2016. URL settps://arxiv.oo///ddae
.ddf
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. arXiv preprint arXiv: 2017.
Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. In ICLR, 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.0900 09055, 2018b.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4095-4104, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisation and optimal transport. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa- Bssa-Bianhhi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 2016-2025. Curran Associates, Inc., 2018.

---

Under review as a conference paper at ICLR 2021
Shengcao Cao, Xiaofang Wang. and Kris M. Kitani. Learaale embedding space for efficient neural architecture compression. In International Conference on Learning Representations, 2019. URL
http://////eenneeeee........ net/ http://aaaeeeeee.eeoooo.aoaaaaaa [orummid---11XX33333
Hanzhang Hu, John Langford, Rich Caruana, Saurajit Mukherjee, Bric Horvitz, and Debadeepta Dey.
Efficient forward architecture search. Neural Information Processing Systems, 2019.
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth Stanley, and Jeff Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data, 2020.
M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. Building large annotated corpus of english: The penn treebank. Computational linguistics, 19(2), 1993.
P. Langley, Crafting papers on machine learning. In Pat Langley, editor, Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pages 1207-1216, Stanford, CA. 2000. Morgan Kaufmann.
Prajit Ramachandran, Barret Zoph, and Quoc Le. Searching for activation functions. 2018. URL https: http:///wxwiv.oogg/o/////// 05941, pddf
10


Reviews:
[{"review_id": "oP_Pb8Pcp1r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\n \nThe paper provides a novel surrogate model method for Neural Architecture Search. Authors motivated the paper from biological study of cells in an artificial petri dish setting. Authors explains the concept of evaluation of small motifs with synthetic learned training and validation data that can predict the performance of larger network.\nLeveraging this, authors proposes speeding up the problem of Neural Architecture Search.\n\n##########################################################################\n\nPros:\n \n1. The problem of Neural Architecture Search is an impactful problem specially with the recent advances in neural networks. This paper tries to solve the scalability issue of this problem which I think has real world impact.\n\n2. The paper is generally well motivated. The motivation behind the methods are well explained. I liked the petri dish inspiration. Also I think the idea of motif for architecture search deserves more attention. I like the use of motifs to speed/scale up the search procedure such that we can train on smaller set of motifs but do inference on a larger set. Also as the motifs are directly evaluated rather than using some NN to evaluate them, it makes the case for their method more convincing. \n \n3. Overall the paper is well written.  Result section is also well structured. \n\n \n##########################################################################\n\nCons: \n\n1. Although the paper puts a lot of importance on motifs, but it does not explain a standard way to generate the motifs for any kind of networks, which makes the scope a bit narrow. Is there a way to standardize the motif generation procedure for different architectures?\n\n2. The performance improvement with time over other methods is not properly shown in the result section. I thought that would have shown more impact in the result. Although it shows perplexity with ground truth evaluations curve, but time vs performance could have probably shown the effect of cheap inference for motifs in these work.\n \n3. Result for only one task is shown in the result section. It's not clear if this method can be for other tasks? Adding more tasks might have strengthened the paper.\n\n4. Authors mentions that original NAO reaches 56 perplexity after 300 GPU days. How many days does it take for the synthetic petri dish with NAO to reach this result? What is its best performance? When does it converges if we keep running for a long time? What is the performance for RS without NAO if we keep running for a long time?\n\n##########################################################################\n\nQuestions during rebuttal period: \n\n \nPlease address and clarify the cons above \n\n \n#########################################################################\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "no4FtOCeCPW", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "tHe paper proposes a method for quickly and cheaply determining the value of a\nparticular motif in neural architecture search by isolating it in a \"petri dish\"\nthat allows it to be evaluated without having to train an entire network. The\nauthors describe their method and evaluate it empirically, showing the promise\nof the method.\n\nThe idea is interesting and seems very promising. As the authors say, it\naddresses one of the bottlenecks in neural architecture search -- the presented\nresearch deals with an important problem. There are clear reasons for preferring\nthe proposed method over alternatives.\n\nHowever, there are important details missing. In particular, it was unclear to\nme after reading the paper how the synthetic data are generated and how exactly\nthe motifs are instantiated. These are crucial parts of the proposed method, and\nits success hinges on representative data and embeddings. This should be\nexplained in some detail; in particular because the paper leaves the reader at a\nloss on how to apply this methodology to their own problems.\n\nFurther, the evaluation performed by the authors, albeit showing good results,\nis very small. Only anecdotal results are presented in two contexts, and it is\nunclear how the approach actually performs -- how good are the predictions the\npetri dish makes, and how does the derived ranking compare to the ground-truth\nranking?\n\nIn summary, while I feel that the idea is interesting and promising, there is\ninsufficient information for it to have a significant impact.", "extra_scores": {}, "extra_text": {}}, {"review_id": "T5xJy2r8-s7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper considers the problem of Neural Architectural Search, and proposes an efficient method called Synthetic Petri Dish inspired by how in-vitro experiments are done in biology. Numerical simulations are shown using two examples to demonstrate the validity of the proposed method.\n\nPros:\n- The paper introduces a nice idea to speed up the problem of architecture/hyper-parameter search. The current search methods are often too prohibitive computationally, so this is a useful contribution.\n- The idea is well-motivated, and also quite general in terms of its applicability. As in, this method could be used to speed up a variety of architecture/hyper-parameter searches.\n\nCons:\n- The numerical evidence provided in the paper is quite minimal. It would be very interesting to see the performance of the method across different datasets as well as across different hyper-parameters. For instance, how does the method perform when it comes to choice of layers in a CNN (like different kinds of residual blocks) or transformers, learning rates, number of filters, etc. While the evidence provided in the paper is alright, there is lots of room to strengthen the paper.\n\nComments:\n- While I understand why Figure 1 is introduced very early in the paper, I feel it should be explained better. In particular, the use of normalised validation accuracy without defining it properly could lead to confusion. Is it possible to just use standard metrics like accuracy instead?\n- Equation (2): I do not understand why one has to define ground-truth loss and motif network loss. Because the two are subtracted, wouldn't the result be just the difference between the ground truth prediction and motif network prediction? In other words, (a-b)-(c-b) = (a-c). So essentially you are trying to minimise the difference between the predictions of the two models right? Is this also why you only optimise on the synthetic input and not bother about the synthetic output?", "extra_scores": {}, "extra_text": {}}, {"review_id": "PmghpBeirh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to accelerating NAS with 'petri-dish' networks, which hope to mimic the response of original networks at a fraction of training time cost. The key idea is to evaluate an architectural setting on a miniaturized network as opposed to the original network. With this approach computational effort is saved by eschewing expensive 'ground truth' original network evaluations.\n\nThe largest hinderance by far in the paper is the quality of the writing. Much of the paper is not well written, and difficult to understand. In particular, the design section is written almost completely in prose without a well delineated algorithm. The paper does not do a good job of guiding the reader through their proposed approach, thus much hunting and guessing is required to understand exactly what it is that the authors are proposing. E.g., at first glance I believed that once the 'petri-dish' was initialized then ground truth evaluations would not be carried out again as it seems that is what the introduction is implying. However the authors combine the petri-dish with ground truth evaluations (Sec. 3.3) but this is not clearly presented in the introduction. Another difficulty is it's unclear exactly the precise experimental settings used, making it impossible to be sure whether the experiments are 'fair' to competing approaches. E.g., to generate Fig. 1 a 'petri-dish' is initialized and trained, however is it true that 'inference' carried out using this petri dish involves training a petri-dish network to convergence then evaluating its accuracy on the synthetic validation data? Although 'petri-dish inference' is defined in Sec 3.2, I see no usage of this exact verb in Sec 4.1. It's difficult to connect these concepts as a reader without a well delineated algorithm which concretizes with precision what is being proposed.\n\nBesides these issues, the proposed idea is not fully well explored which also leads to doubt on the robustness of the approach. One concern I have is that the robustness of petri dish idea at mimicking the original network is not well established outside of a rather simple experiment in Fig. 1. As this is a key linchpin to the proposed approach, I strongly require an extensive experiment in a complex setting (with multiple properties of a network being searched over). Another concern with Fig. 1 is that the 'ground-truth' evaluations used to seed the two models appear to be artificially restricted to the shaded box. Neural networks operate under the i.i.d. setting thus making this approach biased against the NN Model approach. The training and test dataset on this experiments should be sampled i.i.d. unless the authors can provide a compelling reason why.\n\nWith regards to Fig. 3, from what I can understand the authors considered usage of NAO over 3 iterations of 33 motifs each to yield 100 ground truth evaluation. Did the authors consider 5 iterations of NAO with 20 motifs each as in the petri-dish approach. What do the authors mean exactly with, \"For a fair comparison, original NAO is re-run in this limited ground-truth setting and\nthe resulting performance is depicted by the red-curve in Figure 3\"? Can the authors exactly delineate their experimental settings in the appendix or in the main text.\n\nI note that evaluation was not performed for CNNs. Also it would be nice to see the performance of this approach on standard NAS benchmarks such as NASBench.\n\n\nAnother shortcoming with the proposed approach is that the authors haven't proposed a 'general purpose' method to create petri dishes, and only width reduction is explored. Although this is a nice start, clearly this will not work if the architecture parameter being optimized over using NAS is 'width' itself. Have the authors considered such a scenario? How do the authors propose to create petri-dishes in the generalized setting? This shortcoming is glaring, and I'm not sure whether the proposed idea is well explored with a general purpose 'petri-dish' creation mechanism.\n\nPros:\n-Possibly neat idea if executed well.\n\nCons:\n-See above.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers overall appreciated the efforts of the authors in making NAS more computationally efficient. The paper could greatly benefit from further editing/restructuring with the goal of improving clarity, as it’s currently hard to navigate and understand in places. Future submissions of this work would benefit from more extensive empirical validation that motif networks mimic the original network. The reviewers also agreed that for the method to be appealing/useful, a general way to generate motif networks is needed. Overall, the outcome was that this is a very interesting idea but needs further development along the directions outlined above.
Decision: Reject