Transfer NAS with Meta-learned Bayesian Surrogates
Authors: Gresa Shala
Thomas Elsken
Frank Hutter
Josif Grabocka
Venue: ICLR 2023 notable top 5% (2023)

Abstract: While neural architecture search (NAS) is an intensely-researched area, approaches typically still suffer from either (i) high computational costs or (ii) lack of robustness across datasets and experiments. Furthermore, most methods start searching for an optimal architecture from scratch, ignoring prior knowledge. This is in contrast to the manual design process by researchers and engineers that leverage previous deep learning experiences by, e.g., transferring architectures from previously solved, related problems.
We propose to adopt this human design strategy and introduce a novel surrogate for NAS, that is meta-learned across prior architecture evaluations across different datasets. We utilizes Bayesian Optimization (BO) with deep-kernel Gaussian Processes,  graph neural networks for the architecture embeddings and a transformer-based set encoder of datasets. As a result, our method consistently achieves state-of-the-art results on six computer vision datasets, while being as fast as one-shot NAS methods.

Published as a conference paper at ICLR 2023
TRANSFER NAS WITH
META -EEAAED BAYESIAN SURROGATES
Gresa Shala', Thomas Elsken?, Frank Hutter!.2 & Josif Grabocka' 1 Department of Computer Science, University of Freiburg (shalag, fh, grabocka ) (ccs uni uni-freibuse de "ossh Center for Artificial Intelligence
thomas e elsken@de bosch. com
ABSTRACT
While neural architecture search (NAS) is an intensely-researched area, approaches typically still suffer from either (i) high computational costs (ii) lack of robust- ness across datasets and experiments. Furthermore, most methods start searching for an optimal architecture from scratch, ignoring prior knowledge. This is in contrast to the manual design process by researchers and engineers that leverage previous deep learning experiences by, e.g., transferring architectures from previ- ously solved, related problems. We propose to adopt this human design strategy and introduce novel surrogate for NAS, that meta-learned across prior architecture evaluations across different datasets. We utilize Bayesian Optimization (BO) with deep-kernel Gaussian Processes, graph neural networks for obtaining architecture embeddings and transformer-basee dataset encoder. As result, our method consistently achieves state-of-the -feth-a-----t results on six computer vision datasets, while being fast one-shot NAS methods.
INTRODUCTION
While deep learning has removed the need for manual feature engineering, it has shifted this manual work to the meta-level, introducing the need for manual architecture engineering. The natural next step is to also remove the need to manually define the architecture. This is the problem tackled by the field of neural architecture search (NAS).
Even though NAS is an robust and efficient. Blackbox optimization methods, such as reinforcement learning (Zoph Le) 2017), evolutionary algorithms (Real et al. 2019), and Bayesian optimization (Ru et al. 2021 White
area, there is still no NAS method that is both generally
ct al. 202T) work reliably but are slow. On the other hand, one-shot methods (Liu et al., 2019 Dong
& Yang, 2019b) often have problems with robustness (Zela et al. 2020), and the newest trend of zero-cost proxies often does not provide more information about an architecture's 's performance than simple statistics, such the architecture's 's number of parameters (White al. 2022).
An understudied path towards efficiency in NAS is to transfer information across datasets. This idea is naturally motivated by how researchers and engineers tackle deep learning problems: they leverage the knowledge they obtained from previous experimentation and, e.g., re-use architectures designed for one task and apply adapt them to a novel task. While few NAS approaches in this direction exist (Wong et al. 2018 Lian et al. 2020 Elsken et al. 2020 Wistuba 202T; Lee et al. 2021]RR Ru al.) 2021). they typically come with one or more of the following limitations: (i) they are only applicable to settings with little data, (ii) they only explore a fairly limited search space or even can just choose from handful of pre-selected architecture, or (iii) they can not adapt to data seen at test-time. One approach to obtain efficient NAS methods that has been overlooked in the literature so far is to exploit the common formulation of NAS as hyperparameter optimization (HPO) problem (Bergstra et al. 2013 Domhan et al.) 2015 Awad et al. 2021 and draw on the extensive literature on transfer HPO (Wistuba et al. 2016 Feurer et al. 2018a Perrone & Shen 2019 Salinas al. 2020 Wistuba & Grabocka 2021 In contrast to standard transfer HPO methods that meta-learn parametric surrogates from a pool of source datasets (Wistuba et al., 2016 Feurer et al. 2018a Wistuba & Grabocka, 2021). in this work we explore the direction of meta learning

---

Published as a conference paper at ICLR 2023
surrogates by contextualizing them on the dataset characteristies (a.k.a. meta neta-feetures) Vanncchrren 2018 Domaa et al. 2021a; Rivolli et al. )022).
In this work, present an efficient Bayesian Optimization (BO) method with novel deep-kerne surrogate that yields a new NAS method which combines the best of both worlds: the reliability of blackbox optimization
computational cost in the same order of magnitude one-shot approaches. Concretely, we propose a BO method for NAS that leverages dataset- coasstt-cottextullzeeddd surrogates for transfer learning. Following [ee et al. (202I), we use graph encoder (Zhang et al., 2019) to encode neural architectures and an attention-based dataset encoder (Lee et al.] (019) to obtain context features. We then use deep kernel learning (Wilson et al.) 2016) obtain meta-learned kernels for the joint space of architectures and datasets, allowing us to use the full power of BO for efficient NAS. This approach solves two key issues of Lee et al. (202T), which is closest to our work: (i) a lack of trading-off exploration vs. exploitation and (ii) the lack of exploiting new function evaluations on test task, blindly following what has been observed during meta training. As a result, our surrogates are optimized for efficiently transferring architectures for new target dataset based on its meta-features. To sum up, our contributions are as follows:
Inspired by manual architecture design, we treat NAS as a transfer or few-shot learning prob- lem. We leverage ideas from transfer HPO to meta-learn kernel for Bayesian Optimization, which encodes both architecture and dataset information.
We are the first to combine deep-kernel Gaussian Processes (GPs) with a graph neural network encoder, transformer- dataset encoder, the first apply BO with deep GPs to NAS, and the first to do all of this in a transfer NAS setting.
Our resulting method outperforms both state-of- of-tth-artt blackbox NAS methods as well as state-of-the-ar one-shot methods across six computer vision benchmarks.
To foster reproducibility, we make our code available /tttttt://ggiiiii com/ TNAS DCS/
TNAS -
We address the points in the "NAS Best Practices Checklist" in Appendix F
2 RELATED WORK
NAS is an intensely-researched field, with over 1000 papers published in the last two years alone!
We therefore limit our discussion of NAS to the most related fields of Bayesian optimization for
NAS and meta learning approaches for NAS. For full discussion of the NAS literature, we refer the
interested readers to a series of surveys by Elsken et al. (2019). Wistuba et al. (201)) and Ren et al.
(0200) and for an introduction to BO to Shahriari et al.
2016 Hutter et al. (2019).
Bayesian optimization (BO) for NAS. As BO is commonly used in hyperparameter optimization (HPO), one can simply treat architectural choices as categorical hyperparameters and re-use, e.g.. tree-based HPO methods that can natively handle categorical choices well (Bergstra et al. 2013 Domhan et al. 20155 Falkner et 2018). While Gaussian Processes (GPs) are more typically applied to continuous hyperparameters, they can also be used for NAS by creating such kernels for Ga-bd BO can be manually engineered (Swersky et al., 2013) Kandasamy et al. 2018 Ru al. (02T). A recent a alternative is to exploit (Bayesian) neural networks for BO (Snoek ct al. 2015; Sprisggnbenngggggg et al.) 2016: White et al.) 2021). However, while these neural networks are very expressive, they require more data to fit well than GPs and thus are outperformed by Ga-based approaches when only a few function evaluations can be afforded. In this work, we combine the sample efficiency of GPs and the expressive power of neural networks, by using deep GPs combined with graph neural network encoder.
appropriate kernel;
Meta learning for NAS. To mitigate the computational infeasibility of starting NAS methods from scratch for each new task, several approaches have been proposed along the lines of meta and transfer
learning. Most of these warm-star start the weights of architectures in target task WWong et al. (2018);
Lian et al. (2020); Elsken et al. (202)); Wistuba (2021). Ru et al. (202T) extracts architectural motifs
that can be reused on other datasets. Most related to our work is MetaD2A (Lee et al.) 2021), 1), where
the authors propose to generate candidate architectures and rank them conditioned directly on task, utilizing a meta-feature extractor (Lee et al. 2019). However, there are two key differences in our
See list at: https: http:///www.////// www.:/aattoooooo//atteeoeee wrg//uton /ttp:ommm//ttttttttt www.terttrreeooo neural a archi tecture-searon
2

---

Published as a conference paper at ICLR 2023
Dataset encoding
Meta-learned
Task
Joint encoding
Bayesian optimization
E
GP
Architecture encoding
Figure 1: Illustration of TNAS We employ a GNN & to encode architectures, a transformer & to encode dataset, and an MIP to merge the encodings. This joint encoding then fed into GP surrogate used within BO. All encodings are meta-learned.
work: (i) the performance predictor of MetaD2A is not probabilistic and thus can not naturally trade- off exploration exploitation but rather only exploits what has been observed during meta-trade meta-training. (ii) During the meta testing phase. MctaD2A simply proposes N architectures for a new task, trains the top-K (as estimated by the performance predictor) out of N architectures and returns the best. In particular, the performance of the evaluated architectures is not used as feedback for MetaD2A and thus the method never adapts to function evaluations on the test task, blindly following what has been observed during meta training. This can cause problems for new tasks which are poorly correlated with the meta training data. And, indeed, MetaD2A stagnates several of the datasets in our experimental analysis. In contrast, our approach, dubbed TNAS (transferrable NAS), uses the function evaluations from meta testing to update the surrogate employed within our BO framework. thus allowing to adapt to the meta testing scenario.
Meta transfer learning for HPO There are many approaches to achieve meta or transfer learning in HPO, see, e.g., the survey by Vanschoren (2019) FrrFer et al. (2018b, Section 7). One particularly promising approach is to employ Deep Kernel Learning [Wilson et al. 2016) which strives to learn the kernel function by using neural network to transform the input to latent representation, which then used in a kernel function. Wistuba Grabocka (2021) and Jomaa et al.) (2021b) utilized a deep kernel for transfer learning in HPO. While numerical hyperparameters an MLP, we adapt this approach to NAS by using graph neural network to encode architectures as inputs; also extend it by encoding datasets into latent embedding and learning deep kernel that spans the combined space of architectures and datasets.
be encoded using
PROPOSED METHOD
We consider the following problem: given history of datasets, where for each dataset D(9)
(g) we have already evaluated set of neural network architectures #1 , with corresponding performance (e.g., accuracy) (9) yn
(q) For a new dataset D(new), we want quickly discover an optimal architecture by leveraging information from the history of datasets. We build upon the state-of-the-art few-shot Bayesian optimization (BO) framework by Wistuba Grabocka (202T), which was proposed to address similar problem: transferring optimal hyperparameter configurations across datasets. The authors propose to learn deep kernel across tasks, which is then used for Gaussian process (GP) surrogate in the typical BO setup.
However, while hyperparameter configurations can typically be presented by an N-dimensional vector, it is less clear how to represent neural network architectures. Simply representing architectures as vectors and plugging them into an off-the-shelf GP kernel is likely sub-optimal. In fact, White et al.) (2020) have shown that the type of architecture representation substantially impacts the performance of a downstream NAS algorithm. To address this issue, we employ graph neural networks (GNNs) to obtain a learnable representation of neural networks. GNNs are common choice in NAS neural networks architectures can be naturally represented as graphs (Siems et al. 2020; White et al. 2020
Wen et al. 2020 Dudziak et al. 2020
3

---

Published as a conference paper at ICLR 2023
Furthermore, while we could directly feed this architecture encoding into the GP's kernel, we argue that the kernel should also be conditioned on the characteristics of a dataset in order to meaningfully asses (dis- simiilaiiissss between architectures. To motivate this, consider the question "What makes two architectures similar?". We argue that two architectures are not similar only because they share some similar sub-graph components (which will be represented by the GNN encoding), but also because they achieve similar performance on the target dataset. Following this line of reasoning, we condition the deep kernel on the characteristics (neta-features) of dataset (Vanschoren| (2018). In a similar fashion as for the architecture encoding, we again use a learnable representation of datasets via employing a set transformer, as also done by (Lee et al. 2019). The architecture and dataset encoding are then processed by fully-connected neural network, whose output serves as the input for an off-the-shelf kernel function, e.g., a Matérn kernel, which is finally used to compute the distance of two (architecture, dataset) datapoints. This results in an end-to-end learnable encoding of the problem, and the parameters of the GNN, transformer and fully-connected neural network are meta- peeaallarrned in a similar fashion as in Wistuba & Grabocka (202T). We refer to Figure for an overview of our framework.
In the following subsection, we discuss the details of the different components.
3.1 BAYESIAN OPTIMIZATION WITH DEEP KERNEL GAUSSIAN PROCESSES
We start by introducing Gaussian Processes, which represents the surrogate of our method within Bayesian optimization (BO). In typical hyperparameter optimization (HPO) setup, the inputs æ E X represent hyperparameter configurations, and the target y E y denotes the performance of a machine learning method when trained with the hyperparameter configuration x. Consider the training and testing D [(=] v, ) splits of a dataset of evaluated hyperparameters. In that context, GPs are non-parametric models that assume prior over functions, and approximate the target y E y R given the features x E X C R2 The estimation of the target variable y" for K(I,x*)
D [ y:) )== }=)}
( K(x,x) ( K(I", K(I,
the test instances xt is also jointly Gaussian as
Each respective block of the covariance matrix is the result of applying a kernel function k: X x X + R.
y0"
on pairs of instances, e.g., K(x, x=") x=").. == k(2x, (()) The estimated posterior mean and covariance of GPs (Rasmussen & Williams) 2006) for the target y² of the test instances x is given as follows:
Ely" | 1=" x. vl = K(x", K x)
cov(y" x=", .x] K(x", "(I,=))"KKK K(,, K(z,z").
(2)
We refer to, e.g., Murphy (012)) for the derivation. GPs are lazy models that rely on the similarity of the test instances to the training instances via kernel functions k. such as the Matér kernel. Unfortunately, typical kernels used with GPs are designed manually and rely on sub-optimal assump- tions (Cowen- et al.) (020)), which deteriorates the GP's performance. A promising direction for designing powerful and efficient kernel functions that adapt to a learning task is Deep Kernel Learning (Wilson et al.) 2016), where kernels are represented as trainable neural networks. A mapping : g projects the features latent representation, where similar instances are co-located.
The embedding& & for the deep kernel of our GPs is fully-connected neural network, that takes as input the encoding of the architecture & as well as the dataset encoding øo In detail, the L-dimensional architecture encoding & is fused with the K K-ddimensional dataset encoding e and processed through a fully connected neural network & : I.. K+L + RMM. where the last layer has M neurons. We re-use both the GNN-based architecture encoding and the transformer based dataset encoding from Lee jet al. 2021), thus we only briefly describe it below and refer to Lee et al. (202T) for details.
The architecture encoding t consists of a directed acyclic graph encoder (Zhang et al. 2019) to obtain the encoding for the architectures. By using one GRU cell to traverse the topological order of the DAG in the direction from the input the output, and another GRU cell to pass through the DAG in the backward direction, we obtain latent representations of the graph which are then put through a colnecco neural network layer to obtain the encoding for the architecture.
The dataset encoding & consists of two stacked Set- Ber-Tranffommr (Lee et al. 2019) architectures. The first Set- Transformer layer captures the interaction between randomly sampled data points of the same class, whereas the second one captures the interactions between the different classes of the dataset. The resulting output of the second Set- Transforffrr layer represents the dataset encoding.
4

---

Published as a conference paper at ICLR 2023
We tuned the dimensionality of the embedding of the dataset encoder and graph encoder, as well as the architecture of the feed-forward neural network of our method using the fidlliiyy Bayesian optimization method BOHB (Falkner et al. 2018) on the meta-training dataset; please refer to Appendix A for details.
Putting it all together, the kernel/similarity between two architectures, sprpiicii evaluated on dataset D, and x' evaluated on dataset D', is:
k (x,, = ts 0 (<([vox: (xx
(o)))) (x(x)))))))) (D; ( :w(t))
<
Φ (D' :o( : ww(k))
with w(()) being the parameters of the neural network &, w(() the parameters of the architecture encoding vi, wid) the parameters of the dataset encoding 6, and w(() additional parameters of
the kernel function. We denote the cumulative parameters as (w(S) w.(*), w(e), wx()), All
parameters are jointly meta-learned to maximize the marginal likelihood (Wistuba & Grabocka 2021). as will be discussed in the next section.
3.2 META-LEARNING DEEP-KERNEL GP SURROGATES
Recall that we assume we are given a set of Q datasets, where on each dataset Dg we have Ng E N_ evaluated architectures. We denote the n-th architecture evaluated on the q-th dataset as zq.n and its validation accuracy as Vq... The meta-dataset of all the evaluations on all the datasets is defined
as M = - U ( na }.) By x :- (æ ................................ : N.).N == (yI,1,.... :........VV.N., D (D....., D we denote vectors containing all the architectures, accuracies and datasets.
VQ.N.) and
The parameters w of the deep kernel are optimized jointly by maximizing the log marginal likelihood of the GP surrogate on the meta training dataset:
arg max log (y D; w)
arg min
D: w) y + log [(x, D; w)]-
In practice, we resort to sampling mini-batches and employ stochas- tic gradient descent, following estab-
Algorithm Meta-learning our deep-kernel GPs
Require:
meta-dataset M: learning rates NSGD: MREP;
lished practices [Wilson et al. 2016) Wistuba & Grabocka |02 Patacchi
inner update steps
2: while not converged do
ola et al. 2000). As all components of our method are differentiable, our approach is end-to-end differentiable.
3: Sample mini-batch from M:
x = II......dr Ik).
.y....., y.kl. D = [Di...., Dx]
4: L(w) = K
(x, D. w)y - log |K (x, D; w)
5: w' + w
For updating the meta-parameters w.
6: for j =1 to do
we use the meta-learning algorithm REPTILE (Nichol et al. 2018). due to its simplicity compared to, e.g.,
7: w' + w' nSGD V w/ L(w')
8: Update w +- TREP (w w')
MAML (Finn et al. 2017). Our method's pseudocode is shown in Algorithm T The procedure samples mini- batch of (architecture, dataset) pairs with corresponding validation accuracy (line 3). We sample datasets and architectures uniformly at random from the meta-training dataset. Then we fit dataset D using SGD by minimizing
the surrogate to estimate the accuracies of architectures Equation (lines 4-7). Finally, we use REPTILE (Nichol et al. 2018) to update w (line 8).
3.3 META-TESTING
Once the optimal w are found, we plug in the meta-learned kernel for GP, with the posterior and use vanilla Bayesian Optimization (BO) to quickly identify the optimal configuration in the new response surface. Note that by employing BO in the meta-testing phase. TNAS adapts the test task by updating the posterior of the GP surrogate with function evaluations from the test task (remember equations 2. We offer a more detailed description of the BO loop in Appendix
5

---

Published as a conference paper at ICLR 2023
CIFARIO
CIFARI00
MNIST
94
73
99.75
Accuracy
71
70
99.65
68
10
30
40
10
AIRCRAFT
PETS
60
96.6
96.5
96.4
Accuracy
35
96.3
40
30
96.2
30
96.1
96.0
10
15
20
10
15
20
10
15
20
25
GPU hours
# GPU hours
# GPU hours
GP-UCB
BANANAS
NASBOWL
HEBO
TNAS
Figure 2: Comparing TNAS to random search (RS) and the four different Bayesian optimization methods six image datasets.
EXPERIMENTAL SETUP
follow the experimental setup as Lee et al. (202I) for the NAS-Bench-201 (Dong & Yang 2020) and MobileNetV3 search spaces. On the NAS Bene--ennhhh-- search space, we also use the same meta datasets as Lee et al. (2021). It consists of 4230 meta-training datasets derived from ImageNet. For each of these datasets, the accuracy of one (different) architecture from the NAS-Bench-225 search space given. For the evaluation of our method and the baselines, we use six popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN, Aircraft, Oxford IIT Pets, and MNIST. For CIFAR-10 10 and CIFAR-100, we query the performances of architectures from the NAS-Bench. B---h-nnhhh1 benchmark, whereas for the other four datasets train the suggested architectures from scratch using the for each experiment and report the mean and standard deviations.
pipeline (as these are not available in the benchmark). We ran three trials
4.1 BASELINES
Classic HPO. A simple baseline is Random Search (RS) (Bergstra & Bengio 2012) RS samples architectures uniformly at random from the search space and returns the top preforming simple yet powerful bascline is Bayesian Optimization with a vanilla GP surrogate (Snock et al. 2012). We use the Matérn 5/2 kernel and rely on the GPytorch (Gardner et al. 2018 implementation. We tried both Expected Improvement (EI) and Upper Confidence Bound (UCB) as acquisition functions, with UCB performing better our experiments. We also compare to HEBO Ccowen-Rivers et 2020), black-box HPO method that performs input and output warping to mitigate the effects of heteroscedasticity and non-stationarity on HPO problems. HEBO won 2020 NeurlPS blackbox optimization challenge (Turner et al. 2021 We use the implementation provided by the authors.
Another
HPO for NAS. White et al. (2021) proposed a BO method for NAS that uses an ensemble of fully- connected neural networks surrogate, named BANANAS. Moreover, BANANAS uses path encoding for the neural architectures, which serves as an input to the ensemble. When applied to a new test task, BANANAS starts the neural architecture search from seratch. NASBOWL (Ru et al. 2021 is GP-based BO method for NAS and utilizes the Weisfeiler-Lehman kernel (Shervashidze 2011) We use the implementations provided by the authors for BANANAS and NASBOWL.
State-of-the-art in NAS. One-shot methods have recently shown strong empirical performance for
NAS. We compare GDAS (Dong & Yang 2019b), SETN (Dong & Yang 2019a). PC-DARTS (u

---

Published as a conference paper at ICLR 2023
CIFARIO
CIFAR100
MNIST
72.5
99.75
70.0
Accuracy
67.5
99.70
65.0-
99.65
62.5
60.0
10
20
10
AIRCRAFT
PETS
SVHN
60
96.4
Accuracy
50
96.2
96.0
30
95.8
95.6
25
95.4
5
10
15
20
10
15
20
10
20
30
40
GPU hours
# GPU hours
GPU hours
MetaDZA
TNAS
SETN
GDAS
PC-DARTS
DrNAS
Figure 3: Comparison of TNAS state-of-the-art NAS methods.
al. 2020) and DrNAS (Chen al. 2021). For these methods, we compare to published results from the literature for six computer vision datasets.
Transfer NAS. We compare the most related transfer NAS method, MetaD2A (Lee al., (022]) which is transfer learning NAS method with Cataset-contextualiii neural network generator and performance predictor. The neural network generator and performance predictor are meta-trained on the same source datasets that we also use. When applied to a test task, MetaD2A generates 500 candidate architectures conditioned on the test dataset and then selects the top architectures based on its performance predictor. We use the implementation provided by MetaD2A's authors.
RESEARCH HYPOTHESES AND EXPERIMENTAL RESULTS
Our experiments are designed to validate the following research hypotheses for our approach, dubbed TNAS:
Hypothesis 1: TNAS more efficient than classical HPO methods applied NAS well HPO methods specifically adapted to NAS and outperforms them in terms of achieving strong final performance.
while
Hypothesis 2: TNAS is competitive with one-shot approuches in terms of runtime.
In summary, we validate our claim from the introduction:
Hypothesis 3: TNAS achieves the consistency of blackbox optimization algorithms (such as classical HPO methods) while being as efficient as one-shot methods.
Results for Hypothesis In Figure we compare the performance of TNAS, with several HPO baselines. For all methods, use the top-performing architectures from the meta-training dataset as starting point. that sense, all these baselines are "transfer learning" by being initialized with best architectures on the meta- eeta-trrining dataset. On all of the datasets except CIFAR100, TNAS finds top-perfor oop-performin architectures faster than all the baselines and achieves stronger anytime performance. Furthermore, all benchmarks, TNAS eventually performs best.
Results for Hypothesis 2.
We demonstrate that with meta-leamed deep kernel within Bayesian Optimization, the search time can significantly reduced, to the same order as one-shot approaches. Figure 3 shows the performance of both TNAS and MetaD2A compared to state-of-the-art NAS
7

---

Published as a conference paper at ICLR 2023
10
10
X
Average Rank
8
x
Rank
4
CIFAR10
CIFARZOO
MNIST
AIRCRAFT
SWH
2
5
10
15
20
25
30
GPU hours
SETN
RS
HEBO
SETN
RS
HEBO
GDAS
GP
MetaD2A
X
GDAS
GP
MetaD2A
PC-Darts
BANANAS
TNAS
PC-Darts
BANANAS
TNAS
DINAS
DrNAS
(a) Ranking of TNAS across benchmarks.
(b) Ranking (averaged across benchmarks) over the course of runtime.
Figure 4: Consistency of TNAS compared to baselines.
methods. Except for CIFARIO and CIFARI00, TNAS clearly outperforms the NAS baselines by the time the one-shot approaches finish the search. On CIFAR 10, TNAS achieves similar performance as the baselines, while slightly inferior on CIFAR 100 for some baselines and only overtakes them given more time. We refer to Table in the appendix for concrete numbers. We can furthermore observe the drawbacks of MetaD2A discussed earlier: (i) MetaD2A does not trade-off exploration vs. exploitation but rather only exploits what has been observed during meta-training and (ii) MetaD2A does not use evaluations on the new target dataset as feedback and never adapts. As a result, MetaD2A stagnates several of the datasets.
Results for Hypothesis 3. In Figure we show that TNAS consistently achieves strong results, while the existing state-of-the-art NAS baselines have much higher variance their ranking changes across benchmarks. Furthermore, the figure shows how the ranking evolves over the course of running of the methods (by means of runtime in GPU hours). The analysis indicates that TNAS consistently achieves the best performance, for both small and large computational budgets, in particular also when compared to MetaD2A.
5.1
ABLATING OUR DESIGN CHOICES
CIFAR10
CIFAR100
94.5
74
73
Accuracy
83.5
71
70
92.5
25
50
GPU hours
GPU hours
TNAS (no Set Encoder)
TNAS (ne Graph Encoder)
TNAS
Figure 5: Ablation of the components of TNAS.
We empirically analyse our design choices, namely the graph and dataset encoder and demonstrate lift in performance compared to ceteris paribus ablations that do not employ these designs.
Concretely, we ablate our method's components and test them CIFAR-10 10 and CIFAR- 100, reporting the result in Figure TNAS (no Set Encoder) shows the performance of our method using graph encoder, but set-encoder (i.e., without using any dataset meta-features). TNAS (no Graph Encoder) shows the performance of our method using the dataset encoding in combination with matrix encoding for the architectures (i.e., graph encoder). TNAS outperforms the other variations: thus, we conclude that using both the learnable dataset meta peeta-features and graph neural network encoding is the most robust surrogate design. This finding validates the design choices of our method.
8

---

Published as a conference paper at ICLR 2023
94.5
init
(from meta-training set)
94.5
MetaLeeming vs NoMetal minng
94.0
94.0
96.0
93.5
93.5
03.0
92.5
RANDOMI -Metalearning
92.5
TOP2 Matalaanannnnnnng
92.5
92.0
RANDOM2- + Motaleariing
92.0
TOP2 MetaaLearning
92.0-
TCPS MetaLsaming
earning
earning
earming
91.5
RANDONS Metalsarning
91.5
91.5-
TOPS (NoMetaleamiing)
RANDONS Metaleerning
TOPS +Meta earnnig
RANDGMS /eotietteemmnnng
91.0
91.0
25
50
75
100
25
50
75
100
25
50
75
100
Trials
Trials
74-
Random K init
Top K init (frem meta training set) 74
74-
Metal eamning vs NoMetil ea ming
73-
72
Acturary
72
71
73
70
RANDOMI a Motalearning
70-
TOP1 eMetaLearning
70
Meraing
69
eleaming
RANDOMS- Metalsarning
TOPS Metalearrning
RANDOMS M MataLeaming
RANDOMS Met Metaleeriing
TOP4 ToM4+MetaLearme earrnig
TOPS (ettaetteg)nng)
RANDONS-NcLL Mectalanncnn
TOPS MMeaaLarning
RANDOMS INMMetaLasming!
25
50
75
100
67
25
50 * Trials
75
100
67 25
50
75
100
Trials
Trials
Figure 6: Ablation of the initial design and the initialization of our method on CIFAR-10 (top row) and CIFAR- 100 (bottom row). Left: using up to 5 random architectures; middle: using up to 5 top architectures from the veta-training set; right: ablating the comparison of (i) a meta-learned surrogate and a randomly initialized one with (ii) a random initial design and an initial design from the top 5 architectures from the meta-training set.
We also empirically evaluated (i) whether there is actually benefit in meta- leea-leerning the surrogate, and (ii) our method's performance with different initial architectures. Remember that as an initial design for our method, we used the top-5 performing architectures from the meta-training dataset. As alternatives, we consider values other than 5 and also start from randomly sampled architectures. We also turn-off meta-learning. The plots for these experiments are shown in Figure 6 evaluated on CIFAR-10 and CIFAR-100. The results suggest that both design choices are beneficial.
6 CONCLUSIONS AND FUTURE WORK
NAS is an intensely researched task and in essence is an instance of the hyperparameter optimiza- tion (HPO) problem. In this work, we exploited this relationship and, motivated by state-of-the-ar transfer HPO methods, adapted deep Gaussian Process (GP) surrogates to capture architecture repre- sentations. Furthermore, we proposed a novel conditioning of the deep GP on dataset meta-features enable transferring well-performing architectures from source datasets with similar meta-features. We empirically motivated the impact of each component of our proposed method through extensive ablations. In addition, we showed that our novel deep GPs with dataset meta-features and architecture encodings achieve the highest accuracy on six computer vision datasets compared to broad range of HPO methods, BO methods for NAS, and one-shot NAS methods. Lastly, we demonstrated that proxy architecture evaluations allow our method to discover more accurate architectures within the same search time one-shot NAS methods require.
Future work. A deep surrogate in principle allows us to capture the interaction between architectures and hyperparameter configurations. The NAS literature at the moment underexplores the impact of hyperparameters on the performance of an architecture. In fact, it is commonly known that the same architecture would perform differently if the training pipeline is altered, for example by changing the learning rate, the number of epochs, or the degree of regularization. Our previously-defimg surrogate can be trivially extended to model the interaction of architecture embeddings, the dataset meta-feeturee and hyperparameter configurations. We have not explored this direction empirically due to the lack of available NAS meta-datasets that vary both architectures and hyperparameters but would like to do in the future.

---

Published as a conference paper at ICLR 2023
ACKNOWLED
We acknowledge Hadi Jomaa as contributor to this paper on the level of co-author. We would like to acknowledge the grant awarded by the Eva-Mayr-Stihl Stiftung. In addition, this research was funded by the Deutsche Forschungsgemeinscaatt (DFG, German Research Foundation) under grant number 417962828. addition, Josif Grabocka acknowledges the support of the BaainLLink-BrainToooo center of excellence.
Reproducibility Statement. foster reproducibility, we make our code available https V/github. cwm/ TNAS
DCS We give details our experimental protocol in the
"NAS Best Practices Checklist" in Appendix
REFERENCES
Noor Awad, Neeratyoy Mallik, and Frank Hutter. Dehb: Evolutionary byberband for scalable, robust and efficient hyperparameter optimization. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence (IJCAI'21, 2021.
Bergstra, D. Yamins, and D. D. Cox. Making science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the 30th International Conference on International Conference on Machine Learning Volume 28, 2013.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. J. Mach. Learn. Res., 13:281-305. 2012.
Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsich. Dr(nas]: Dirichlet neural architecture search. In International Conference on Learning Representations, 2021. URL
net. ftt.m/orrmmmiiiee 9F 6YbmB3
Alexander Cowen-Rivers, Wenlong Lyu, Zhi Wang, Rasul Tutunov, Hao Jianye, Jun Wang, and Haitham Bou Ammar. Hebo: Heteroscedastic evolutionary bayesian optimisation. arXiv preprint arXiv:20/2.03822 2020. winning submission to the NeurIPS 2020 Black Box Optimisation Challenge.
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation learning curves. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI' 15, 2015.
Xuanyi Dong and Yi Yang. One-shot neural architecture search via self-evaluated template network. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019a.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1761-1770, 2019b.
Xuanyi Dong and Yi Yang. NAS-Bench-201: Extending the scope of reproducible neural architecture search. International Conference on Learning Representations (ICLR), 2020. URL https
//openreview.oo net.ewewwweeeeeeeeeeeem..e.eeeee
Lukasz Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas Lane. Brp-nas: Prediction-based nas using gens. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Prucessing Systems, volume 33, pp. 10480-10490. Curran Associates, Inc., 2020. URL http:: httttsss//pooeeeennggg neur cc/ 9b8fe8
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. Meta-learnin of neural architectures for few-shot learning. In CVPR, pp. 12362-12372, 2020. URL https:://///...... //d
2.00 01238
10

---

Published as a conference paper at ICLR 2023
S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning (ICML' (8), volume 80, pp. 1437-1446. Proceedings of Machine Learning Research, 2018.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for bayesian optimiza- tion. CoRR, abs/1 0bss/180..02299, 2018a.
Matthias Feurer, Benjamin Letham, Frank Hutter, and Eytan Bakshy. Practical transfer learning for 02219
bayesian optimization, 2018b. URL
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agmostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 22-1113. 2017.
Jacob R. Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew Gordon Wilson. Gpytorch: Blackbox matrix-matrix gaussian process inference with GPU acceleration. In Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 7587-7597, 2018.
F. Hutter, L. Kotthoff, and J. Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges, volume 5. 2019. Available for free at http://automml.or/bboo
Hadi S. Jomaa, Lars Schmidt-Thiene and Josif Grabocka. Dataset2vec: learning dataset meta- features. Data Min. Knowl. Discov. 35(3):964-99) 2021a.
Hadi Samer Jomaa, Sebastian Pineda Arango, Lars Schmidt-Thieme, and Josif Grabocka. Transfer learning for bayesian hpo with end-to-end landmark meta-features. In Fifth Workshop on Meta- Learning at the Conference on Neural Information Processing Systems, 2021b.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisation and optimal transport. In Advances in Neural Information Processing Systems, 2018
Hayeon Lee, Eunyoung Hyung, and Sung Hwang. Rapid neural architecture search by learning to generate graphs from datasets. In International Conference Learning Representations, 2021. URL http:ss:/ itt.://wwwweeeeeeeeeeeeeeeee wowoummiddrkuuuuuuoomo
net forum?id
Juho Lee, Yoonho Lee, Jungtack Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for iitention-based permutation-invariint neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 3744-3753, 2019.
Dongze Lian, Yin Zheng, Yintao Xu, Yanxiong Lu, Leyu Lin, Peilin Zhao, Junzhou Huang, and Shenghua Gao. Towards fast adaptation of neural architectures with meta learning. In International Conference on Learning Representations, 2020. httlhttts://opeeeeeee net/ forum?
Id=rleowANPVi
M. Lindauer and F. Hutter. Best practices for scientific research neural architecture search. Journal of Machine Learning Research, 21:1-18, 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference Learning Representations, 2019. URL httlttts////peeeiiieee net/ fttt:/forumiiddsleeeee...
K. Murphy. Machine learning: a probabilistic perspective. The MIT Press, Cambridge, MA, 2012.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. CoRR, abs/1803.02999, 2018
Massimiliano Patacchiola, Jack Turner, Elliot J. Crowley, Michael P. O'Boyle, and Amos J. Storkey. Bayesian meta-learming for the few-shot setting via deep kernels. In Advances in Neural Information Processing Systems 33; Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
11

---

Published as a conference paper at ICLR 2023
Valerio Perrone and Huibin Shen. Learning search spaces for bayesian optimization: Another view of hyperparameter transfer learning. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 12751-12761, 2019.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, 2006. ISBN 026218253X.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4780-4789, 2019.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. comprehensive survey of neural architecture search: Challenges and solutions. arXiv preprint arXiv:2006.02903 2020.
Adriano Rivolli, Luís P.F. Garcia, Carlos Soares, Joaquin Vanschoren, and André C.P.L.F. de Carvalho. Meta-features for meta-learning. Knowledge- Systems, 240: 2022. ISSN 0950-7051. doi: https://bb.org//0.0066j...ooov 200. 108101. URL httpp:///wwt sci wcwesssciiineieccccc.cc.coo.
com/ http://caooeeerraaee////////7770
Binxin Ru, Xingchen Wan, Xiaowen Dong. and Michael Osbore.. Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels. International Conference on Learning Representations, 2021.
David Salinas, Huibin Shen, and Valerio Perrone. A quantile-based approach for hyperparameter transfer learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pp. 8438-8448, 2020.
B. Shahriari, K. Swersky, Z. Wang, R. Adams, and N. de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148-175, 2016.
Nino Shervashidze, Pascal Schweitzer, Erik Jan Leeuwen, Kurt Mehlhom, and Karsten M. Borgwardt. Weisfeiler-Iehmar graph kernels. Journal of Machine Learning Research, 12(77): 2539-2561, 2011.
Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas- bench-301 and the case for surrogate benchmarks for neural architecture search. NeurIPS 4th Workshop on Meta-Learning, 2020.
Jasper Snock, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, 2012.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat, and Ryan P. Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on Machine Learning, SCM1.2015 Lille, France, 6-11 July 2015, pp. 2171-21800 2015.
Springenberg, A. Klein, S. Falkner, and F. Hutter. Bayesian optimization with robust Bayesian neural networks. In D. Lee, M. Sugiyama, U. von Luxburg. I. Guyon, and R. Garnett (eds.), Proceedings of the 29th International Conference on Advances Neural Information Processing Systems (NeurIPS' 16), 2016.
K. Swersky, D. Duvenaud, J. Snoek, F. Hutter, and M. Osborne. Raiders of the lost architecture: Kernels for Bayesian optimization conditional parameter spaces. In NeurIPS Workshop on Bayesian Optimization in Theory and Practice (BayesOpr' 13), 2013.
R. Turner, D. Eriksson, M. McCourt, Kiili. Laaksonen, Z. Xu, and I. Guyon. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. arXiv:2104 1020/ [cs.LG), 2021.
Vanschoren. Meta-learning. In Hutter et al. (2019) chapter 2, pp. 35-61. 61. Available for free at
http://automl.org/oook.
12

---

Published as a conference paper at ICLR 2023
Joaquin Vanschoren. Meta-learning: A survey. CoRR, abs/1810.03548, 2018.
Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 660-676, Cham, 2020. Springer International Publishing. ISBN 978-3-003-55522-66
Colin White, Willie Neiswanger, Sam Nolen, and Yash Savani. A study on encodings for neural archi- tecture search. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 20309-20319, Curran Associates, Inc., 2020.
Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural architectures for neural architecture search. 2021.
Colin White, Mikhail Khodak, Renbo Tu, Shital Shah, Sébastien Bubeck, and Debadeepta Dey. A deeper look at zero-cost proxies for lightweight nas. In ICLR Blog Track, 2022. URL https: // fclr-blog-traack . githuuthhhhg.o bigg::rr....ghhhho////2222222222
https://iclr-
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, 2016.
Martin Wistuba. Xfernas: Transfer neural architecture search. In Frank Hutter, Kristian Kersting, Jefrey Lijftijt, and Isabel Valera (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 247-262, Cham, 2021. Springer International Publishing. ISBN 978-3-030-67664-3
Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In International Conference Learning Representations, 2021.
Martin Wistuba, Nicolas Schilling, and Lars for automatic byperparameter optimization. In Machine Learning and Knowledge Discovery in Databases- European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I, pp. 199-214, 2016.
Two-stage transfer surrogate model
Martin Wistuba, Ambrish Rawat, and Tejaswini Pedapati. A survey on neural architecture search. arXiv preprint 1905.019000199 2019.
Catherine Wong, Neil Houlsby, Yifeng Lu, and Andrea Gesmundo. Transfer learning with neu- ral automl. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018 URL
www.:eiiiiiiinnnnnnnoeeeeeeeeeee ctcaaps.ccc/eee http//ppprr/22110 file/ pdd
f 13622222786448773333333222222383
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pe-darts: Partial channel connections for memory-efficient architecture search. In International
Conference on Learning Representations, 2020. URL http://///ppeeeeeeeeeeeeeeeeeeee
1d-BJ1S634tPr
A. Zela, T. Elsken, T. Saikia, Y. Marrakchi, Brox, and F. Hutter. Understanding and robustifying differentiable architecture search. In Proceedings of the International Conference on Learning Representations 20), 2020. Published online: iclr.cc.
Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Garnett, and Yixin Chen. D-vae: A variational autoencoder for directed acyclic graphs. In Advances in Neural Information Processing Systems, volume 32, 2019.
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. In Proceedings of the 5th International Conference on Learning Representations (ICLR 2017), 2017.
13

---

Published as a conference paper at ICLR 2023
A HYPERPARAMETER OPTIMIZATION FOR GNN AND SET TRANSFORMER ARCHITECTURES
We tuned the dimensionality of the embedding of the dataset encoder and graph encoder (Embedding dims.), the architecture of the feed-forward neural network of our method (Num. layers, Num. units in layer Num. units in layer 2, Num. units in layer 3, Num. units in layer 4), and the learning rate of the joint meta-training using BOHB (Falkner et al. 2018) on the meta-traini meta-training dataset.
Hyperparameter
Range
Embedding dims.
{8, 16, 20, 28, 32, 48, 56] [1. 4| (integer)
Num. layers
Num. units in layer 1
[16, 512 (log space)
Num. units in layer 2
[16, 512) (log space)
Num. units in layer 3
[16, 512 (log space)
Num. units in layer 4
[16, 512] (log space)
Learning rate
[10-6, 10- (log space)
Table 1: The hyperparameter space for tuning our method with BOHB (Falkner et al. 2018
B ARCHITECTURE ENCODING IN TNAS
To encode the architectures, we use directed acyclic graph encoder (Zhag et al. (2019). It takes architectures represented graph as input, and provides as output vector representations for those architectures. Following Zhang et al. (2019), the graph encoder consists of two GRU cells and fully- fully-connected neural network layer. Each GRU cell traverses the graph representation of the architecture in opposite directions to the other. The output of both cells is then concatenated and input through the fully-connected layer, which outputs the vector representation that will be used as an encoding for the architecture. We meta-train the graph encoder, well the dataset encoder, jointly with the embedding of the deep kernel during meta-training to maximize the log marginal likelihood of the GP surrogate.
C BAYESIAN OPTIMIZATION WITH TNAS
Once we meta-train our surrogate on the meta-training set using Algorithm[ we use it at meta-testing time a Bayesian optimization (BO) loop. First, we evaluate the top-5 architectures from the meta-training set the test dataset. Once the GP is fit to these architectures and their respective accuracies, we use EI as the acquisition function to find the next architecture to evaluate. In the NASBench201 search space, CIFAR100, whereas for SVHN, Aircraft, Pets, and MNIST, we do 40 evaluations. In the MobilenetV3 search space we do 50 evaluations for each of the datasets.
repeat this BO loop for total of 100 evaluations for CIFAR10 and
For maximizing the acquisition function in order to find the next candidate architecture, we either exhaustively evaluate the entire search space (in the case of small search spaces such NAS-Bench- 201), or simply sample K architectures randomly and pick the best (in the case of the MobileNet V3 space, K 104).
D COMPARISON OF TNAS AND METAD2A
We refer Table 2 for comparison of the key ingredients of al. (2021) and approach.
14

---

Published as a conference paper at ICLR 2023
predictor
candidate architecture generation GNN-bused architecture generutor. top hased predictor BO, maximuze acquisition function
test-fime adaptation
Lee et al. 2021
(dataset, architecture) encoding. (deterministic prediction) (dataset, architecture) encoding.
yes, adapt GP predictor w.r. new candidute evaluations
Ours
MLP, GP (probabilistic prediction)
Table 2: Comparison of our work to Lee et al. (2021).
E COMPARISON OF TNAS TO SOTA NAS METHODS AND MOBILENET V3 SPACE
Table 3:
Accuracy of our method (TNAS) in terms of time and accuracy compared to state-of-the-art
NAS methods the NASBench201 search space.
Data
Method GPU days
Accuracy
SETN
0.40
87
87.6440.00
GDAS
0.34
93.61
PC-DARTS
0.17
93.6610.1
CIFAR-10
DrNAS
0.30
94.3612.00
arch2vec
1.38
91.41+0.22
MctaD2A
2.08
94.37 +0000
TNAS
2.18
94.37=0.00
SETN
0.73
59.09 59.0919.24
GDAS
0.64
70. 70.70+0.30
PC-DARTS
0.28
66. 06.64422.34
CIFAR-100
DrNAS
0.45
73.5110000
arch2vec
1.38
73 73.35
MetaD2A
2.08
10.11
TNAS
2.18
73 51 +0.00
SETN
0.87
99.6949.04
GDAS
0.76
99 1..04
PC-DARTS
0.35
99.6610.04
MNIST
DrNAS
0.57
99.58 +0.59
MetaD2A
0.83 99.71+0.02
TNAS
0.89
08.00.0.00
SETN
0.46 44.84 #3.96
GDAS
0.46
53.52 23.524
PC-DARTS
0.29
26.33+3.40
Aircraft
DrNAS
0.65
46.08+7.00
MetaD2A
0.83
57.71+0.72
TNAS
0.89
59.5110.0
SETN
0.35 17
GDAS
0.33
24.02+2.76
PC-DARTS
0.28
25.31+1.38
Pets
DrNAS
0.31
26.73
MetaD2A
0.83
39.04+0.72
TNAS
0.89
43. 24.0.0
SETN
1.61
96.02+0.00
GDAS
1.46
95.57 +0.04
PC-DARTS
0.99
95.40+0.04
SVHN
DrNAS
1.24
96.3040.02
MetaD2A
1.08
96.4440.05
TNAS
1.18
96.57 10.00 1.000
15

---

Published as a conference paper at ICLR 2023
CIFARI00
CIFAR10
AIRCRAFT
PETS
17
90.5
Accuracy
94
16
90.0
89.5
93
14
2
GPU hours
GPU hours
GPU hours
GPU hours
MetaD2A
TNAS
Figure 7: Comparison of TNAS to MetaD2A on the MobileNetV3 search space.
MNIST
AIRCRAFT
PETS
SVHN
99.8
60
99.6
96.5
Accuracy
99.4
40
99.2
99.0
20
30
10
MetaD2A
TNAS
Figure 8: Comparison of TNAS to MetaD2A in terms of the accuracy of the suggested architectures evaluate on the NASBench201 search space. For MetaD2A the architectures generated by its architecture generator are evaluated sequentially based on MetaD2A's predictor ranking. TNAS adapts during the BO loop iterations, and thus suggests architectures conditioned on the previous evaluations.
CIFAR100
CIFAR10
AIRCRAFT
PETS
76
95
90
Accuraty
94
86
70
92
5
84
91
68
90
hours
INAS
MetaDZA
Figure 9: Comparison of TNAS to MetaD2A in terms the accuracy of the suggested architectures evaluate the MobileNetV3 search space.
16

---

Published as a conference paper at ICLR 2023
F NAS BEST PRACTICE CHECKLIST
We now describe how we addressed the individual points of the NAS best practice checklist Lindauer Hutter 2020).
1. Best Practices for Releasing Code
For all experiments you report:
(a) Did you release code for the training pipeline used to evaluate the final architectures? The code for the training pipeline for the architectures be found in the repo provide.
(b) Did you release code for the search space? We used the NAS-Bench-201 search space our experiments, the description and code for which publicly available.
(c) Did you release the hyperparameters used for the final evaluation pipeline, as well random seeds? We the -AS-Beech-220 pipeline and hyperparameters
final
evaluation pipeline. We release as well as the random seeds in the repo we provide. (d) Did you release code for your NAS method? The code for our NAS method can be found in battps://anenyyne
4open sci science/r FNAS-DCS-COR (c) Did you release hyperparameters for your NAS method, as well as random seeds? The hyperparameters for our NAS method, as well random seeds for the experiments can be found in the repo we provide.
2. Best practices for comparing NAS methods
(a) For all NAS methods you compare, did you use exactly the same NAS benchmark, including the same dataset (with the same training-test split), search space and code for training the architectures and hyperparameters for that code?? Yes, for fair comparison we made sure to use the same evaluation pipeline for all NAS methods we compare.
(b) Did you control for confounding factors (different hardware, versions of DL libraries, different runtimes for the different methods)? We ran all the methods on the same hardware and the same environment to control for confounding factors.
(c) Did you run ablation studies? We ran extensive ablation studies on the components of our method.
(d) Did you use the same evaluation protocol for the methods being compared? Yes.
(e) Did you compare performance time? Yes.
(f) Did you compare random search? Performance comparison to random search as well as other baselines can be found in Figure
(g) Did you perform multiple runs of your experiments and report seeds? For each of the experiments we performed three runs with different seeds (333, 444, 555).
(h) Did you use tabular or surrogate benchmarks for in-depth evaluations? We used NAS-Bench-201 as a tabular benchmark.
3. Best practices for reporting important details
(a) Did you report how you tuned hyperparameters. and what time and resources this required? We use FaHkeal et al. (2018) to tune the hyperparameters of our method. We ran BOHB with three different random seeds for 24 hours.
(b) Did you report the time for the entire end-to-end NAS method (rather than, e.g., only for the search phase)? Yes.
(c) Did you report all the details of your experimental setup? The details of our experimen- tal setup can be found in Section
17


Reviews:
[{"review_id": "z5t5LnAmAc", "reviewer": "Reviewer_9VTq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nSummary:\nThis paper proposes a neural architecture transfer technique. Although this problem is not new, this paper takes some reasonable measures. Its performance is worthy of recognition. This article is an above-average article and deserves a weak acceptance.\n\n", "summary_of_the_review": "\nSee \"Summary Of The Paper.\" This article is an above-average article and deserves a weak acceptance.\n\n", "strength_and_weaknesses": "\n(Negative) The neural frame transfer technique is not a new technique. Prior arts have had some exploration.\n\n(Positive) This article provides the code and also provides the \"NAS Best Practices Checklist.\" This is very positive because the reproducibility of NAS is widely questioned.\n\n\n(Positive) The method of this article is very simple.\n\n\n(Positive) This article introduces Dataset encoding, which is very reasonable.\n\n\n(Negative) In the article, the definition of architecture encoding is very vague. Please give more details of it.\n\n\n(Negative) During the acquisition of dataset encoding, positive and negative samples need to be randomly sampled. Is dataset encoding random? If so, please quantify this randomness or expectation.\n\n(Positive) The experimental results in this paper are very good (Figure 2 and Figure 3).\n\n\n(Negative) It would be better to add experimental validation in real open environments, such as the original ImageNet dataset and the open neural architecture search space. After all, neural architecture transfer and search techniques are meant to be used in real open environments. If the search can only be carried out in these fixed search spaces, its practical significance will be greatly reduced.\n\n\n(Positive) This article performs a full ablation analysis.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XDxz78UGEz", "reviewer": "Reviewer_3f7e", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a transfer NAS approach by leveraging the previous knowledge, i.e., (architecture, dataset, accuracy) triples.\nIt uses a graph neural network to encode the architecture information, a set transformer to encode the dataset information, and then learning a kernel function with an MLP. Empirical results on NAS-Bench-201 are strong.", "summary_of_the_review": "The quality of this paper is generally high. However, there are some problems in the method (how to propose new architecture) and the empirical evaluation part (lack of comparison, small search space). I would like see the rebuttal and then adjust my rating.", "strength_and_weaknesses": "### Strengths\n- The motivation of this paper is great, the idea of transfer previous knowledge is straightforward and lack of exploration. To my knowledge, there are previous methods trying to learn from history [1], but using on NAS is novel.\n- The results on NAS-Bench-201 are pretty robust across six datasets, and it's good to see from the ablation study that both the architecture and the dataset embedding have contributions to the final performance.\n\n### Weaknesses\n- When encounter a new dataset / architecture space, how do you propose a new architecture? Do you need to traverse lots of architectures and compare their score? I don't see discussions of this part in the paper.\n- Lack of comparison to predictor based methods [2, 3], which also learns an architecture embedding and directly predict the accuracy. It could be better if the authors can show that learning the Gaussian kernel function rivals.\n- For the learning process of the architecture / dataset embedding, do you just leverage the previous checkpoints or train them by yourselves? If it does, do you include the training cost into the NAS search cost?\n- Another concern is that all the experiments are shown in the NAS-Bench-201 space, which is pretty small. Is it possible to show the result on larger space, e.g., DARTS space. This can also show that the learnt embedding can transfer across spaces.\n\n[1] Yang et al. \"OBOE: Collaborative filtering for AutoML model selection.\" KDD 2019.  \n[2] Yan et al. \"Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?\" NeurIPS 2020.  \n[3] Ning et al. \"A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS.\" ECCV 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S07lajv_Rc", "reviewer": "Reviewer_kATh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "As most NAS algorithms search for well-performing architectures from scratch given the target dataset, this paper follows the existing line of transferring information across different datasets to accelerate the search process. Specifically, this paper follows (Lee et al., 2021) to learn the representation of datasets and architectures from many other datasets (i.e., as training data) and then propose to leverage Bayesian Optimization (BO) with deep-kernel Gaussian Processes to find the optimal architecture for a specific dataset. Empirical results show that this method can improve NAS from scratch and also the method in (Lee et al., 2019).", "summary_of_the_review": "In general, my major concerns lie in the novelty and clarity of this paper. I hope the authors can address my concerns during the rebuttal period.", "strength_and_weaknesses": "## Strengths\n1. Different from (Lee et al., 2019), this paper introduces deep-kernel Gaussian Processes as a probabilistic performance surrogate and intends to adaptively update this surrogate using newly evaluated architectures on the test task.\n2. This paper introduces BO to achieve query-efficient (i.e., search-efficient) optimization.\n\n\n## Weaknesses\n1. From my understanding, this paper follows the idea of (Lee et al., 2019) (i.e., meta-learn the representation of datasets and architectures and then predict the performance of architectures based on this learned representation) and only makes a few changes to the method, i.e., replacing its deterministic performance predictor with a probabilistic performance surrogate (i.e., the deep-kernel Gaussian Processes) and using BO for the optimization based on existing works. In this view, this paper mainly combines techniques from different fields for its method and therefore the novelty of this paper is kind of weak to me.\n2. In the abstract and introduction section, this paper doesn't provide a clear and detailed demonstration of its motivations, which not only makes me feel confused about why this paper is needed in the literature but also makes the contributions and the importance of this paper less convincing to me. Specifically, this paper only gives a brief introduction to the line of NAS using transferred information across datasets without elaborating more on why these existing works can not well satisfy the demand in practice and why the method in this paper is needed. Moreover, the introduction section of this paper lacks a comparison with its most related work (Lee et al., 2019) to justify the necessity of proposing the method in this paper.\n3. Some claims of this paper are not well supported. For example, in the related work section, this paper does not show (a) why the missing trade-off between exploration and exploitation in (Lee et al., 2019) is important and desirable for NAS using transferred information across datasets, and (b) why deep-kernel Gaussian Processes (GP) rather than GPs using the existing kernels are needed in this paper. Moreover, the result of Table 2 in this paper (a) can not unveil the drawbacks of MetaD2A, i.e., the missing exploration vs. exploitation tradeoff and the non-adaptive update in the optimization process and (b) may not support that these drawbacks lead to the stagnated performance of MetaD2A (located at the \"Results for Hypothesis 2\" of Sec. 5).\n4. While this paper claims that the method in this paper can adapt to function evaluations on the test task, Sec. 3 can not well support it because of the missing details of BO. I highly recommend the authors provide the details of how BO works to further justify this point.\n5. The empirical results in this paper show that in most cases the proposed method can only marginally improve over MetaD2A. More experiments that can distinguish these two methods may help to justify the superiority of this newly proposed method.\n\n## Questions\n1. What's the training cost for w in your proposed method? While there is only marginal improvement over existing NAS from scratch in Figure 2, the total cost (including the training cost for w and the search cost in the x-axis of Figure 2) for the method in this paper may become a problem in practice since it may be more time-consuming.\n3. As the w in this paper is trained based on the meta-datasets M, how does this method perform for the NAS problem with out-of-distribution datasets? Will it become a big issue for the proposed method? Because when only considering in-distribution datasets, directly transferring the selected architectures from other datasets can already provide competitive performances with compelling search costs as shown in Figure 2. So, the proposed method in this paper may not be necessary for this scenario. Meanwhile, if the proposed method can not guarantee its performance for the out-of-distribution datasets, this may also hinder the application of this method in practice.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2AaPkHCoTnU", "reviewer": "Reviewer_AX7F", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a transferrable surrogate for NAS based on Bayesian Optimization with deep-kernel Gaussian Processes. The proposed predictor can be adapt to unseen datasets rapidly by significantly reducing the search cost of NAS. On the NAS-Bench-201 and multiple unseen datasets, this work outperformed recent NAS methods including MetaD2A for the performance of the obtained architecture and search efficiency.", "summary_of_the_review": "The good points are that this work successfully addressed the important problem of the meta-learning-based prediction model in NAS with solid experiments. The bad points are that BO is not new in NAS domain, the composition of the paper is needed to be improved, and the benchmark that they used is small.", "strength_and_weaknesses": "- Strengths\n   - I think this work well addressed the main limitation of MetaD2A which is the meta-learning-based transferrable NAS method. As this paper said, the meta-learning-based predictor proposed by MetaD2A can be exploited to unseen datasets after once meta-training, which significantly reduces the search time for unseen datasets. However, MetaD2A only exploits the meta-learned predictor, can not adapt to the unseen dataset even if the unseen task provides few-shot samples. This work combines the BO method to tackle the problem of transferrable predictors by allowing them to reflect feedback from unseen tasks.\n\n- Weaknesses\n  - I think while the contribution of this work in the structure of predictor is limited, this work assigned the part of the paper for them too much. For example, using both transformer-based set encoder and graph encoder together is almost same with MetaD2A, this paper described it  as the figure and performed ablation study about that. I think it would be better to emphasis the difference between MetaD2A and this work or BO + predictor as a Figure.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%