DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models
Authors: Sohyun An
Hayeon Lee
Jaehyeong Jo
Seanie Lee
Sung Ju Hwang
Venue: ICLR 2024 poster (2024)
Keywords: Neural Architecture Search
Diffusion Model
Meta Learning
Primary area: general machine learning (i.e., none of the above)
TL;DR: A novel Diffusion-based Neural Architecture Generative framework called DiffusionNAG where the gradients of parameterized model-based predictors guide the generation process toward neural architectures with desired properties.

Abstract: Existing NAS methods suffer from either an excessive amount of time for repetitive sampling and training of many task-irrelevant architectures. To tackle such limitations of existing NAS methods, we propose a paradigm shift from NAS to a novel conditional Neural Architecture Generation (NAG) framework based on diffusion models, dubbed DiffusionNAG. Specifically, we consider the neural architectures as directed graphs and propose a graph diffusion model for generating them. Moreover, with the guidance of parameterized predictors, DiffusionNAG can flexibly generate task-optimal architectures with the desired properties for diverse tasks, by sampling from a region that is more likely to satisfy the properties. This conditional NAG scheme is significantly more efficient than previous NAS schemes which sample the architectures and filter them using the property predictors. We validate the effectiveness of DiffusionNAG through extensive experiments in two predictor-based NAS scenarios: Transferable NAS and Bayesian Optimization (BO)-based NAS. DiffusionNAG achieves superior performance with speedups of up to 35$\times$ when compared to the baselines on Transferable NAS benchmarks. Furthermore, when integrated into a BO-based algorithm, DiffusionNAG outperforms existing BO-based NAS approaches, particularly in the large MobileNetV3 search space on the ImageNet 1K dataset. Code is available at https://github.com/CownowAn/DiffusionNAG.

Published as a conference paper at ICLR 2024
DIFFUSIONNAG: PREDICTOR-GUIDER NEURAL ARCHI- TECTURE GENERATION WITH DIFFUSION MODELS
Sohyun An'; Hayeon Lee Jachyeong Jo', Seanie Lee', KAIST DeepAuto.ai", Seoul, South Korea
Sung Ju Hwang ..
(schyunan, hayeon926, harryje97, 1snfamily02, jawangg22 1kaistt. aa.ist..c.k....
ABSTRACT
Existing NAS methods suffer from either an excessive amount of time for repet- itive sampling and training of many task-irrelevant architectures. To tackle such limitations of existing NAS methods, we propose paradigm shift from NAS to novel conditional Neural Architecture Generation (NAG) framework based on diffusion models, dubbed DiffusionNAG. Specifically, we consider the neural architectures as directed graphs and propose a graph diffusion model for generating them. Moreover, with the guidance of parameterized predictors, DiffusionNAG flexibly generate task-optimal architectures with the desired properties for diverse tasks, by sampling from a region that is more likely to satisfy the proper- ties. This conditional NAG scheme is significantly more efficient than previous NAS schemes which sample the architectures and filter them using the property predictors. We validate the effectiveness of NiG through extensive ex- periments in two predictor-based NAS scenarios: Transferable NAS and Bayesian Optimization (BO)-based NAS. DiffusionNAG achieves superior performance with speedups of up to 35x when compared to the baselines on Transferable NAS benchmarks. Furthermore, when integrated into a BO-based algorithm, Diffusion- NAG outperforms existing BO-based NAS approaches, particularly in the large MobileNetV3 search space on the ImageNet IK dataset. Code is available at http:///b..ooo/ooooo/////illii
INTRODUCTION
While Neural Architecture Search (NAS) approaches automate neural architecture design, eliminating the need for manual design process with trial-and-error (Zoph & Le, 2017; Liu et al., 2019; Cai et al., 2019; Luo et al., 2018; Real et al., 2019; White et al., 2020), they mostly suffer from the high search cost, which often includes the full training with the searched architectures. To address this issue. many previous works have proposed to utilize parameterized property predictors (Luo et al., 2018; White al., 20b1a;b; 2023: Ning al., 2020; 2021; Dudziak et al., 2020; Lee et al., 2021a;bb 2023a) that can predict the performance of an architecture without training. However, existing NAS approaches still result in large waste of time as they need to explore an extensive search space and the property predictors mostly play passive role such as the evaluators that rank architecture candidates provided by a search strategy to simply filter them out during the search process.
To overcome such limitations, we propose paradigm shift from NAS (Neural Architecture Search) to novel conditional NAG (Neural Architecture Generation) framework that enables the generation of desired neural architectures. Specifically, introduce novel predictor- guiiodd Diffusion-barst Neural Architecture Generative framework called DiffusionNAG, which explicitly incorporates the predictors into generating architectures that satisfy the objectives (e.g., high accuracy robustness against attack). To achieve this goal, we employ the diffusion generative models (Ho et al., 2020; Song al., 2021), which generate data by gradually injecting noise into the data and learning to reverse this process. They have demonstrated remarkable generative performance across a wide range of domains. Especially, we are inspired by their parameterized model-guidann et al., 2015; Vignac et al., 2022) that allows the diffusion generative models to excel in conditional generation over diverse domains such as generating images that match specific labels (Ramesh et al., 2021) or discovering new drugs meeting particular property criteria (Lee et al., 2023b).
mechanism -Soks-Dickstein
"These authors contributed equally this work.

---

Published as a conference paper at ICLR 2024
In this framework, begin by training the base diffusion generative model to generate architectures that follow the distribution of a search space without requiring expensive label information, e.g., accuracy. Then, to achieve our primary goal of generating architectures that meet the specified target condition, we deploy the trained diffusion model to diverse downstream tasks, while controlling the generation process with property predictors. Specifically, we leverage the gradients of parameterized predictors to guide the generative model toward the space of the architectures with desired properties. The proposed conditional NAG framework offers the key advantages compared with the conventional NAS methods as follows: Firstly, our approach facilitates efficient search by generating architectures that follow the specific distribution of interest within the search space, minimizing the time wasted exploring architectures that are less likely to have the desired properties. Secondly, DiffusionNAG, which utilizes the predictor for both NAG and evaluation purposes, shows superior performance compared to the traditional approach, where the same predictor is solely limited to the evaluation role. Lastly, DiffusionNAG is easily applicable to various types of NAS tasks (c.g.. latency or robustness-constrained NAS) retraining the base generative model, making it practical for diverse NAS scenarios.
swap out the predictors in plug-and and-play manner without
Additionally, to ensure the generation of valid architectures, design novel score network for neural architectures. In previous works on NAS, neural architectures have been typically represented directed acyelic graphs (Zhang al., 2019) model their computational flow where the input data sequentially passes through the multiple layers of the network to produce an output. However, existing graph diffusion models (Niu al., 2020a; Jo et al., 2022) have primarily focused on undirected graphs, which represent structure information of graphs while completely ignoring the directional relationships between nodes, and thus cannot capture the computational flow in architectures. To address this issue, we introduce score network that encodes the positional information of nodes to capture their order connected by directed edges.
We demonstrate the effectiveness of DiffusionNAG with extensive experiments under two key predictor-based NAS scenarios: 1) Transferable NAS and 2) Bayesian Optimization (BO)-based NAS. For Transferable NAS using transferable dataset-aware predictors, DiffusionNAG achieves superior or comparable performance with the specdup of up to 35x on four datasets from Transferable NAS benchmarks, including the large MobileNetV3 (MBv3) search space and NAS-Bench-201. Notably, DiffusionNAG demonstrates superior generation quality compared to MetaD2A (Lee et al., 2021a), closely related unconditional generation-basee method. For BO-based NAS with task-specific predictors, DiffusionNAG outperforms existing BO-based NAS approaches that rely on heuristic acquisition optimization strategies, such as random architecture sampling or architecture mutation, across four acquisition functions. This is because DiffusionNAG overcomes the limitation of existing BO-based NAS, which samples low-quality architectures during the initial phase. by sampling from the space of the architectures that satisfy the given properties. DiffusionNAG obtains especially large performance gains on the large MBv3 search space on the ImageNet IK dataset, demonstrating its effectiveness in restricting the solution space when the search space is large. Furthermore, we verify that our score network generates 100% valid architectures by successfully capturing their computational flow, whereas the diffusion model for undirected graphs (Jo et al., 2022) almost fails.
Our contributions can be summarized as follows:
We propose paradigm shift from conventional NAS approaches to novel conditional Neural Architecture Generation (NAG) scheme, by proposing a framework called DiffusionNAG. With the guidance of the property predictors, DiffusionNAG can generate task-optimal architectures for diverse tasks.
DiffusionNAG offers several advantages compared with conventional NAS methods, including efficient and effective search, superior utilization of predictors for both NAG and evaluation purposes, and easy adaptability across diverse tasks.
Furthermore, ensure the generation of valid architectures by accurately capturing the compu- tational flow, we introduce a novel score network for neural architectures that encodes positional information in directed acyclic graphs representing architectures.
We have demonstrated the effectiveness of DiffusionNAGG in Transferable NAS and BO-NAS scenar- ios, achieving significant acceleration and improved search performance in extensive experimental settings. Diffusion siggniiicann significantly outperforms existing NAS methods in such experiments.
2

---

Published as a conference paper at ICLR 2024
One-time Score Network Training
Search Space
Forward SDE (Arch > Noise]
0.1
0.9
3,4
-2.6
Dz
Preffittr-quuided
Diffusion
GEX LOS
Uncardtnnd Eflccion
2.6
Malse
guided
Acuuady
(1)) F.. Xaan LA
&
Diffusion
Reverse SDE (Noise + Arch)
Noise
Uitenr
Training Dataiett-wware Prodictor over p(C)
Tou Dit
Bur Trahing
Task
Unioon T
VA. log (As)
Acc. Transter f
f
VA, log for (V+ Dr, Ad)
Figure 1: Illustration DiffusionNAG in Transferable NAS Scenarios. DiffusionNAA generates desired neural architectures for a given unseen task by guiding the generation process with the transferable datasct-aware performance predictor fø- (ys (D, Ac).
METHOD
In Section 2.1, first formulate the diffusion process for the generation of the architectures that follow the distribution of the search space. In Section 2.2, we propose a conditional diffusion framework for NAG that leverages a predictor for guiding the generation process. Finally, extend the architecture generation framework for Transferable NAS in Section 2.3.
Representation of Neural Architectures A neural architecture A in the search space is typically considered as a directed acyclic graph (DAG) (Dong & Yang, 2020b). Specifically, the architecture A with N nodes defined by its operator type matrix V R&X* and upper triangular adjacency matrix E N XN as A = (V, E) E I X x IN N where F' is the number of predefined operator sets. In the MobileNetV3 search space (Cai et al., 2020), N represents the maximum possible number of layers, and the operation sets denote set of combinations of the kernel size and width.
2.1 NEURAL ARCHITECTURE DIFFUSION PROCESS
As first step, we formulate an unconditional neural architecture diffusion process. Following Song al. (2021), we define forward diffusion process that describes the perturbation of neural archi- tecture distribution (search space) to a known prior distribution (e.g., Gaussian normal distribution) modeled by stochastic differential equation (SDE). and then learn reverse the perturbation process to sample the architectures from the search space starting from noise.
Forward process We define the forward diffusion process that maps the neural architecture distri- bution p( An) to the known prior distribution pl (A)) as the following Itô SDE:
dA, f, + g.d
where I-subscript represents a function of time (F.()) F(-()), fe(:): ( is the linear drift coefficient, gt:
is the scalar diffusion coefficient, and w the standard Wiener process. Following Jo et al. (2022), we adopt a similar approach where architectures are regarded as entities embedded in a continuous space. Subsequently, during the forward diffusion process, the architecture is perturbed with Gaussian noise at each step.
Reverse process The reverse-time diffusion process corresponding to the forward process is modeled by the following SDE (Anderson, 1982: Song et al., 2021):
dA, = (A.) of A. log p. (A.)) di + g:dw,
(2)
where Pt denotes the marginal distribution under the forward diffusion process, dł represents an infinitesimal negative time step and w is the reverse-time standard Wiener process.
In order to use the reverse process as generative model, the score network is trained to approxi- mate the score function V A, log P. (A.) with the following score matching (Hyvärinen, 2005; Song
3

---

Published as a conference paper at ICLR 2024
et al., 2021) objective, where A(t) is given positive weighting function:
0 arg arg E. A, EA,14, ( () A., Po
pa (A.)1
(3)
Once the score network has been trained, we can generate neural architectures that follow the original distribution p(Ao)) using the reverse process of Equation (2). To be specific, we start from noise sampled from the known prior distribution and simulate the reverse process, where the score function approximated by the score network 80. (Az,t). Following various continuous graph diffusion models (Niu et al., 2020a; Jo et al., 2022), we discretize the entries of the architecture matrices using the operator 2.s to obtain discrete 0-1 matrices after generating samples by simulating the reverse diffusion process. Empirically, we observed that the entries of the generated samples after simulating the diffusion process do not significantly deviate from integer values of 0 and 1.
Score Network for Neural Architectures To generate valid neural architectures, the score network should capture 1) the dependency between nodes, reflecting the computational flow (Dong & Yang, 2020a; Zhang et al., 2019), and 2) the accurate position of each layer within the overall architecture to comply with the rules of specific search space. Inspired by Yan al. (2021a) on architecture encoding, we use I. transformer blocks (T) with an attention mask M E N that indicates the dependency between nodes, i.e., an upper triangular matrix of DAG representation (Dong & Yang, 2020a: Zhang al., 2019), to parameterize the score network. (See Appendix B for more detailed descriptions) Furthermore, we introduce positional embedding Em (v.) to more accurately capture the topological ordering of layers in architectures, which leads to the generation of valid architectures adhering to specific rules within the given search space as follows:
Emb. Embops (v:) + Emb (v.) + Emb (t), where V. i-th row of V for i [N]]
(4)
s0 (A.,1) 1) = MLP (Hr), where Hº- = Emb; H' = T (H' .)) and H' H Hy],
(5)
where Embope (v,) and Emb (t) are embeddings of each node v; and time f, respectively.
While simulating Equation (2) backward in time can generate random architectures within the entire search space, random generation is insufficient for the main goal of DiffusionNAG. Therefore, we introduce a conditional NAG framework to achieve this goal in the following section.
2.2 CONDITIONAL NEURAL ARCHITECTURE GENERATION
Inspired by the parameterized model-guidance scheme (Sohl-Dickstein et al., 2015; Vignac et al., 2022; Dhariwal & Nichol, 2021), we incorporate parameterized predictor in our framework to actively guide the generation toward architectures that satisfy specific objectives. Let y be the desired property (e.g., high accuracy or robustness against attacks) we want the neural architectures to satisfy. Then, we include the information of y into the score function. To be specific, we generate neural architectures from the conditional distribution Pr( (A y) by solving the following conditional reverse-time SDE (Song et al., 2021):
dA = [G. - of A. pi (A, [x)] di + g.d..
(6)
Here, we can decompose the conditional score function V A, log Pe (A, y) in Equation (6) the sum of two gradients that is derived from the Bayes' theorem p( A, (y) x p( A,) ply/ At):
VA, log
(At) + V log pa A.). Aoloog (PAAA. (A, y) = V pa
(7)
By approximating the score function V A, p. with the score network Syr, the conditional generative process of Equation (6) can be simulated if the term VA, log Pa A,) could be estimated. Since log Pe (y) At) represents the log- likelihood that the neural architecture A, satisfies the target property y, we propose to model log Pe (At) using a pre-trained predictor (2( parameterized by Φ, which predicts the desired property given a perturbed neural architecture A,:
V V Po A.) = V A, log (4(v)
(8)
As result, we construct the guidance scheme with the predictor as follows, where hy is a constant that determines the scale of the guidance of the predictor:
dA, (f((At) 3² se- (A., + V A. lo fvl1) dt + g.dw.
(9)
Intuitively, the predictor guides the generative process by modifying the unconditional score function which is estimated by sg- at each sampling step. The key advantage of this framework is that we only need to train the score network once and can generate architectures with various target properties by simply changing the predictor. Our approach can reduce significant computational overhead for the conditional NAG compared to the classifier- free guidance scheme (Hoogeboom et al., 2022) that requires retraining the diffusion model every time the conditioning properties change.
4

---

Published as a conference paper at ICLR 2024
Table I: Comparison with Transferable NAS MBv3 Search Space. The accuracies are reported with 95% confidence intervals over runs. The p-value represents the result of conducted on the accuracies 30 architecture samples obtained by our method and cach baseline method.
Transferable NAS
MetaD2A (Lee et al., 2021a)
TNAS
Stats.
DiffusionNAG
(Shala et al., 2023)
Max
97.45-10.07
97.48 97.48-10.14
97.52:10.07
Mean
97.28++0.01
97.22-+0.00
97.39==0.01
CIFAR-10
Min
97.09
95.62:10..99
97.23-10.00
p-value
0.00000191
0.0024
Max
86.00-10.19
85.95-1:0.29
86.07
Mean
85.56:10.02
85.30110.04
8.07
CIFAR-100
Min
84.74+0..13
81.30=t-.118
85.42=00088
p-value
0.0037
0.0037
Max
82.18:100.0
82.31
82.28 12288.0.22
Mean
81.19:0.11
80.86100.15
0.05
Aireraft
Min
79.71+00,54
74.99--0.65
80,88
p-value
0.0169
0.0052
Max
95,28+00.50
95.04
95.34=0.29
Mean
94.0511:..3
94,47 0.110
94.75 1.751...00 0.10
Oxford-IITT Pets
Min
03.6
92.39110.04
9..2
p-value 0.0025
0.0031
2.3 TRANSFERABLE CONDITIONAL NEURAL ARCUITECTRR GENERATION
Transferable NAS (Lee et al., 2021a; la; Shala et al., 2023) offers practical NAS capabilities for diverse real-world tasks, by simulating human learning. They acquire knowledge from past NAS tasks to improve search performance on new tasks. this section, achieve highly efficient Transferable NAS, we extend the conditional NAG framework discussed earlier into a certuuiion-aased transferable NAG method by combining our framework with the transferable dataset-aware predictors from Trans- ferable NAS methods (Lee al., 2021a; Shala al., 2023). A dataset-aware predictor f (D, AA) is conditioned on dataset D. In other words, even for the same architecture, if datasets are different, the predictor can predict accuracy differently. The predictor is meta-learned with Equation (21) over the task distribution PTT) utilizing a meta-dataset == (+), Yis D with K tasks consisting of (dataset, architecture, accuracy) triplets for each task. We use the meta-dataset collected by Lee et al. (2021a). The key advantage is that by exploiting the knowledge learned from the task distribu- tion, we can conduct fast and accurate predictions for unseen datasets without additional predictor training. We integrate the meta-learned dataset-aware predictor fø (D A,) into the conditional neural architecture generative process (Equation (9)) for an unseen dataset D as follows:
dA: (f((Ar) o² s0- (Ac,t) kt V log folv folui A.)] } dt + g.dw.
(10)
EXPERIMENT
We validate the effectiveness of DiffusionNAG on two predictor-based NAS scenarios: Transferable NAS (Section 3. and BO-based NAS (Section 3.2). In Section 3.3, we demonstrate effectiveness of the proposed score network.
Search Space We validate our framework two Transferable NAS benchmark search spaces (Lee et al., 2021a): MobileNetV3 (MBv3) (Cai et al., 2020) and MAS-Bench-22 201 (NB201) (Dong & Yang, 2020b). Especially. MBv3 is large search space, with approximately 10" 19 architectures. (Please see Appendix C. for detailed explanations.)
Training Score Network The score network is trained only once for all experiments conducted within each search space. Note that training the score network only requires architectures (graph) without the need accuracy which is expensive information. The training process required 21.33 GPU hours (MBv3) and 3.43 GPU hours (NB201) on Tesla V100-SXM2, respectively.
3.1 COMPARISON WITH TRANSFERABLE NAS METHODS
Experimental Setup Transferable NAS methods (Shala et al., 2023; Lee et al., 2021a) are designed to leverage prior knowledge learned from previous NAS tasks, making NAS more practical
5

---

Published as a conference paper at ICLR 2024
Table 2: Comparison with Transferable NAS on NB201 Serach Space. We present the accuracy achieved on four unseen datasets. Additionally, we provide the number of neural architectures (Trained Archs) that are actually trained to achieve accuracy. The accuracies are reported with 95% confidence intervals over 3 runs.
CIFAR- 10
CIEAR- 100
Aircraft
Oxford HIT Pets
Type
Method
Accuracy (%)
Trained Archs
Accuracy (8)
Trained Archs
Accuracy (%)
Trained Archs
Accuracy (R)
Trained
Archs
ResNet al. 20166)
23e97
N/A
70.86:200
NIA
47.01gm
N/A
25.58
RS (Bergstra & Bengio, 2012)
93.70
500
21.04
> 500
REA (Real al., 2019)
93,92.200
500
71,84
500
REINFORCE (Williams, 1992)
93.85.011
500
71.71
500
RSPS (Li Talwalkar: 2019)
84.07:33x
N/A
-2.31
NIA
12.19:155
NIA
22.91
NEA
SETN (Dong & Yang, 2019g)
87.64:200
N/A
59.09
NIA
44. 14
NIA
25.17
NIA
One-shot NAS
GDAS (Dong & Yang, 30196)
93.61 201e
N/A
70.
70.70
NIA
53. 15
N/A
24.02
NIA
PC-DARTS (Xu et 2020)
93.66-0.17
66.611223
NIA
26.3
25
DrNAS (Chen et al., 2021)
94,36.00
N/A
73.51
N/A
46.08
NIA
26.73zza1
NEA
BOHB (Fallner et al., 2018)
92.61 aoda
>x 500
a0.
+.0
GP-UCB
94.37
13.14
100
41.72
40
40 40.60
30-bbsed NAS
BANANAS (White 2021a)
94.37:sue
73,51:439
88
41.72=da
40.15
NASBOWL (Ru et al., 2021)
94.34
100
$7
s3.73
40
11.299
HEBO (Cowen Rivers et al. 2022)
94,34a000
100
72.62
100
49.32
40
40.55
18
TNAS (Stala 2023)
14.37.20e
59
59.15
76
40
Transferable NAS
MenalZAA (L.s et 20211a
94.37:sae
79.34
100
57.71 Lg
DiflusionNAG (Ours)
94.37:2xe
73.51
2
58.83
3
41.80
2
We report the search of one-shot NAS methods in Appendix C.3.
Table 3: Statistics of the generated architec-
Figure 2: The distribution of generated architectures.
tures. Each method generates 1,000 architectures.
CIRR1000
CIPAR100
Target
Stons.
Oracie Top 1,000
Random
MetaD2A
Uncond. SSorting
Cond. (Ours)
200
1.9
Cardd (Sur)
190
Max
94:37
94.37
94.37
94.37
94.37
CIFARIO
Mean
93.50
K7.12
01:53
00.77
03.13
300
93.18
10.00
10.00
10.00
86.44
Max
73.51
72.74
73.51
73.16
73.51
CIFARI60
Mean
70.62
61.59
67.14
66.37
71.34
Min
69.91
1.00
1.00
1.00
58.09
Accuraty
Accuracy
unseen task. To achieve this, all Transferable NAS methods, including our DiffusionNAG, utilize a transferable dataset-aware accuracy predictor, as described in Section 2.3. The dataset-aware predictor is meta- eettntraiied on the meta-dataset provided by Lee et al. (2021a), which consists of 153,408/ 1/00/4,25 /42330 meta-training tasks for MBv3/NB201, respectively. For more details, please refer to Lee et al. (2021a). MetaD2A (Lee et al., 2021a), which is the most closely related to our work, includes an unconditional architecture generative model that explicitly excludes the dataset-aware predictor during the generation process. Instead, MetaD2A needs to search for optimal architectures across multiple tasks, train these architectures to obtain their accuracy data, and use this costly accuracy collection to train its generative model. Besides, it uses the dataset-aware predictor only during the subsequent evaluation stage to rank the generated architectures. During the test phase, it first abjeetive-unconditiondd
generates architectures and then evaluates the top architectures using its predictor. TNAS (Shala et al., 2023) enhances the meta-learned dataset-aware predictor's adaptability to unseen datasets by utilizing BO with the deep-kernel GP strategy without involving any generation process (Please see Appendix C.2 for details of the baselines.). DiffusionNAG conditionally generates architectures with the diffusion model guided by the dataset-aware predictor. Our generation process, with sampling batch size of 256 takes up to 2.02 GPU minutes on Tesla V100-SXM2 to sample one batch. Finally, we select the top architectures sorted by the predictor among the generated candidates. We conduct experiments on Transferable NAS benchmarks (Lee et al., 2021a) such as four unseen datasets CIFAR-10. 10, CIFAR- 100, Aircraft (Maji et al., 2013), and Oxford IIT Pets (Parkhi et al., 2012) from large search space MBv3 (Table 1) and, NB201 (Table 2).
Results on MBv3 Search Space In Table 1, MetaD2A, TNAS, and DiffusionNAG obtain the top 30 neural architectures for each datasets. Subsequently, we train these architectures on the datasets following the training pipeline described in Appendix C.5. Once the architectures are trained, we analyze the accuracy statistics for each method's group of architectures. Additionally, we calculate p-value to assess the statistical significance of performance differences between the architecture groups obtained via DiffusionNAG and each method. A p-value of 0.05 or lower denotes that statistically meaningful difference exists in the performances of the generated architectures between the two groups
The results demonstrate that, except for the Aircraft dataset, DiffusionNAG consistently provides architectures with superior maximum accuracy (max) compared other methods across three
6

---

Published as a conference paper at ICLR 2024
NB201 CIFAR1OD
MBv3 ImageNet 1K
MBV3 MBv3-ImageNee 1K
73.5
03.50
73.0
83.25
83.25
72.5
72.0
03.00
ROHAMAAN
Gu deed GemeBO IDund
82.75
82.75
EXCULTION
71.0
Huzation
Glided Gan (Cur))
02:50
Geided Gen 1Dun)
Putation
82:50
LOCAL SEARCH
RANDON
70.5
Randem
62.25
Fardorn
82.25
GCN PREDACTIA
40
60
100
20
60
100 20
60
80
100
Trained Archs
Trained Archs
Traired Archs
Figure 3: Comparison Results on Existing AO Strategies. Guided Gen (Ours) strategy provides pool of candidate architectures, guiding them toward perfforrrrmeec distribution using the current population with DiffusionNAG. We report the results of multiple experiments with different random seeds.
MBv3 ImageNet 1K
MBv3 ImageNet 1K
MBV3 ImageNet 1K
83.50
Guided Can (0uns
63.50
Gupke Gan IDuni
Cuided Ger (Oun)
83:25
Sandim
Mutetion
03.25
Petstton
fardern
83:25
Nyzatian
Random
83.00
83.00
03:00
02.75
82.75
82.75
02.50
82.50
82.25
Thained Archs
Traindd Archs
Trained
(a) EI
(b) ITS
(c) UCB
Figure 4: Experimental Results on Various Acquisition Functions. Ours consistently outperforms the
heuristic approaches on various acquisition functions. run experiments with 10 different random seeds.
datasets. Additionally, the mean accuracy and minimum accuracy (min) of architectures within the DiffusionNAG group are higher across all datasets. In particular, the p-values obtained from comparing the groups of architectures suggested by DiffusionNAG and those from other baselines are consistently below the 0.05 threshold across all datasets. This indicates that the architectures generated by DiffusionNAG have shown statistically significant performance improvements compared to those provided by the baseline methods when using transferable dataset-aware predictors. Furthermore, the results clearly support the superiority of the proposed predictor-guided conditional architecture generation method compared with either excluding predictors during generation (MetaD2A) or relying solely predictors without generating architectures (TNAS).
Results on NB201 Search Space We highlight two key aspects from the results of Table 2. Firstly, the architectures generated by DiffusionNAG attain oracle accuracies of 94.37%% and 73.51% on CIFAR-10 and CIFAR-100 datasets, respectively, and outperform architectures obtained by the baseline methods on Aircraft and Oxford-IIIT Pets datasets. While Metal22 and TNAS achieve accuracies of 59.15% 1 /5.757..1111 and 40. 50.00%%/22 00%/39004% on Aircraft and Oxford-IIIT Pets datasets, respectively, DiffusionNAG achieves comparable or better accuracies of 58.83% and 41 80%, demonstrating its superiority. Secondly, DiffusionNAG significantly improves the search efficiency by minimizing the number of architectures that require full training (Trained Archs) to obtain final accuracy (For CIFAR-10 and CIFAR-100, accuracy is retrieved from NB201 benchmarks) compared to all baselines. Specifically, when considering the Aircraft and Oxford-IIIT Pets datasets, DiffusionNAG only needs to train 3/2 architectures for each dataset complete the search process while MetaD2A and TNAS require 40/40 /0 and 26/6 6 architectures, respectively. This results in a remarkable speedup of at least 15x and up 35x on average.
Further Anaylsis We further analyze the accuracy statistics of the distribution of architectures generated by each method within the NB201 search space. Specifically, we conduct an in-depth study by generating 1. 1,000 architectures using each method and analyzing their distribution, as presented in Table and Figure 2. We compare DiffusionNAAG with two other methods: random architecture sampling (Random) and MetaD2A. Additionally, to assess the advantage of using a predictor in both the NAG and evaluation phases compared to an approach where the predictor is solely used in the evaluation phase, we unconditionally generate 10,000 architectures and then employ the predictor to select the top 1,000 architectures (Uncond. + Sorting). DiffusionNAG (Cond.) leverages the aataset-aware predictor f A.) to guide the generation process following Equation (10).
7

---

Published as a conference paper at ICLR 2024
The results from Table 3 and Figure highlight three key advantages of our model over the baselines. Firstly, our model generates a higher proportion of perfornnnnn architectures for each target dataset, closely following the Oracle Top-1000 distribution within the search space. Secondly, our model avoids generating extremely low-accuracy architectures, unlike the baseline methods, which generate architectures with only 10% accuracy. This suggests that our model is capable of focusing on a target architecture distribution by excluding underperforming architectures. Lastly, as shown in Figure 2, DiffusionNAG (Cond.) outperforms sorting after the unconditional NAG process (Uncond. + Sorting). These results highlight the value of involving the predictor not only in the evaluation phase but also in the NAG process, emphasizing the necessity of our conditional NAG framework.
3.2 IMPROVING EXISTING BAYESIAN OPTIMIZATION-BAST NAS
In this section, we have demonstrated that DiffusionNAG significantly outperforms existing heuristic architecture sampling techniques used in Bayesian Optimization (BO)-based NAS approaches, leading to improved search performance in BO-based NAS.
BO-based NAS The typical BO algorithm for NAS (White et al., 2023) is as follows: 1) Start with an initial population containing neural architecture-accuracy pairs by uniformly sampling no architectures and obtaining their accuracy. 2) Train a predictor using architecture the population, and 3) Sample с candidate architectures by the Acquisition Optimization strategy (AO strategy) (White et al., 2021a) and choose the maximizing an acquisition function based on the predictions of the predictor. 4) Evaluate the accuracy of the selected architecture after training it and add the pair of the chosen architecture and its obtained accuracy to the population. 5) Repeat steps 2) to 4) during N iterations, and finally, select the architecture with the highest accuracy from the population as the search result. (For more details, refer to Algorithm I in the Appendix.)
pairs in
Our primary focus is on replacing the existing AO strategy in step 3) with DiffusionNAG to improve the search efficiency of BO-based NAS approaches. Baseline AO Strategy: The simplest AO strategy is randomly sampling architecture candidates (Random). Another representative AO strategy is Mutation, where we randomly modify one operation in the architecture with the highest accuracy in the population. Mutation Random combines two aforementioned approaches. Guided Gen (Ours): Instead of relying on these heuristic strategies, we utilize DiffusionNAG to generate the candidate architectures. Specifically, we train predictor {3 At), as described in Equation (8), using architecture-accuraa process of our diffusion model to generate architectures. We then provide these generated architecture candidates to the acquisition function in step 3) (See Algorithm 2 in the Appendix.)
pairs the population. The trained predictor guides the generation
Comparison Results with Existing AO Strategies The left and middle sides in Figure 3 illustrates our comparison results with existing AO strategies. These results clearly highlight the effectiveness of DiffusionNAG (Guided Gen (Ours)), it significantly outperforms existing AO strategies such as Random, Mutation, and Mutation + Random on the CIFAR 100 dataset from NB201 and the large-scale ImageNet 1K (Deng al., 2009) dataset within the extensive MBv3 search space. In particular, BO-based NAS methods employing Random or Mutation strategies often suffer from the issue of wasting time on sampling low-quality architectures during the initial phase (White et al., 2020; Zela et al., 2022). In contrast, DiffusionNAG effectively addresses this issue by offering relatively high-performing architectures right from the start, resulting in significant reduction in search times. As result, as shown in the right side of Figure 3, our approach outperforms existing BO-based NAS methods, by effectively addressing the search cost challenge of them.
Experimental Results on Various Acquisition Functions In addition to the Probability of Improve- ment (PI) used in Figure 3, we investigate the benefits of DiffusionNAG across various acquisition functions, such as Expected Improvement (ED, Independent Thompson sampling (ITS), and Upper Confidence Bound (UCB) as shown in Figure 4. (Please see Appendix D.2 for more details on acquisition functions.). The experimental results verify that DiffusionNAG (Ours) consistently out- performs heuristic approaches, including Mutation, Random, and Mutation + Random approaches, on four acquisition functions: PI, EI, ITS, and UCB, in the large MBv3 search space.
3.3
THE EFFECTIVENESS OF SCORE NETWORK FOR NEURAL ARCHITECTURES
In this section, we validate the ability of the proposed score network to generate architectures that follow the distribution of NB201 and MBv3 search spaces. For NB201, we construct the training set
8

---

Published as a conference paper at ICLR 2024
Table 4: Generation Quality. generate 1,000 samples with each method for 3 runs different seeds.
NAS- Bench-201
MobileNetV3
Method
Validity (5))-
Uniq. (%) f
Novelty (%)
Validity (%)
Uniq. (%) f
Novelty (%)
GDSS Jo al. (2022)
4.56_1.4
1.00
Ours (w/ Pos. Emb.)
100.00=s.0
98.96-1.
49.08+215
42.17
100.00=so
100,00+a.e
Ours (w/ Pos. Emb.)
100.00
98. 70
19.
100.00:are
100.00=cmm
100.00
by randomly selecting 50% of the architectures from the search space, while for MBv3, we randomly sample 500,000 architectures. We evaluate generated architectures using three metries (Zhang et al., 2019): Validity, Uniqueness, and Novelty, Validity measures the proportion of valid architectures generated by the model, Uniqueness quantifies the proportion of unique architectures among the valid ones, and Novelty indicates the proportion of valid architectures that are not present in the training set. As shown in Table 4, our score network generates valid architectures with 100% % Validity, whereas GDSS (Jo et al., 2022), a tfate-of-the-art graph diffusion model designed for undirected graphs, fails to generate valid architectures, with the validity of only 4.56% and 0.00 % for NB201 and MBv3, respectively. Furthermore, our positional embedding yields significant improvements, indicating that it successfully captures the topological ordering of nodes within the architectures. Notably, in the MBv3, Validity improves from 42.17% % to 100.00%, highlighting the necessity of positional embedding for generating architectures with a large number of nodes (a.k.a. "long-range"). Additionally, our framework generates 49.20% //100.00000000000000000000000000 novel architectures that are not found in the training set, as well as unique architectures 98.70 %/100.00 % for NB201 and MBv3, respectively.
4 RELATED WORK
Neural Architecture Search NAS is an automated architecture search process (Ning et al., 2021; Zoph & Le, 2017) and roughly can be categorized into reinforcement learning-based (Zoph & Le, 2017; Zoph et al., 2018; Pham et al., 2018), evolutionary algorithm-based (Real et al., 2019; Lu et al., 2020), and gradient-based methods (Luo al., 2018: Liu et al., 2019; Dong & Yang, 2019b; Xu et al., 2020; Chen et al., 2021). Recently, Shala et al. (2023); Lee et al. (2021a) have proposed Transferable NAS to rapidly adapt unseen tasks by leveraging prior knowledge. However, they still suffer from the high search cost. DiffusionNAG addresses these limitations by generating architectures satisfying the objective with a guidance scheme of meta-leamned dataset-aware predictor.
Diffusion Models Diffusion models, as demonstrated in prior work (Song & Ermon, 2019; Ho al., 2020; Song et al., 2021), are designed to reverse the data perturbation process, enabling them to generate samples from noisy data. They have achieved success in a variety of domains, including images (Nichol et al., 2022: Rombach al., 2022), audio (Jcong al., 2021; Kong et al., 2021), and graphs (Niu et al., 2020b; Jo et al., 2022). However, existing diffusion models are not well-suited for Neural Architecture Generation (NAG) because their primary focus is on unconditionally generating undirected graphs. To overcome this limitation, this study introduces conditional diffusion- based generative framework tailored for generating architectures represented as directed acyelic graphs that meet specified conditions, such as accuracy requirements.
CONCLUSION
This study introduced a novel conditional Neural Architecture Generation (NAG) framework called DiffusionNAG, which is the paradigm shift from existing NAS methods by leveraging diffusion models. With the guidance of property predictor for a given task, DiffusionNAG can efficiently generate task-optimal architectures. Additionally, the introduction of a score network ensures the generation of valid neural architectures. Extensive experiments under two key predictor-based NAS scenarios demonstrated that DiffusionNAG outperforms existing NAS methods, especially effective in the large search space. We believe that success underscores the potential for further advancements in NAS methodologies, promising accelerated progress in the development of optimal neural architectures.
9

---

Published as a conference paper at ICLR 2024
Reproducibility Statement To ensure reliable and reproducible results, have provided the source code, available at https://gtthub.co. Cttthub.co//oooowww. www.ooooowwwwffffoein.n An/DiffusionNAG and the "NAS Best Practices Checklist" Ap- pendix A. Furthermore, we have deseribed the details of the diffusion process and models used in DiffusionNAG in Appendix B, as well as the experimental details in Appendix C.
This paper introduces novel conditional Neural Architecture Generation framework called DiffusionNAG, aimed at advancing the field of generative models, especially for neural architectures. We believe that work can open up new avenues for neural architecture search/generation, achieving significant acceleration and improved search performance in extensive experimental settings. Although possibility of causing any negative societal impacts through our work is low, we should be cautious about the potential for our work to be utilized to generate neural architectures for unethical purposes.
Ethics Statement
Acknowledgement This work was supported by the National Research Foundation of Korea(NRI) grant funded by the Korea MSII) (No. RS-20023444 2023-00256255), Center for Applied Research in Arti- ficial Intelligence (CARAI) grant funded by DAPA and ADD (UD19003IRD), Institute of Information communications Technology Planning & Evaluation (ITP)) grant funded by the Korea government(MSIT) (No.2022-0-007 13), and Institute of Information & communications Technology Planning & Evaluation (ITP) grant funded by the Korea govermmen(MSITT) (No.2019-0-000) Artificial Intelligence Graduate School Trogram(KAIST))
REFERENCES
Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12 (3)313-322, 1982.
James Bergstra and Yoshua Bengio. Random search for papem-earameter optimizatio... The Journal of Machine Learning Research, 13(1):281-303 2012.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and
hardware. In International Conference Learning Representations, 2019.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and
specialize it for efficient deployment. In International Conference on Learning Representations, 2020.
Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsich. DrNAS: Dirichlet neural architecture search. In International Conference on Learning Representations, 2021.
Alexander Cowen-Rivers Rivers, Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan-Rhys Griffiths, Alexandre Maravel, Jianye Hao, Jun Wang, Jan Peters, and Haitham Bou Ammar. HEBO: Pushing the limits of sample-efficient hyperparameter optimisation. Journal of Artificial Intelligence Research, 74, 07 2022.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fie attacks. International conference on machine learning, pp. 2206-2216. PMER, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. CVPR, 2018.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.0455: 2017.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8709 2021.
Xuanyi Dong and Yi Yang. One-shot neural architecture search via self-evaluated template network. In Proceedings of the IEEE International Conference on Computer Vision (CVPR), 2019a.
Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In Proceedings of the IEEE Conference on computer vision and pattern recognition (CVPR), 2019b.
Xuanyi Dong and Yi Yang. Nas-bench-201. Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020a.
Xuanyi Dong and Yi Yang. Nas-hench-201s Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020b.
10

---

Published as a conference paper at ICLR 2024
Lukasz Dudziak, Thomas Chau, Mohamed S Abdelfattah, Royson Lee. Hyeji Kim, and Nicholas D Lane. Brp-nas: Prediction-basee nas using gens. Advances neural information processing systems (NeurIPS), 2020.
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter optimization scale. International Conference on Machine Learning, pp. 1437-1446. PMLR, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Emiel Hoogeboom, Victor Garcia Satorras. Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference Machine Learning, pp. 8867-8887. PMLR, 2022.
Jie Hu, Li Shen, and Gang Sun. Squeeze-ance and-andddieee exciiteiittioon networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
Aapo Hyvärinen, Estimation of non-normalized statistical models by score matching. J. Mach. Learn. Res., 6: 695-709, 2005.
Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Choi, and Nam Soo Kim. Diff-tts: denoising diffusion model for text-to-speech, In Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August September 2021, pp. 3605-3609. ISCA, 2021.
Jachyeong Jo, Scul Lec, and Sung Ju Hwang. Score-basee generative modeling of graphs via the system of stochastic differential equations. In Proceedings of the 39th International Conference on Machine Learning, 2022.
Steffen Jung, Jovita Lukasik, and Margret Keuper. Neural architecture design and robustness: A dataset. In The Eleventh International Conference Learning Representations, 2023.
Zhifeng Kong, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In ICLR, 2021.
Alex Krizhevsky, Learning multiple layers of features from tiny images, 2009.
Hayeon Lee, Eunyoung Hyung, and Sung Ju Hwang. Rapid neural architecture search by learning generate graphs from datasets. International Conference on Learning Representations, 2021a.
Hayeon Lee, Sewoong Lee, Song Chong, and Sung Ju Hwang. Hardware-adaptinn efficient latency prediction for nas via meta-learning Advances Neural Information Processing Systems, 34:27016-27028, 2021b.
Hayeon Lee, Sohyun An, Minseon Kim, and Sung Ju Hwang. Meta-prediction model for distillatonnawaree NAS on
datasets. In The Eleventh International Conference on Learning Representations, 2023a. URL https: http::////peeeeeiieeeee
n forum? ? (---sShSSSsggggeseeeeee
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: framework for attention based permutation-invariae neural networks. International Conference on Machine Learning (ICML), 2019.
Scul Lee, Jachyeong Jo, and Sung Ju Hwang. Exploring chemical space with score-based -uttr--distribution generation. Proceedings of the 40th International Conference on Machine Learning, 2023b.
Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, Cong Hao, and Yingyan Lin. Hw-nas-bench: Hardware-aware neural architecture search benchmark. In International Conference on Learning Representations, 2021.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in Artificial Intelligence, pp. 367-377. PMLR, 2019.
Marius Lindauer and Frank Hutter. Best practices for scientific research neural architecture search. The Journal of Machine Learning Research, 21(1):9820-98 000799777448 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In In International Conference on Learning Representations (ICLR), 2019.
11

---

Published as a conference paper at ICLR 2024
Ily Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:/688.0390 2016.
Zhichao Lu, Kalyanmoy Deh, Erik Goodman, Wolfgang Banzhaf, and Vishnu Naresh Boddeti. NSGANetV2: Evolutionary objei-obeective surrogate- assisted neural architecture search. In European Conference on Computer Vision (ECCV). 2020.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. In Advances in neural information processing systems (NeurIPS), 2018.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:/30.55113 2013.
Willie Neiswanger, Kirthevasan Kandasamy, Barabass Poczos, Jeff Schneider, and Eric Xing. Probo: Versatile bayesian optimization using any probabilistic programming language. arXiv preprint arXiv: 1911:1190111155.. 2019.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. International Conference on Machine Learning, ICML. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp- 16784-16804. PMIR, 2022.
Xuefci Ning, Yin Zheng, Tianchen Zhao, Yu Wang, and Huazhong Yang. A generic graph-based neural architecture encoding scheme for predictor- based nas. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 28, 2020, Proceedings, Port XIII, pp. 189-204. Springer, 2020.
Xuefei Ning. Changcheng Tang, Wenshuo Li, Zixuan Zhou, Shuang Liang, Huazhong Yang. and Yu Wang. Eval- uating efficient performance estimators of neural architectures. Advances in Neural Information Processing Systems, 34:12265-12277, 2021.
Chenhao Niu, Yang Song. Jiaming Song. Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score- based generative modeling. In International Conference Artificial Intelligence and Statistics, pp. 4474 4484. PMLR, 2020a.
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. In AISTATS, 2020b.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference on Machine Learning (ICMI), 2018
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Tyya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. PMLR, 2021.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture scarch. In Proceedings the aaai conference artificial intelligence (AAAI), 2019.
Robin Rombach, Andreas Blatmaann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Puttern Recognition. CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10674-10685. IEEE, 2022.
Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Osborne. Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kemnlls. In International Conference on Learning Representa- tions, 2021. URL https: / http//ppnnccccccceeeee/e. net fertt//orummice jww.eeddddddodoooooo jd.
Gresa Shala, Thomas Elsken, Frank Hutter, and Josif Grabocka. Transfer NAS with meta-learned bayesian surrogates. The Eleventh International Conference on Learning Representations, 2023.
Jasper Snock, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. arXiv preprint 120.vv:106..944 2012.
Jascha Sohl-Dickse Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine I Learning, pp. 2256-2265. PMI.R, 2015.
12

---

Published as a conference paper at ICLR 2024
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances Neural Information Processing Systems, 32, 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021.
Simo Särkkä and Arno Solin. Applied Stochastic Differential Equations. Institute of Mathematical Statistics Textbooks. Cambridge University Press, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser. and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.1473 2022.
Wei Wen, Hanxiao Liu, Yiran Chen. Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In European Conference on Computer Vision, pp. 660-666660 Springer, 2020.
Colin White, Sam Nolen, and Yash Savani. Local search state of the art for nas benchmarks. arXiv preprint arXiv:2005.02960 pp. 76, 2020.
Colin White, Willie Neiswanger, Yash Savani. Bananas: Bayesian optimization with neural architectures for neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021a.
Colin White, Arber Zela, Robin Ru, Yang Liu, and Frank Hutter. How powerful are performance predictors in neural architecture search? Advances in Neural Information Processing Systems, 34:28454-2846) 202
Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber 7ela, Debadeepta Dey, and Frank Hutter. Neural architecture search: Insights from 1000 papers. arXiv preprint arXiv:2301.08722 2023.
Ronald Williams. Simple statistical gradient- following algorithms connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pe-darts: Partial channel connections for memory-efficient architecture search. In International Conference on Learning Representations (ICLR), 2020.
Shen Yan, Kaiqiang Song, Fei Liu, and Mi Zhang. Cate: Computation-aware neural architecture encading with transformers. In ICML, 2021a.
Shen Yan, Colin White, Yash Savani, and Frank Hutter. Nas-bench-xI1 and the power of learning curves. Advances in Neurul Information Processing Systems, 2021b.
Arher 7ela, Julien Niklas Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter. Surrogate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks. In Tenth International Conference on Learning Representations, 2022.
Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Garnett, and Yixin Chen. D-VAE: A variational autoencoder for directed acyelic graphs. Advances in Neural Information Processing Systems, 32, 2019.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. International Conference on Learning Representations (ICLR), 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEFE conference computer vision and pattern recognition (CVPR), 2018
13

---

Published as a conference paper at ICLR 2024
Organization This supplementary file contains detailed explanations of the materials that not covered in the main paper, along with additional experimental results. The supplementary file organized as follows:
Appendix A We cover the points in the NAS Best Practices Checklist (Lindauer & Hutter, 2020).
Appendix B We elaborate the details of our approach, DiffusionNAG.
Appendix C- elaborate on the detailed experiment setups, such as search space, baselines, datasets, architecture training pipeline, and implementation details.
Appendix D - We provide the results of additional experiments.
A NAS BEST PRACTICE CHECKLIST
In this section, provide description of how we cover each point in the NAS Best Practice Checklist (Lindauer Hutter, 2020).
Best Practices for Releasing Code For all experiments you report:
(a) Did you release the code for the training pipeline used evaluate the final architectures? We released code for the training pipeline.
(b) Did you release the code for the search space? In our main experiments, utilized the NAS- Bench-201 MobileNetV3 search space, which is publicly available along with description code.
(c) Did you release hyperparameters used the final evaluation pipeline, as well random seeds? Yes, can be found in the code we provide.
(d) Did you relcase code for your NAS method? The code for our work is available on GitHub: https://itthub.o www.hlu.cooo/ooooooeeee An/DiflusionAAG.
(c) Did you release hyperparameters for your NAS method, well random seeds? Yes, can be found in the code we provide.
2. Best practices for comparing NAS methods
(a) For all NAS methods you compare, did you use exactly the same NAS benchmark, including the same dataset (with the same training-test split), search space and code for training the architectures and hyperparameters for that code? Yes, for a fair comparison, we used the same evaluation pipeline for all NAS methods compare.
(b) Did you control for confounding factors (different hardware, versions of DL libraries, different runtimes for the different methods)? To control for confounding factors, ensured that all methods
executed the same hardware and environment. (c) Did you run ablation studies? Yes.
(d) Did you use the same evaluation protocol for the methods being compared? Yes.
(e) Did you compare performance over time? Yes.
Did you compare random search? Performance comparison random search and other baselines can be found Table of main paper.
(g) Did you perform multiple of your experiments and report seeds? For each of the experiments we performed three runs with diffierntt seeds (777,888,999)
(h) Did you use tabular surrogate benchmarks for in in-depth evaluations? We used NAS-Bench- (Dong & Yang. 2020b) as a tabular benchmark.
3. Best practices for reporting important details
(a) Did you report how you tuned hyperparamcters, and what time and resources this required? Yes.
(b) Did you report the time for the entire end-to-end NAS method (rather than, e.g., only for the scarch phase)? Yes.
(c) Did you report all the details of your experimental setup? The details of our experimental setup can be found in Appendix C.
B DETAILS OF DIFFUSIONNAG
B. OVERVIEW
In this section, we first describe the forward difftusion process that perturbs neural architectures towards random graphs in Appendix B.2. Subsequently, we illustrate our specialized score network for estimating neural architecture scores in Appendix B.3. Finally, elaborate the components of the dataset-aware predictor in Dif in Appendix B.4.
14

---

Published as a conference paper at ICLR 2024
B.2 DIFFUSION PROCESS
Our approach aims to apply continuous time generative diffusion processes to neural architectures. Specifically, we utilize the Variance Exploding (VE) SDE (Song al., 2021) to model the forward diffusion process for our neural architecture generation framework. The following SDE describes the process of the VE SDE (Io et al., 2022):
dA = Uiii
Omax
2 log
a m'ax dw:
(11)
mmin
Omin
where Omin. mmax and 1 (0.1) are hyperparameters. The transition distribution the process Gaussian, given that Equation (11) has a linear drift coefficient. We can obtain the mean and covariance using the result of Equation (5.50) and (5.51) of Särkkä & Solin (2019) 9) as follows:
Pajo (A) -
Ae: AD omin
mmax
I
(12)
Omin
During the forward diffusion process, Gaussian noise is applied to perturb architecture, following the noise scheduling modeled by the prescribed SDE. order to employ the reverse process, which corresponds to the forward process, as a generative model, the score network so is trained to approximate the score function VA, log pe( (At).
B.3 SCORE NETWORK FOR NEURAL ARCHITECTURES
As proposed in Section 2.1 of the main paper, we introduce score network specifically designed for neural architectures. The distinctive aspect of our score network is the incorporation of positional embedding for nodes within directed acyelic graphs (DAGs), which allows for capturing the topological ordering of neural architectures. The input embodding and computation of this score network are as follows:
Emb. Embopa (v.) + Emb pos (v.) + Embrime (t), where i-th of V for i [N],
(13)
Se (A..)) = MLP (Hr), where Hº = Emb; H' = T (H'' .M) and H Hy] T_ (14)
Here, we denote Emb..p (v.) as the embedding
node (operation) v.. and Embrrme (v:)
embedding of time t. Additionally, we introduce the positional embedding Emb (v.) to incorporate positional information of the node into the input embeddings. Inspired by the previous work (Yan et al., 202 la),
we employ L transformer blocks (T) (Vaswani al., 2017) parameterize the score network. These blocks utilize an attention mask M N where represents the number of nodes in the architecture. approach, a pair nodes (operations) the architecture considered dependent directed edge is connected. The attention mask M in the transformer block is determined by the value of the adjacency matrix of a neural architecture A following the equation:
0, -10", if
if E, E,, =
M.,
(15)
Fach Transformer block (T) consists of Thead attention heads. The computation of the 1-th Transformer block is described follows:
H Wi qp: K, H'-' Vp H w!
(16)
12K7T + +M) Vo Vdh.
H softmax
(17)
H' concatenate
(it), H/s ............... Hmmee)
(18)
H' ReLU (H'' bi) W + b¹
(19)
where the size of H' and H is N d, and N x (da. (da./Theu)), respectively. Additionally, in the attention operation of the p-th head, denote Qp, Kp, and Vp as the "Query", "Key", and "Value" matrices, respectively. Moreover, we utilize Wi and W to represent the weights in the feed-forward layer.
B.4 DATASET-AWARY PREDICTOR IN DIFFUSIONNAG
To ensure that the predictor is dataset-aware, we extend its design by incorporating not only the architecture encoder hut also dataset encoder. In this section, we provide detailed explanation of this dataset-aware predictor configuration utilized in Section 3.1 of the main paper.
15

---

Published as a conference paper at ICLR 2024
Neural Architecture Encoding For encoding neural architectures, we employ the DiGCN layer (Wen al., 2020), which is a modified version of Graph Convolutional Networks (GCNs) specifically designed for directed graphs. This layer specifically designed to effectively capture structural information and dependencies within the architecture. To encode the architecture A, represented as (V. E) E IR" x R NN*N we first obtain the normalized adjacency matrix & by adding an identity matrix (representing the self-cycles) to the original matrix& & and normalizing it. Subsequently, to enable bidirectional information flow, we utilize two GCN layers. One GCN layer utilizes the normalized adjacency matrix propagate information the "forward" direction. while the other GCN layer uses & to propagate information the "reverse" direction. The outputs from these two GCN layers are then averaged. In the (-th layer of the architecture encoding module, the computation is performed as follows:
H' = ReLU (!!" + + ReLU (e" "
(20)
where H = V and W and w are trainable weight matrices. By stacking multiple layers, can effectively learn high-quality representations directed graphs. After applying the graph convolutional layers, we utilize average pooling on the architecture representations obtained from the final layer to aggregate information across the graph. Finally, one or more fully connected layers can be attached to obtain the latent vector ZA of the neural architecture A.
Dataset Encoding Following Lee et al. (2021a), we employ a dataset encoder based on Set Transformer (Lee et al., 2019) to accurately capture the characteristics of the target dataset. This dataset encoder is specifically designed to process imput sets of varying sizes and output fixed size latent code, denoted æp, that effectively encapsulates the information within the dataset. The dataset encoder consists of two stacked modules that are commutation-invariann
meaning output the dataset encoder is always the same regardless order of elements in the input set. Additionally, these modules leverage attention- based learnable parameters. The lower-level module, known as the "intra-class" encoder, captures class prototypes that represent each class information present in the dataset. On the other hand, the higher-level module, referred to the "inter-class" encoder, considers the relationships between these class prototypes and aggregates them into a latent vector ZD. The utilization of this hierarchical dataset encoder enables us to effectively model high-order intcractions among the instances within the dataset, thereby allowing us to capture and summarize crucial information about the dataset.
Ultimately, the encoded representation of the dataset zD is combined with the encoded architecture representation #A through MLP layers. This integration enables the predictor to make conditional predictions by considering both the architecture and the dataset information.
B.5 TRANSFERABLE TASK-GUIDED NEURAL ARCHITECTURE GENERATION
We aim to design a diffusion-basee transferable NAG method that can generate optimal neural architectures for unseen datasets. In order to achieve this, the surrogate model needs to be conditioned on a dataset D (a.k.a. "dataset-aware" (Lee et al., 202 la: Shala et al., 2023). For the surrogate model to be dataset-aware, we train it over the task distribution p(T) utilizing meta-dataset S consisting of (datasct, architecture, accuracy) triplets. To be precise, we define the meta-dataset as S = {(A (6) D consisting of K tasks, where each task is created by randomly sampling architecture A (8) along with their corresponding accuracy yi on the dataset D,, We train the dataset-aware accuracy surrogate model using S hy minimizing the following MSE loss function, L:
Φ² E arg min
Σ 1
(21)
An important aspect is that this surrogate model only requires one-time training phase since it is trained over the task distribution. Thereby, the surrogate model can make accurate predictions even for unseen datasets. After one-time training, we utilize this meta-learned datasct-aware accuracy surrogate model for (D, Ar) as a guiding surrogate. To be precise, we integrate the -awaaeet-wware surrogate model fa- (D, A.) trained with the objective function (Equation (21)) into the conditional generative process (Equation (9)). which enables us to control the neural architecture generation process for an unseen task, including unseen datasets with the following generation process:
dA, = ((((A)) - o8 [e[80-(AA))) (A,,t) + ke V log for (v) Ab)} di g.d..
(22)
16

---

Published as a conference paper at ICLR 2024
Table 5: Statisties of NAS-Bench-201 and MobileNetV3 Search Spaces used the experiments.
Search Space
Number Training Architectures
Number of nodes
Number operation types
NAS-Bench-201 MobileNetV3
15,625 500,000
7
20
9
C EXPERIMENTAL DETAILS
C.I SEARCH SPACE
As explained in Scction 2 of the main paper, the architecture A consisting of N nodes is defined by its operator type matrix E R and upper triangular adjacency matrix & E RNNN Thus, A can be represented as (V.E) E R NNN x IR N where F denotes the number predefined operator sets. Detailed statistics, including the number of nodes (N) and the number of operation types (F) for each search space utilized in our experiments (excluding Appendix D.3), are provided in Table 5. In Appendix D.3, the only difference is that we train the score network using subset of 7,812 architectures randomly selected from the NAS-Bon Bench-201 search space. This subset is selected due to utilizing the Novelty metric as part of our evaluation process. Further elaboration on the search space is presented below.
NAS-Bench-201 search space consists of cell-based neural architectures represented by directed acyelic graphs (DAGs). Each cell in this search space originally consists of 4 nodes and 6 edges. The edges offer 5 operation can- didates, including zeroize, skip connection. 1-by-1 convolution: 3-by-3 convolutin ion, and 3-by-3 average poo
Consequentl, there are a total of 15.626 possible architectures within this search space. To utilize the NAS- S--Bench-221 search space, we convert the original cell representation into a new graph representation. In the new graph, cach node represents an operation (layer), and the edges represent layer connections. After the conversion, each cell contains eight nodes and seven operation types for cach node. An additional input and output operation are added to the existing set of operations. To represent the neural architectures in the NAS-Bench-201 search space, we adopt an 8x7 operator type matrix (V) and an 8x8 upper triangle adjacency matrix (E). Furthermore, the macro skeleton is constructed by stacking various components, including one stem cell, three stages consisting of 5 repeated cells each, residual blocks (He et al., 2016) positioned between the stages, and final classification layer. The final classification layer consists of an average pooling layer and fully connected layer with a softmax function. The stem cell, which serves as the initial building block, comprises a 3-by-3 convol u on with 16 output channels, followed by a hatch normalization layer. Fach cell within the three stages has a varying number of output channels: 16, 32, and 64, respectively. The intermediate residual blocks feature convolution layers with stride of 2, enabling dowa-sampling.
MobileNetV3 search space is designed as a layer-wiss wise space, where each building block incorporates MBConvs. squeeze and exci a on Hu al. (2018), and modi fi ed swi s' non i n ty to create a more efficient neural network. The search space consists of 5 stages, and within each stage, the number of building blocks varies from 2 to 4. As result, the maximum number of layers possible in MobileNetV3 is calculated as 5 x = 20. For each building block, the kernel size be chosen from the set (3.5, 7).. and the expansion ratio can be chosen from (3. 4, 63. This implies that there are total of 3 x = 9 possible operation types available for each layer. To represent the neural architectures in the MobileNetV3 search space, we utilize a 20x9 9 operator type matrix (V) and a 20x 20 upper triangle adjacency matrix (E), Furthermore, the search space contains approximately 10 architectures, reflecting the extensive range of choices available.
C.2 BASELINES
this section, we provide the description various Neural Architecture Search (NAS) baselines in Section 3.1 and of the main paper.
Basic approach We begin by discussing the basic NAS baseline approaches. ResNet (He et al., 2016) is a popular deep learning architecture commonly used in computer vision tasks. Following Lee al. (2021a), we report results obtained from ResNet56. Random Search (RS. Bergstra & Bengio (2012)) is basic NAS approach that involves randomly sampling architectures from the search space and selecting the top- cop-performing one. Another basic NAS method we consider is REA (Real et al.. 2019), REA utilizes evolutionary aging with tournament selection as its search strategy. REINFORCE (Williams, 1992) is a reinforcement learning hased NAS method. this approach, the validation accuracy obtained after 12 training epochs is used as the reward signal.
We also compare our method with various one-shot methods, which have shown strong empirical performance for NAS. SETN (Dong & Yang, 2019a) focuses selectively sampling competitive child candidates by training model to evaluate the quality of these candidates based on their validation loss.
One-shot NAS
17

---

Published as a conference paper at ICLR 2024
Table 6: Search Time of One-shot NAS.
CIFAR-I0
CIFAR-100
Aircraft
Oxford-IIl IIT
Pets
Method
Accuracy (98)
GPU Time (s)
Accuracy (5)
GPU Time (6)
Accuracy (%)
GPU
Accuracy (%)
GPU
Time (s)
Time (s)
RSPS (Li & Talwalkar, 2019)
84.07
10200
52.31
18841
42.19
18697
12.91
3360
SETN (Dong & Yang. 90)
87. +000
30200
59.09+024
58808
44.84+396
18564
25. 25.17+188
8625
GDAS (Dong & Yang, 2019h)
23.6
25077
70.70aa0:
51580
53.522n44
18508
24.02
6965
PC-DARTS (Xu al., 2020)
93. 66+017
10395
66.64+234
19951
26.33+344
3524
25.31
2844
DrNA5 (Chen et ul., 2021)
20000
21760
73.51
34529
46.08
34529
26.73g361
6019
GDAS (Dong Yang, 2019b) a differentiable neural architecture sampler that utilizes the Gumbel-Softmax relaxation technique. GDAS sampler is designed to be trainable and optimized based on the validation loss after training the sampled architecture. This allows the sampler to be trained in an end-to-en to--tt-end fashion using gradient descent. PC-DARTS (Xu al., 2020) is gradient- based NAS method that improves efficiency by partially sampling channels. By reducing redundancy and incorporating edge normalization, PC-DARTS enables more efficient architecture search. DrNAS (Chen et al., 2021) is a differentiable architecture search method that formulates the search as a distribution learning problem, using Dirichlet distribution to model the architecture mixing weight. By optimizing the Dirichlet parameters with madienttbbsse optimization and introducing a progressive learning scheme to reduce memory consumption, DrNAS achieves improved generalization ability.
Bayesian Optimization based NAS A variety of Bayesian Optimization (BO) methods have heen proposed for NAS, and in our study, we conduct comprehensive comparison of these methods. One approach involves using a vanilla Gaussian Process (GP) surrogate (Snoek et al., 2012), where the Upper Confidence Bound (UCB) acquisition function is employed (GP-UCB). Additionally, we evaluate our method against other approaches such HEBO (Cowen-Rive Rivers et al., 2022), which is black-box Hyperparameter Optimization (HPO) method that addresses challenges like heteroscedasticity and non-stationarity through non-linear input and output warping and robust acquisition maximizers. Another BO-based NAS method, BA BANANAS (White et al., 2021a), utilizes an ensemble of fully-comnected neural networks as a surrogate and employs path encoding for neural architectures. BANANAS initiates new architecture search for each task. On the other hand, NASBOWL (Ru et al., 1021) combines the Deisseiler-Lehmee graph kernel with Gaussian process surrogate, allowing efficient exploration of high-dimensional search spaces.
TransferNAS and TransferNAG The key distinction between TransferNAS(NAG) methods and conven- tional NAS methods lies in their ability to leverage prior knowledge for rapid adaptation to unseen datasets. Unlike conventional NAS methods that start the architecture search from scratch for each new dataset, Transfer- NAS(NAG) methods utilize prior knowledge gained from previous tasks to accelerate the search process new datasets. By leveraging this prior knowledge, these methods can expedite the search process and potentially yield better performance on unseen datasets.
One of the TransferNAG methods called MctaD2A (Lee et al., 2021a), employs an amortized meta-learning approach to stochastically generate architectures from given dataset within a cross-modal latent space. When applied to and then selects the top architectures based on dataset-aware predictor. TNAS (Shala et al., 2023), another TransferNAS method, is a subsequent work of MetaD2A that aims at enhancing the dataset-aware predictor's adaptability to unseen test datasets utilizing BO with the deep-kernel GP. TNAS achieves this by making the deep kernel's output representation he conditioned on both the neural architecture and the characteristics of dataset. The deep kernel is meta-trained on the same training dataset used MetaD2A and DiffusionNAG (Ours). In the test phase, TNAS adapts to the test dataset using BO with the deep-kernel GP strategy. As described in Shala et al. (2023), TNAS starts the BO loop by selecting the top-5 architectures with the highest performance the meta-training set. However, this starting point can be detrimental when searching for an optimal architecture on the unseen target dataset since may start the BO loop from architectures that perform poorly on the target dataset. As a result, it can lead to long search time and high computational cost to obtain the target performance on the target dataset.
task, MetaD2A generates 500 candidate architectures by conditioning on the target dataset
C.3 SEARCH TIME OF ONE-SHOT NAS
As described in Table 2 of the main paper. we provide the search time for each dataset of the one-shot NAS methods in Table 6.
C.4 DATASETS
We evaluate our approach on four datasets following Lee et al. (2021a): CIFAR- 10 (Krizhevsky, 2009), CIFAR- 100 (Krizhevsky, 2009), Aireraft (Maji et al., 2013), and Oxford IIT Pets (Parkhi et al., 2012), as described in Section 3.2 of the main paper. In this section, we present a detailed description of each datasct used in
18

---

Published as a conference paper at ICLR 2024
study. 1) CIFAR-10 consists of 32x32 color images from 10 general object classes. The training set has 50,000 images (5,000 per class), and the test set has 10,000 images (1,000 per class). 2) CIFAR-I00 contains colored images from 100 fine-grained general object classes. Each class has 500 training images and 100 test images. 3) Aircraft is a fine-grained classification benchmark dataset with 10,000 images from 30 aircraft classes. All images are resized to 32x32. 4) Oxford-IIIT Pets is dataset for fine-grained classification, including 37 breeds of pets with approximately 200 instances per class. The dataset is divided into 85% for training and 15% for testing. All images are resized to 32x32. For CIFAR-10 and CIFAR-I06 we use the predefined splits from the NAS MAS-Bench-221 benchmark. For Aircraft and Oxford-111 Pets, we create random validation and test splits by dividing the test set into two equal-sized subsets.
C.5 TRAINING PIPELINE FOR NEURAL ARCHITECTURES
In this section, we provide the architecture training pipeline for both the JAS-Bench-25 and MobileNetV3
scarch spaces.
NAS-Bench-201
Following the training pipeline presented in Dong & Yang (2020b), we train each architec- ture using SGD with Nesterov momentum and employ the cross-entropy loss for 200 epochs. For regularization, we set the weight decay to 0.0005 and decay the learning rate from 0.1 to using a cosine annealing sched- ule (Loshchilov Hutter, 2016). We maintain consistency by utilizing the same set of hyperparameters across different datasets. Data augmentation techniques such as random flip with a probability of 0.5, random cropping of 32 32 patch with 4 pixels padding on each border, and normalization over RGB channels are applicd to each dataset.
MobileNetV3 In our training pipeline, we fine-tuned subnet of pretrained supernet from the MobileNetV3 search space the ImageNet IK dataset for CIFAR-IO. CIFAR-I00, Aircraft, and Oxford-IIIT Pets datasets, following the experimental setup by Cai et al. (2020). The process involved several steps. Firstly, we activated the subnet and its parameters pretrained on ImageNet 1K for each neural architecture. Then, we randomly initilized the fully-connected layers for the classifier, where the output size of the classifier is matched with the number of classes in the target dataset. Next, conducted fine- tuning the target dataset for 20 epochs. We follow the training settings of MetaD2A Lee al. (2021a) and NSGANetV2 Lu et al. (2020). Specifically, we utilized SGD optimization with cross-entropy loss, setting the weight decay to 0.0005. The learning rate decayed from 0.1 to using a cosine annealing schedule (I (Loshchilov & Hutter, 2016). We resized images as 224x224 pixels and applied dataset augmentations, including random horizontal flip, cutout DeVries & Taylor (2017) with length of 16. AutoAugment Cubuk al. (2018), and droppath Huang al. (2016) with rate 0.2. ensure a fair comparison. we applied the same training pipeline to all architectures obtained by our TransferNAG and the baseline NAS methods.
In Section 3.2 of the main paper, we leveraged a surrogate strategy to report the performance of architectures, which is inspired by the existing surrogate NAS benchmarks Li et al. (2021); Zela et al. (2022); Yan et al. (b021b) that utilize predictors estimate the performance of architectures within search space. This approach eliminates the need for exhaustively evaluating all architectures in the large search spaces. For evaluating the performance of each neural architecture, we utilized an accuracy predictor provided by Cai et al. (2020). They randomly sampled 16K subnets from the MobileNetV3 search space with diverse architectures. The accuracy of these subnets measured on validation set comprising 10K images randomly selected from the original training set of ImageNet 1K. By training an accuracy predictor using these [architecture, accuracy] pairs, Cai al. (2020) obtained model capable of predicting the accuracy of given architecture. Notably, Cai et al. (2020) demonstrated that the root-mean-squaee error (RMSE) between the predicted and estimated accuracy on the test set is as low as 0.21%. To ensure a fair comparison, we applied an identical training pipeline to all architectures obtained by our method and the baseline NAS methods.
C.6 ACQUISITION FUNCTION OPTIMIZATION STRATEGY IN BAYESIAN OPTIMIZATION BASED NAS
In Section 3.2 of the main paper, we address the limitation of existing Bayesian Optimization (BO) based NAS methods by incorporating our conditional architecture generative framework into them. Specifically, we replace the conventional acquisition optimization strategy with conditional generative framework, guided by the predictor. To facilitate understanding, we present an algorithm for General Bayesian Optimization NAS that utilizes an ensemble of neural predictors. Additionally, we demonstrate an algorithm for Bayesian Optinization NAS with DiffusionNAG, where our conditional generative framework replaces the conventional acquisition optimization strategy.
19

---

Published as a conference paper at ICLR 2024
Algorithm 1: General Bayesian Optimization NAS
Input: Search space A, dataset D, parameters Ro, N, c. acquisition function y, function h A - returning validation accuracy of an architecture A E ...
Draw no architectures A
(mo) uniformly from the search space AA and train them on D ..... (A (((A")))
construct B {(A(),
for 7 T0, ,., N do
Train an ensemble of M surrogate model (h J based on the current population B,;
tm E arg min Mhn Σ (b(A()) he (4(1) for l...., .M.
Generate set of candidate architectures from A using acquisition optimization strategy. For each candidate architecture A, evaluate the acquisition function (p(A) with (h... 1M Im= Select architecture A (x+1) which maximizes (pA)
Train architecture A (m+11) to get the accuracy h(A(mt)) (m+1)) and add the population B,:
Ba41 B, {(A (x+1) (m)))))))
Output: A" arg max ((A()))
Algorithm 2: Bayesian Optimization with DiffusionNAG
Input: Pre-trained score network 8g*, search space A, dataset D, parameters no, N, acquisition function y2, function h A IR returning validation accuracy of architecture E AA.
Draw no architectures A (1) construct Bmp {(A(), ())()))))............ ...... (A(")), h( h(A(()) (mm)))))
A (mo) uniformly from the search space A and train them on D
for n To.....,
do
Train an ensemble of M surrogate model (hv. Jmz based on the current population B,:
vi E arg min tim E (h(A hrm (4()) for I...., M.
M(A(1) = m=l
M Pup (A(1)
MN
(A)
M-
p(yl (1: D) = NWw (((A(((())) (1)), (A())))
/- Generate candidate architecture with the score network 80r. x/ S
Minimize Σ - (y: w.r. where y h(A()).
for ,,...,,........... C do
Draw a random noise A (n+i) from the prior distribution.
Generate architecture A (x++) with sg- by the reverse process:
d.A. (n+i)
[AA
(m)())
96 so. (A. (A) ,t) + A tott (+)))) di g.dw
S SU (UAAA(mt))
For each candidate architecture A S, evaluate (AA with (a) D). Select architecture A (n+1) which maximizes spA))
Train architecture A (m++1 to get the accuracy h(A(")))))) (n+1) and add it to the population B,;
Ba+1 B, {(A (m+1), hhAAAhAA (m(+)))))
Output: A arg max, mI.... (nA)))
C.7 IMPLEMETATION DETAILS
In this section, we provide a detailed description of the DiffusionNAG implementation and the hyperparameters used in our experiments.
Diffusion Process In Equation (11), which describes the diffusion process DiffusionNAG, we set the minimum diffusion variance, denoted dmin. to 0.1 and the maximum diffusion variance, denoted dmaxx to 5.0. These values determine the range of diffusion variances used during the diffusion process. Additionally, utilize 1000 diffusion steps. This means that during the diffusion process, iterate 1000 times progressively
20

---

Published as a conference paper at ICLR 2024
Table 7: Hyperparameter Setting of the Score Network DiffusionNAG on NAS-Bench-201 Search Space.
Hyperparameter
Value
The number of transformer blocks
12
The number of heads Thead
Hidden Dimension of feed-forward layers
128
Hidden Dimension of transformer blocks
64
Non-linearity function
Swish
Dropout rate
Learning rate Batch size Optimizer
2 x 10-6
256
Adam
81 in Adam optimizer
0.9
82 in Adam optimizer
0.999
Warmup period Gradient clipping
1000
1.0
Table 8: Hyperparameter Setting of the Dataset-aware Surrogate Model in DiffusionNAG on NAS-Bench- 201 Search Space.
Hyperparameter
Value
The number of DiGCN layers
Hidden Dimension of DiGCN layers
144
Hidden Dimension of dataset encoder
56
The number of instances each class
20
Hidden Dimension MI.P layer
32
Non-lincarity function
Swish
Dropout rate Learning rate Batch size Optimizer
0.1
x 10 3
256
Adam
8, in Adam optimizer in Adam optimizer Warmup period Gradient clipping
0.9
0.999
1000
1.0
generate and update architectures. Furthermore, we set the sampling epsilon e 1 X 10-5, This value used in the sampling step to ensure numerical stability (Song et al., 2021).
Score Network To train the score network so DiffusionNAG, we emplay the set of hyperparameters in Table 7. We conduct grid search to tune the hyperparameters. Specifically, we tune the number of transformer blocks from the (4, 8, 12), the number of heads from the set [4, 12), and the learning rate from the set (2 10-2 2 2 x 10-3 2 10 4,
$3. Additionally, we apply Exponential Moving Average (EMA) to the model parameters, which helps stabilize the training process and improve generalization.
Dataset-aware predictor To train the datasct-aware predictor fo in DiffusionNAG, we employ the set of hyperparameters in Table 8. We conduct grid search to tune the hyperparameters. Specifically, we tune the number of DiGCN layers from the set [1, 2, 4), the hidden dimension of DiGCN layers from the set (36, 72, 144, 288). the hidden dimension dataset encoder from the set (32, 64, 128, 256, 512), learning rate from the set [1 x 10 1,- x 10 x 10 ").
21

---

Published as a conference paper at ICLR 2024
N6201 - CIFAR100
NB201 - CIFAR100
NB201 - CIFARR00
73.5
73.
73.0
73.0
73.0
72.5
72.5
72.0
72.0
72.0
0 71.5
71.5
Sadeed Gan Doua
Grided Gen 1Dux
Gudded Get (Cur)
71.0
Milazion
71.0
Putation
71.0
Mazattan
70.5
Rancon
farder
70.5
-andamm
60
100
20
40
80
100
20
60 Trained Archs
100
Trained Archs
Trannd Archs
(a) EI
(b) ITS
(c) UCB
Figure 6: Experimental Results on Various Acquisition Functions (NB201). Ours consistently shows
superior or comparable performance compared with the hcuristic approaches on various acquisition functions. We rum experiments with 10 different random seeds.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 DISTRIBUTION OF GENERATED NEURAL ARCHITECTURES
In Scction 3.1 of the main paper, we evaluate the effectiveness of DiffusionNAG in generat- ing birg--eerorninng architectures. We generate 1,000 architectures using each method and an- alyze their distribution. The target datasets (D) are CIFARIO/CIFS 100 (Krizhevsky, 2009), for which have access the Oracle dis- tribution through the NAS-Bench-201 bench- mark (Dong & Yang. 2020b). Alongside the architecture distribution for CIFAR 100 depicted in Figure 2 of the main paper, we present the architecture distribution for CIFARIO in Fig- ure Similar to CIFARI00, DiffusionNAG produces greater number high-perf architectures compared to other baselines for the CIFARIO 10 dataset as well. This observation confirms that the three advantages discussed in Section 3.1 of the main paper also extend to CIFARIO.
CIFAR10
CIFARIO
Random
Uncond. + Sorting
MetaCZA
Cond. (Gum))
Counts
200
Counts
200
100
100
50
50
Accuracy
Accuracy
Figure 5: The Distribution of Generated Architectures for CIFARIO.
D.2 IMPROVING EXISTING BAYESIAN OPTIMIZATION NAS ACROSS VARIOUS ACQUISITION FUNCTIONS
Let the actual validation accuracy of an architecture A in the search space A he denoted as h(A), and let B. represent the set of trained models accumulated during the previous Bayesian Optimization (BO) loops. given by B. == (AA ((AA")))..... (A (m), (()). Moreover, we define the mean of predictions as a(A) = M hyg (A), where h i. (A) represents the prediction made by the m-th predictor from an ensemble of M predictors. Additionally, we define the standard deviation of predictions as ((A) =
2: (thez (A) (A)-2())) By utilizing an ensemble of predictors (h.. MM we can estimate the mean and --1
standard deviation of predictions for each architecture in the search space (Please refer to Algorithms and 2 in Appendix C.6).
Following Neiswanger et al. (2019) and White al. (2021a), we utilize the following estimates of acquisition functions for an input architecture A from the search space A::
PP (A E
[1 J A) Vinax]]
N (A(A), 0² (A)) dy
(23)
/ (P(A) - Vmmx)) N (B(AA))0(A)) dy
(PE = E [1 (A(A)= ymma (R(A) yimax)]
(24)
Umaa
PITs (A) = M(A)) i(A) ~ N ()A)))(((A))88))
(25)
(UCD (A) A(A) + (A))),
(26)
where Ymax = 1(AA).)))) = arg max h(4())).
We conduct the same experiment Figure 4 of the main paper in Figure 6 on the different search space, NB201. The experimental results of different acquisition function optimizaaion strategies across various acquisition
22

---

Published as a conference paper at ICLR 2024
functions in NB201 search space are presented in Figure 6. The results demonstrate that proposed acquisition optimization strategy with the conditional generative framework (Guided Gen (Ours) consistently shows superior or comparable performance compared to conventional acquisition optimization strategics. This improvement is observed across various acquisition functions, including Probability of Improvement (PI), which is used in Section 3.2 of the main paper. as well as Expected Improvement (EI,, Independent Thompson sampling (ITS), and Upper Confidence Bound (UCB) in the NB201 search space.
D.3 ADAPTATION ACROSS DIFFERENT OBJECTIVES
In Table 9, we demonstrate that DAG easily adapts to new tasks by swapping the task-specifee predictor in plug- and-play manner without retraining the score network. We train three predictors with different objectives: Clean accuracy, robust accuracy against an adversarial attack such as APGD (Croce & Hein, 2020) with perturbation magnitude of = 2. and robust
Table 9; Adaptation ability across differ- ent objectives.
Guidance f
Stats.
fócias
foase
Max
93,90
80.40
86.60
accuracy against glass Blur comuption. For each objective, randomly select 50 neural architectures from the NB201 and collect their corresponding test accuracy data on CIFARIO as training data (i.c., (architecture, corresponding test accuracy) pairs). Subsequently, we train each predictor using their respec- tive training data, denoted Sdewnn SHAAED: and femm With the guidance of these predictors, we generate pool of 20 architectures for each. For each generated architecture pool, we provide statistics of all objectives (i.e., test accuracies), including robust accuracies under the APGD attack and glass Blur corruption and clean accuracy from the NB201-based robustness benchmark (Jung et al., 2023).
Test Accuracy
Clean
Mean
93.17
69.24
74.23
APGD
Max
2.60
26.60
22,50
Mean
1.51
17.66
14.02
Max
37.10
53.50
56,90
Blur
Mcan
33.28
40.46
47.88
The maximum (max) and mean accuracies of the pool of architectures generated by using the predictor trained on the target objective higher than those obtained hy using predictors trained with other objectives. For example, architectures generated with the guidance of fecm.. have poor rohust performance on the APGD attack and Blur corruption, with a max accuracy of 2.6% and 37.10% % for each. In contrast, the guidance of feac and fors yield better architectures with max accuracy of 26.60% and 56.90% on the APGD attack and Blur corruption, respectively, demonstrating the superior adaptation ability of DiffusionNAG across various objectives.
D.4 ANALYSIS OF GENERATED NEURAL ARCHITECTURES ACROSS DIFFERENT OBJECTIVES
In the Adaptation Across Different Objectives part of Appendix D.3, we demonstrate the capability of our conditional generative framework casily adapt to new tasks. This adaptability is achieved by seamlessly swapping the predictor, trained with the specific target objective, without the need to retrain the score network.
By leveraging predictors trained with different objectives, such as clean accuracy, robust accuracy against adversarial attacks (e.g., APGD with perturbation magnitude e = 2), and robust accuracy against glass blur corruption (Blur), we generate pool of 20 neural architectures in the section. In this section, we analyze the architecture pool generated for each objective. Specifically, we examine the occurrence of cach operation type for each node in the architecture pool guided by each objective.
Interestingly, each generated architecture pool exhibits distinct characteristics, as illustrated in Figure 7. In Figure 7(a), we observe that the architecture pool guided by the Clean Accuracy objective has a higher pro- portion of 3-by-3 convolution operation types compared to the other architecture pools. Furthermore, Figure 7(b), we can observe that the architecture pool guided by the Robust Accuracy against APGD Attack objective exhibits distinct characteristics. Specifically, nodes 1, 2, and 3 are dominated by the zeroi ze, 3-by-3 average pooling, and skip connection operation types, while nodes 4, 5, and 6 are pre- dominantly occupied by the 3-by-3 average pooling operation type. Lastly, in Figure 7(c), depicting the architecture pool guided by the Robust Accuracy against Glass Blur Corruption objective, we observe that nodes 1, 2, and are predominantly occupied by the zeroi zc operation type. Based on our analysis, it intriguing to observe that the characteristics of the generated architectures vary depending on the objective that guides the architecture generation process.
D.5 ADAPTATION ACROSS DIFFERENT OBJECTIVES: CORRUPTIONS
In Table 9, we evaluate the adaptation abilities of DiffusionNAG to new tasks by swapping the task-specific predictor in plug-and-play manner without retraining the score network. In the Section, we conduct further experiments to explore broader range of corruptions introduced not only in glass blur but also in various corruptions in the NB201-bass robustness benchmark (Jung et al., 2023). We conduct experiments to find robust neural architectures for 15 diverse comuutions at severity level 2 in CIFAR- we demonstrated the capabilities of DiffusionNAAG to easily adapt to new tasks without retraining the score network. We achieved this by replacing task-specific predictors in plug-and-play manner. Specifically, we train clean predictor, ferm.) ). predict the clean accuracy of neural architectures using randomly sampled pairs of
Similar to Table 9,
23

---

Published as a conference paper at ICLR 2024
Node
Count
zer0kze
3x3 avg pool
3x3 com
avg. pool
skip connect
conw
1xt comu
zeroize
aug. pool
skip connect
1xt conw.
zoroize
ship connect
conw
zerozze
3x3 avg poot
skpp connact
3x3 come
zeroze
3x3 avg. poot skip
3x3 conw
3x3
skip
Operation Tyoe
Operetion Type
Operation Type
Operation
Operation
Operation
(a) Clean Accuracy
Node 1
Node
Node
Node 4
Node 5
Node
15
15
15
15
Count
10
10
10
10
10
10
5
5-
zeroize
3x3 avg. pool.
connect
3x3 conv.
3x3 ovg. pool
connect
3x3 conv.
1x1 conv.
zeroize
slip connect
conv.
1x1 conv.
zerpice
skip connect
1xl conv.
zeroiee
3x3 aeg pool
dei connect
3x3 conv.
conv.
zeroito
5x3 aeg pool
connect
3x3 conv.
1xl
avg,
3x3
Operation Type
Operation Type
Operation Type
Operation Type
Operation Type
Operation Type
(b) Robust Accuracy against APGD Attack
Node
Node
Node
Node 4
Node 5
Node
15
15
15
15
15
Count
10
zeroize
3x3 tvg pool.
connect
3x3 conv.
zeroize
3x3 avg. pook.
skip connect
3x3 cone.
cone
zeroize
3x3 nug. pook.
ship connect
cone.
1el conv
zetoze
skip connect
cone
1xl conv.
zerozze
3x3 aug pook
sMp connect
3x3 cone
cone.
3x3 avg. pook
3x3 conv
5x3 aug.
skip
Operetion Tyge
Operetion Type
Operation Type
Operation Type
Operation Type
Operation Type
(c) Robust Accuracy against Glass Blur Corruption
Figure 7: Analysis Generated Architectures across Different Objectives. analyze distribution of operation types for each node across the pool of 20 generated architectures. The architectures are generated by DiffusionNAGG, guided by surrogate models trained with different objectives such as Clean Accuracy, Robust Accuracy against APGD Attack, and Robust Accuracy against Glass Blur Corruption.
architectures and their corresponding CIFAR clean accuracy from the NB201-basee robustness benchmark. Afterward, we train the target corruption predictor, f to predict the robust accuracy of neural architectures under specific corruption types. We create the training dataset by including pairs of architectures and their respective robust accuracy for each corruption scenario from the NB201-based rohustness benehmark. With the guidance of the trained ferrrr generated pool of 20 architectures (Saran Similarly, with the guidance of
24

---

Published as a conference paper at ICLR 2024
Table 10: Adaptation Across Different Objectives: Corruptions.
Coruption Elastic
Type Fog
Predictor
Brightness
Contrast
Defocus
Frost
Gaussian
f clean
89.3
58.9
64.6
61.0
74.7
60.2
30.5
cooempption
89.4
59.6
64.9
61.4
74.4
58.9
49.6
Impluse
Jpeg
Motion
Pixelate
Shot
Snow
Zoom
Average
S deleen f commuption
61.6
43.7
71.7
39.7
70.4
44.9
49.2
71.2
62.6
the task-specific predictors (fommppii)) ). we generate another pool of 20 architectures for each target corruption (Scorppiin) ) Subsequently, we retrieve the robust accuracy on both førm -guided and fo -guided architecture pools (Setenn and Sueerption)) from the NB201 -based robustness benchmark for cach coruuttion scenario.
The maximum accuracies the architecture pool generated using the predictor focaupi trained for the target corruption surpass those obtained using the accuracy predictor fec trained for clean accuracy. For 13 out of the 15 corruptions, excluding fog and frost corruption, the maximum accuracy of the architecture pool guided by the foumppio predictor exceeds that of the pool guided by ferm.. The average maximum accuracy increased by 3.6%, reaching 59. 0% and 62.6 6%, respectively.
25


Reviews:
[{"review_id": "oFfde5b7aj", "reviewer": "Reviewer_qikk", "summary": "The researchers are introducing a new approach called DiffusionNAG to overcome limitations in Neural Architecture Search (NAS) methods. Instead of traditional NAS, they propose a conditional Neural Architecture Generation (NAG) framework based on diffusion models. This framework, utilizing graph diffusion models, allows for efficient generation of task-optimal neural architectures with desired properties. DiffusionNAG incorporates parameterized predictors to guide the generation process, making it more efficient than previous NAS methods. The effectiveness of DiffusionNAG is demonstrated through experiments in Transferable NAS and Bayesian Optimization-based NAS scenarios.", "questions": "Please address questions in \"Weaknesses\".", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work treats neural architectures as directed graphs and introducing a graph diffusion model. It enables flexible generation of task-optimal architectures guided by parameterized predictors, surpassing the efficiency of traditional NAS methods.\n\n2. DiffusionNAG outperforms baselines in Transferable NAS, achieving up to 20x speedups. Additionally, when integrated into a BO-based algorithm, it outperforms existing approaches, especially in the extensive MobileNetV3 search space on the ImageNet 1K dataset.", "weaknesses": "1. This work only reports results on two NAS benchmark search spaces. It is unclear whether it can achieve similar performance on much larger search spaces, such as the ones proposed in [1] and [2], which are widely used in previous works.\n\n2. Fig. 3 and 4 compare results on existing AO strategies and various acquisition functions. However, there is no comparison to other methods on ImageNet, as shown in Tab. 1 and 2.\n\n3. In Tab. 5, the authors include robust accuracy against the APGD attack, which is good. However, the corruption evaluation is insufficient, considering only glass blur. It would be better to include more comprehensive corruptions, such as those in ImageNet-C or CIFAR-10-C.\n\n[1] Learning Transferable Architectures for Scalable Image Recognition, CVPR 2018.\n\n[2] DARTS: Differentiable Architecture Search, ICLR 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4glZDOiQaQ", "reviewer": "Reviewer_qPhs", "summary": "The authors propose a diffusion based model for neural architecture generation. By using the guidance from the predictor, DiffusionNAG can generate task-specific architectures. They verify the effectiveness of DiffusionNAG on Transferable NAS and Bayesian Optimization-based NAS.", "questions": "Please see the Weaknesses part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-motivated. Previous work only uses the diffusion model to model undirected graphs for neural architecture generation. Using the diffusion model for modeling the directed graph is needed. Moreover, classifier guidance is well suited for the Transferable NAS and substitute step 3) in BO-based NAS approaches, where the dataset and the pre-trained classifier are given as the setup.", "weaknesses": "1.\"Whether diffusion model for neural architecture generation is more efficient than Mutation + Random\" is a good question. It is important to ask \"when they are better\".\n\n* The strength of the predictor guidance is an important hyperparameter. In this paper, a good performance can be achieved by cross-validation. Could the author provide some ablation studies of how this hyperparameter affects the performance? How could we effectively define the search range for cross-validation?\n\n* Will using the diffusion model for neural architecture generation have some model collapse phenomenon? Does the Diffusion model only generate some similar architecture, and achieve the better \"worse case accuracy\" by sacrificing the diversity? If that is the case, any explore and exploit framework for it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WkQXVxYy6v", "reviewer": "Reviewer_ZVno", "summary": "The paper proposes DiffusionNAG, a novel conditional Neural Architecture Generation framework based on diffusion models, which considers the neural architectures as directed graphs. The proposed method is shown to be significantly more efficient than previous NAS schemes. Empirically, the authors demonstrate it can provide speedups up to 20x, while being superior in performance.\n\nI would also like to note that I am no expert in Neural Architecture Search, which could affect my judgments below.", "questions": "* I would not say that I have the best time understanding the parts about \"Score Network for Neural Architectures\": what exactly is being pursued here? if our adjacency matrix is defined to be upper triangular, should it not already be a DAG? What is the goal of the proposed positional embedding? A bit more explanation is welcomed here.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed method, which uses diffusion models to essentially generate the entire search space and then utilizes the gradient of property predictor to guide the generation towards the more potent candidate, is novel to my knowledge.\n* Diffusion model sampling is essentially treated as a search algorithm, which enables more aggressive searches that result in faster procedures.\n* The proposed framework could utilize different predictors dependent on the tasks, so the base diffusion model only needs to be trained once.\n* The paper also explores diffusion models suitable for generating directed acyclic graphs, which could be useful for other fields as well.\n* Empirically, the DiffusionNAG is shown to be scalable and effective, providing speedups while improving the performances.\n* The paper is well-written and easy to follow for the most parts.", "weaknesses": "* I think some parts could be explained a bit more, even if they are from prior works, e.g. the meta-learned dataset-aware predictor in Sec.2.3. This, along with the diffusion background, could be covered in a dedicated background section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qdItaU63uz", "reviewer": "Reviewer_EDN5", "summary": "The paper proposes a method to perform neural architecture search using diffusion models. More precisely, they model the neural architecture as a stochastic process and model it using diffusion. Then at inference with appropriate initialization the reverse process samples a new neural architecture to train a model.", "questions": "Can such an approach be used to find optimal architectures for diffusion models? Instead of the standard classification problems. \nOne can start with discrete NAR transformers for instance (like MaskGIT) which can model the NAS process and token predictions, and then the produced model will be used for generation which is another variant of cross entropy minimization problem. Since the foundation models are increasing in size day by day it will be interesting to see if their training methods can be used to find optimal sparse architectures.\n\n[1]MaskGIT: Masked Generative Image Transformer (https://arxiv.org/abs/2202.04200)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The use of diffusion models to neural architecture search is in an interesting idea. \n2. Modeling NAS as a generative problem enables creating multiple architectures starting with random noise, compared to exhaustively searching the entire space. This enables reducing the NAS cost. \n3. Empirically the method outperforms the baselines considered in the paper.", "weaknesses": "1. Even though the motivation to the use of diffusion models to neural architecture search is interesting, it seems more appropriate to use discrete diffusion models for discrete random variables instead of using a continuous variables. For instance [1] is an interesting approach which seems to be more correlated to the desired problem instead of continuous diffusion. \n2. The use of diffusion models for NAS seems to be very incremental, as the proposed approach is a direct application of the standard available diffusion based approaches. What are some of the major hurdles faced by the authors in making this work for NAS?\n3. For conditional NAG, the authors can consider convert A_t to A_0 using the appropriate equations depending on the choice of diffusion sampler used and then apply the pre-trained predictor. This can truly enable using a pre-trained predictor which is just a function of A_0 instead of A_t.\n\n[1]Vector Quantized Diffusion Model for Text-to-Image Synthesis(https://arxiv.org/abs/2111.14822)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)