Prior Knowledge Guided Neural Architecture Generation
Authors: Jingrong Xie
Han Ji
Yanan Sun
Venue: ICML 2025 poster (2025)
Keywords: neural architecture search
neural architecture generation
Primary area: deep_learning->everything_else

Abstract: Automated architecture design methods, especially neural architecture search, have attracted increasing attention. However, these methods naturally need to evaluate numerous candidate architectures during the search process, thus computationally extensive and time-consuming. In this paper, we propose a prior knowledge guided neural architecture generation method to generate high-performance architectures without any search and evaluation process. Specifically, in order to identify valuable prior knowledge for architecture generation, we first quantify the contribution of each component within an architecture to its overall performance. Subsequently, a diffusion model guided by prior knowledge is presented, which can easily generate high-performance architectures for different computation tasks. Extensive experiments on new search spaces demonstrate that our method achieves superior accuracy over state-of-the-art methods. For example, we only need $0.004$ GPU Days to generate architecture with $76.1\%$ top-1 accuracy on ImageNet and $97.56\%$ on CIFAR-10. Furthermore, we can find competitive architecture for more unseen search spaces, such as TransNAS-Bench-101 and NATS-Bench, which demonstrates the broad applicability of the proposed method.

Prior Knowledge Guided Neural Architecture Generation
Jingrong Xie
Han Ji
Yanan Sun
x
Abstract
Automated architecture design methods, espe- cially neural architecture search, have attracted increasing attention. However, these methods nat- urally need to evaluate numerous candidate ar- chitectures during the search process, thus com- putationally extensive and time-consuming. In this paper, we propose prior knowledge guided neural architecture generation method gener- ate perhhorrffrmmee architectures without any search and evaluation process. Specifically, in der to identify valuable prior knowledge for archi- tecture generation, we first quantify the contribu- tion of each component within an architecture to its overall performance. Subsequently, a diffusion model guided by prior knowledge presented, which can easily generate high-performance chitectures for different computation tasks. Ex- tensive experiments new search spaces demon- strate that our method achieves superior accuracy over state-of- -fehe-a--t methods. For example, we only need 0.004 GPU Days to generate architec- ture with 76. % top-1 accuracy on ImageNet and 97.56% on CIFAR-10 Furthermore, we can find competitive architecture for more unseen search spaces, such as MeansNAS-Bench-I 101 and NATS- Bench, which demonstrates the broad applicabil- ity of the proposed method.
PG-NAG(our)
Better
76.00
S-DARTSUle
LagleLLooo(aan 2023)
75.75
BaLeNAS TriZhang et 2023)
ImagaNett accurscy
SWD NASINO of aL, 2024)
75.50
TE-NASICHu al M021)
75.25
EG al. 2024)
PIANT-T(La 2023)
FairNASICha et 2020)
75.00
dt 2020)
74.50
EoINAS/ZZou 2021)
0.1
0.3 Search Cost
0.5
Figure I. Search cost and performance comparison of our proposed PG-NAG method against the state-of-the-art NAS methods on ImageNet.
ation assessing each architecture to find the optimal one. However, the exhaustive search for an optimal architecture is computationally demanding due the vast search space, which often contains over 10 possible architectures (Cai et al., 2019). Furthermore, evaluating each candidate archi- tecture requires training from scratch on a dataset to obtain accuracy, leading to significant time and resource costs.
To alleviate these costs, several methods have been intro- duced to accelerate architecture evaluation, such as per- formance predictors (Ji et al., 2024) and zero-shot meth- ods (Peng et al., 2024). These methods aim predict the performance of architectures without training from scratch. However, even with faster evaluations, current NAS meth- ods still need to evaluate architectures individually, which remains time-intensive, especially with a large number of candidates. Therefore, minimizing computational costs re- mains a central challenge NAS research.
1. Introduction
Neural Architecture Search (NAS) is an effective approach to automatically designing neural architectures and has seen success in diverse tasks. NAS generally involves three main components (Elsken et al., 2019); search space defining pos- sible architectures, search strategy exploring this vast space to identify promising candidates, and performance evalu-
Department of Computer Science Sichuan Univer- sity, Chengdu, China. Correspondence to: Yanan Sun <ysun scu.edu.en>.
This leads to growing interest in the Neural Architecture Generation (NAG) paradigm, which reduces costs by avoid- ing the exhaustive search approach and instead focuses on directly generating herh-peeformance architectures. For in- stance, AutoBuild (Mills et al., 2024) trains a predictor to rank the performance of the modules in architecture, thus
Proceedings of the 4 International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Prior Knowledge Guided Neural Architecture Generation
constructing architecture by combining high-valued mod- ules in each layer. DiffusionNAG (An al., 2024) trains diffusion model to generate architectures and utilizes a pre- dictor to guide the generative direction to high-performance architectures, thus achieving significant acceleration and improving search performance.
Experimental results demonstrate that PG-NAG suc- cessfully generates architectures across multiple un- seen search spaces, achieving superior performance compared to state-of-the-ard NAS and NAG methods. The generated architectures not only achieve 97.56% accuracy on CIFAR- 10, but also demonstrate 76. top-1 accuracy when transferred to ImageNet.
However, these NAG methods still rely on predictors to guide the generation, as they lack the methods investigate the search space toward optimal solutions systematically. Without such predictors, these models would randomly se- lect architecture components, resulting in inefficient explo- ration. Furthermore, training these predictors will introduce additional computational overhead.
2. Related Works
2.1. Neural Architecture Search (NAS)
NAS can discover the optimal architecture from a set of possible architectures automatically and can be mainly clas- sified into reinforcement Dearning-based NAS, evolutionary- based NAS, and gradient-baser based NAS (Elsken et al., 2019). However, regardless of which NAS algorithm is used, many architectures require being evaluated, resulting in high eval- uation costs. To alleviate this, various accelerating methods have been proposed, such as performance predictor (Lu et al., 2023) and zero-shot (Peng et al., 2024). In any case, they still need to evaluate massive architectures iteratively, PG-NAG bypasses this limitation and offers more efficient way to generate architectures automatically.
This paper aims at generating promising architectures with- out relying on additional predictors. Inspired by (Xiao al., 2022), try to utilize prior knowledge to analyze the im- portance of operations to improve the efficiency of NAG. We propose a novel framework to generate architectures with prior knowledge guidance rather than additional pre- dictors, entitled PG-NAG, A key consideration for PG-NAG ensuring the quality of prior knowledge. This quality contains two aspects: one is that prior knowledge should be derived from reliable sources, and the other is that it should accurately reflect the contributions of various components within the architecture. PG-NAG not only provides rele- vant and precise guidance but also allows us to effectively generate architectures across various search spaces without additional training.
2.2. Diffusion Models
Diffusion models are probabilistic generative models that progressively destroy data by injecting noise, then learn to reverse this process for sample generation (Yang et al., 2023a). Since the initial diffusion model is designed to learn the data distribution at noise level, it is often modified for adapting to various controlled generation scenarios. Pre- vious studies have already made preliminary attempts to control generation. For example, Classifier-free guidance methods (Ho & Salimans, 2022) enhance the quality and diversity of generated samples by mixing score estimates from conditional and unconditional diffusion models. Fur- thermore, the research ( (Vignac al., 2022) has shown that noise distributions closely resembling the sample data dis- tribution can lead to improved performance. Building on these findings, PG-NAG employs prior knowledge to modify the noise distribution, thereby better guiding the diffusion model in generating high-performance architectures.
As shown in Figure 1, PG-NAG can generate promising architectures and significantly reduce computational com- plexity. To achieve this, our method learns the key compo- nents of peeefrreffmmccce architectures from benchmark datasets to form prior knowledge, which then serves as a guiding condition to construct the diffusion model. In the meantime, architectures from benchmark datasets eliminate the cost of training architectures obtain accuracy. Notably, PG-NAG achieves this efficiency with minimal computa- tional resources, requiring only 0.004 GPU days to generate well-performing architectures.
We propose novel method for neural architecture generation, i.e., PG-NAG, relying on learning compo- nent knowledge from high-performance architectures sampled from benchmark datasets. Significantly differ- ent from traditional automated design approaches, PG- NAG generates high-performant architectures with- out iterative search and additional training costs.
3. Methods
In this section, we present PG-NAG, a method designed to generate high-performance architectures for various tasks by leveraging a small selection of pre-existing high- performance architectures from benchmark datasets. We first present the overview. Then, we show how to obtain prior knowledge and the generation progress based on the prior knowledge, as illustrated in Figure
We design new diffusion model to generate archi- tectures, which can leverage prior knowledge to guide the generation of architectures. This helps generate architectures that meet the needs of different tasks.
2

---

Prior Knowledge Guided Neural Architecture Generation
Prepare prior knowledge
Generate architecture
Prior knowledge P
Shapley value analyze
Pt
validation accuracy
Pretrained generation model
Unseen dataset
well-trained architecture
operation
connect
Pretrain generation model
0.2
Forward diffusion process (Architectuee-Noisse
0.7
0.9
Ao
AT
(xxe|xt-1,
0.4
0.8
Prior knowledge P
P
Ao
AT
Reverse generative process A rchitecture
(a) Generation Model Preparation
(b) Generate A rchitecture
Figure 2. An illustration of PG-NAG, which contains the preparation and utilization of architecture generation model, where the preparation includes preparation of prior knowledge and pretrain model.
Algorithm 1 Overall Framework of PG-NAG
and connection in architectures. PG-NAG then proposes diffusion model to newly generate high-performance ar- chitectures under the guidance of prior knowledge K,, en- suring that generation process is both informed and efficient, without further evaluations. As illustrated in Al- gorithm 1, each architecture A E A is represented graph GR (O,C), O for,.... ,,, } is set of nodes, C (cy (i,j) C |O] [O]} is set of edges, represent- ing operations and connections between operations, respec- tively.
Input: High--peffommance architectures A from benchmarks, parameter / for diffusion model, the number of time steps T Output: The resulting generated high-performance archi- tecture *
1: Let architectures represent as graphs, each be represented as GR (O,
2: Obtain prior knowledge K. = Shapley value (GR
(O,C)) as Equation (3) and (4)
3: Sample t Uniform(,.....
4: Sample e ~ (N,0
Subsequently, we provide a explanation of the proposed diffusion model. Considering this task in the Denoising Diffusion Probabilistic Model (DDPM) (Ho al., 2020), small Gaussian noise is gradually injected into data as Markov chain. This process ultimately converges into white noise distribution after I' iterations, leading to the following forward diffusion process:
5: Take gradient step on A, Po(At-1|A, K,, as Equa-
tion (5)
6: Sample Ar (AT) 10.
7: for 1trt to do
8: Compute (AT, A,t) from e(Ar, A,t)
9: Sample A,-1 ~ 4 Ap| At-1, Kc, 14) as Equation (6)
10: end for
return -"
((1TT Ao, KC) = II ((A,A)
(1)
3.1. Overview
where Ag..... Ar denote sequence of latent variables with the same dimension, t is the index of diffusion steps, as the data Ao q(An).
In PG-NAG, to avoid the computational cost of evaluating pre-existing architectures, we utilize subset of architec- tures A from NAS benchmarks, which contains many archi- tectures and corresponding accuracy values. We first obtain prior knowledge K. from these architectures using Shapley values, which quantify the contribution of each operation
In the reverse generative process, the noise is gradually re- moved from the data to recover architecture from the white noise. This is achieved by iteratively applying a care- fully designed reverse process that targets the original data
3

---

Prior Knowledge Guided Neural Architecture Generation
distribution. Specifically, at each step I, we aim approx- imate the conditional distribution po( (o,,-1|A4, K.), which represents the probability of the architecture before noise has been added. This distribution is calculated under the condition of the current noisy state and the prior knowledge K. Thus, this process is essentially a data generation process as a reverse dynamics of the above diffusion process. The reverse process with a neural network parameterized by is represented by Equation (2).
performance gains V(N) to each operation and one-hop subgraphs in N, while N represents the number of opera- tions and subgraphs in this architecture. Therefore, a set of individual operations No and set of individual subgraphs Ng can be modeled as players in the cooperative game, where all players work together towards the architecture performance V(N), with N = No + Ng. A value func- tion V maps each subset of players S S NN to a real value ((S) which represents the expected payoff that the players can obtain by cooperation. The Shapley value & (P) and &y (P) for an operation o² and a one-hop subgraph g can be computed as follows, respectively:
(o AT) = II po (A,)
During this reverse process, the start of generating archi- tectures (A(r) is set as a standard Gaussian and then is used as the prior distribution, i.e., from which new sam- ples are generated. For an architecture, it can be gener- ated by drawing chaotic statuses Ar from (Ar), and then iteratively refined through the reverse Markov kernels pe (A A.
1
v(SU - -(())
(P(P)
(3)
Nool
[Nol-1
SCN,
[S] VISU {g'}) V(S)
df(P) = N
Σ SEN
(4)
IIN
3.2. Prior Knowledge Learning
After the validation accuracy is used as the value func- tion V to measure the architecture performance, the perfor- mance of the whole architecture is the sum of contributions of operations and one-hop subgraphs, i.e., Eo'ENo % + Σg gN &g = V(N). If the operation does not influence performance, whether added or removed, its contribution is considered zero and we define % O. For instance, the zeroize operation in NASSB and zero operation in DARTS have no effect on the final performance. Further- more, if two different operations or one-hop subgraphs have the same impact on the performance of architecture, they are assigned with equal contributions. Based on these proper- ties, the Shapley value can quantify individual contributions uniquely.
To effectively guide the generation process, PG-NAG lever- ages prior knowledge obtained from high- high-performance ar- chitectures in benchmark datasets, eliminating the need for additional training. The process begins by sampling top-k architectures from popular benchmarks to study the compo- nents in -eghpperformacc architectures. With the Shapley value evaluation, we quantify the contribution of each oper- ation and connection to overall architecture performance.
Due to the interdependence between operations and con- nections within an architecture, the Shapley values reflect these interactions, treating the architecture as coopera- tive game. Specifically, we evaluate the contributions of two components: operations and the connections between operations. The connections are represented as edges. In architecture, edges are directional to represent the flow of information, and learning the interactions of operations is critical to obtain important information contained in the architecture (Chen et al., 2021b). While connections and operations significantly impact architecture performance, their individual contributions alone do not guarantee high- performance architectures. Instead, we focus on learning ooe-hop subgraphs for each node to effectively capture the connections and their architectural relevance. The Shapley value measures the average marginal contribution of oper- ations and one-hop subgraphs to the overall performance of an architecture. It is caleulated by evaluating the per- formance disparity between the complete architecture and those without specific operations or subgraphs.
Based on the above calculation, we can quantify the specific contribution of operations and connections to the perfor- mance of architectures. This allows us to pinpoint the opera- tions and connections that are important in the generation of segh-performance architectures. With a clear understanding of these critical elements, we are better equipped to generate high-performance architectures.
3.3. Diffusion Model with Prior Knowledge
To generate high-performance architectures, we design a conditional diffusion model that incorporates prior knowl- edge for guidance. The model operates as latent variable model with two processes: the forward diffusion process and the reverse generative process. In the forward process, Gaussian noise gradually injects the input architecture into a white noise distribution over T iterations. The reverse process then removes this noise step-by- recovering the original architecture. By leveraging prior knowledge, the
Specifically, each architecture A is represented as a graph, which contains operations of e O and one-hop subgraphs g" E 9. The Shapley value is utilized to distribute the total
4

---

Prior Knowledge Guided Neural Architecture Generation
model can approximate the architecture distribution more effectively than pure Gaussian prior. During the reverse generative process of the diffusion model, the prior knowl- edge obtained from the Shapley value evaluation can guide generated architectures that are closer to the original archi- tecture.
graphs, respectively. After completing the forward process of the diffusion model, the diffusion model can probabilis- tically select key operations and one-hop graphs around the selected operations during the reverse process. In this way, the proposed diffusion model can generate efficient architectures optimized for specific task requirements.
We first formalize the prior knowledge to clearly demon- strate the guiding role of prior knowledge in the conditional diffusion model. The prior knowledge is formalized as
4. Experiments
In this section, we evaluate PG-NAG five new and different search spaces, including DARTS (Liu et al., 2018), NATS-Bench (Dong et al., 2021), TransNAS-Bench 101 (Duan et al., 2021), NAS-Bench-ASSS (Mehrotra et al., 2021), and NAS-Bench-NL.F (Klyuchníkov et al., 2022). We first explain the experimental settings in Section 4.1, then compare PG-NAG with state-of-the-ard algorithms in Section 4.2, and finally analyze the generated architectures and perform an ablation study in Sections 4.3 and 4.4, re- spectively. More experimental results on other search spaces can be found in Appendix C. and Appendix C.2.
set of data-dependent priors K = Oi.ii G Ei.T): which includes operations and connections that have im- portant contribution to high performance, N denotes the number of prior knowledge components. O₂ is prior opera- tions and G is the one-hop subgraph around operation O in high performance architectures, and Σ1.N is the prior covariance matrix.
Next, we will describe how the diffusion process and gener- ative process will be adjusted under the guidance of prior knowledge. Our diffusion model aims build the distribu- tion (A) Po (A, the process of operations O and one-hop subgraph G are similar, we only focus the deriva- tion on the diffusion and generative process of operations O. For connections, we use the same method to get the generated ones.
4.1. Experimental Settings
To generate high high-performance architectures for new search spaces and unseen tasks, we first learn prior knowledge from three popular benchmarks. Specifically, we select top-20 high gegh-performance architectures from each of the three popular benchmarks, which are NAS-Bench-100 (Ying et al., 2019), NAS-Bench-30M (Zela et al., 2020). Please note that NAS- Bench-101 101 and NAS-Bench B-S--Becch-211 contain architecture with accuracy. and NAS-Bench-301 contains accuracies of archi- tectures in DARTS search space by using a proxy model to evaluate architectures without training. Therefore, we can learn the contribution of components to architectures across different search spaces without addi- tional training for architectures. To implement the PG-NAG method, we integrate graph convolution network with 64 hidden layers to extract operation features in our diffusion model. Additionally, a three-layer MLP is utilized to cap- ture knowledge of connections in architectures. To validate the effectiveness of PG-NAG in five new search spaces, we design five architectures for each search space. Subse- quently, depending the convention of the different search spaces, we select the highest value or the average value of these five architectures as the validation result. For different search spaces, we used different methods to compare with PG-NAG, including different search methods NAS and efficient evaluation methods in NAS. More details about experimental implementation are in Appendix B.
We follow the definition and notations related to the noise schedule ot, Be, ot, & in (Ho & Salimans, 2022). With prior knowledge guidance, the forward and reverse process can be defined as follows:
(Dong & Yang, 2020b), and
g(A,1A,,,, NA N N (Aeni
A Ar- Br Σ)
(5)
Po(A
IIII N
A1-1,ni
n=
(6)
pr (Ar, Aa,
,e
where 14 (Ar. Au,n) var(1-ā.-) 1-01
Ar,
vo Ao, T-6i
The prior distribution in the forward diffusion process can be expressed
NA N
P(A = II II N AT: Σ 1=1 1=1
As illustrated in the above equations, the proposed diffusion model learns the knowledge of architecture design based on prior knowledge. To enable the learned prior knowl- edge to be effectively utilized by the diffusion model, we construct a graph convolutional network and multilayer per- ceptron (MIP) to extract features of operations and one-hop
4.2. Experimental Results of Search Spaces
Results on DARTS. Table presents the results of architec- tures generated by PG-NAG within the extensive DARTS
5

---

Prior Knowledge Guided Neural Architecture Generation
Table 1. Performance comparison with state-of-the-art architectures ImageNet and CIFAR-IO DARTS search space.
COST (GPU-DAY)
TEST ERROR (%)
PARAMS
FLOPs
METHODS
IMAGENET
CIFAR-10
(M)
(M)
DARTS (LIU ET AL., 2018)
73.3
97. 97.24++009
4.7
574
SNAS (XIE AL., 2018)
1.5
72.7
97.15+0.02
4.3
474
P-DARTS (CHEN ET AL., AL., 2019)
75.6
97.50
4.9
557
GDAS (DONG YANG, 2019)
0.21
74.0/91.5
97.18
5.3
545
PC-DARTS (Xt ET AL., 2020)
0.13
74.9/92.2
97.4310.00
5.3
586
ISTA-NAS (YANG AL.,, 2020)
4.2
76.0
97.46.10.05
5.65
638
SDARTS- (CHEN & HSIEH, 2020)
74.8/92.2
97.52+0.02
3.3
FAIR DARTS (CHU ET AL., 2020B)
0.4
75.1
97.46
4.8
541
DARTS+PT (WANG ET AL., 2021)
0.8
74.5192.0
97.52
4.6
TE-NAS (CHEN ET Al.,, 2021A)
0.17
75.5
97.37+0.06
5.4
EOINAS (ZHOU Al.., 2021)
0.6
74.4191.7
97.50+0.10
5.0
BALENAS-TF (ZHANG ET AI.,, 2022)
75.8
97.5+0.07
597
PRE-NAS (PENG ET AL., 2022)
0.6
76,0/92.6
97.5110.09
6.2
EAEPSO (YUAN ET AL., 2023)
73.1
97.49
4.9
MOEA-PS (XUE ET AL., 2023)
2.6
73.6/91.5
97.23
4.7
ANGLELOSS (YANG
AL., 2023B)
0.11
15.9
97.44
5.9
PIANT-T (LU ET AL., 2023)
0.11
72.5
97.46
5.2
583
SWD-NAS (XUE ET AL., 2024)
0.13
7
97.49
6.3
EG-NAS (CAI ET AL., 2024)
0.1
7-.1
97.47
5.2
IS-DARTS (HE AL.,, 2024A)
0.42
75.9
97.44110.04
6.4
PG-NAG (AVERAGE)
0.004
-1-
97.48 + 0.07
5.5
554
PG-NAG (BEST)
0.004
76.1/92.7
97.56
5.5
554
search space, which is evaluated on ImageNet (Deng et al., 2009) and CIFAR-10 (Krizhevsky et al., 2009). All results for SOTA methods in Table are sourced directly from their respective papers, where only the best accuracy (de- noted as xxx) or the mean accuracy (denoted as xxxXxx) is reported. PG-NAG achieves state-of-the-ard performance, with the highest ImageNet accuracy and CIFAR-10 accu- racy. Remarkably, the architecture generated by PG-NAG only needs 0.004 GPU days, approximately 5.76 minutes. In terms of model complexity, architectures generated by PG-NAG maintain competitive parameter size and FLOPs, striking a balance between performance and resource effi- ciency. These results highlight the effectiveness of PG-NAG in generating high-ques quglitualtty architectures with minimal com- putational resources. Further visualizations and architecture stability analysis can be found in Appendix C.3 and Ap- pendix C.4, respectively.
Results on NATS-Bench. We evaluate PG-NAG within the NATS- Bench search space sss to assess its efficiency compared to several state-of-the-arr NAS methods bench- marked on NATS-Bench. These comparison methods em- ploy range of different search strategies, all aiming at identifying high-performance architectures. As shown in Table 2, PG-NAG achieves competitive performance across three datasets-CCIAR-. 120 (Chrabaszez et al., 2017) in NAS-Bench-201 search space, demonstrating superior results without a search pro- cess. In addition, the cost of PG-NAG maintains a signif- icantly lower search cost of 4, 171 seconds compared to other methods. The results demonstrate that PG-NAG can generate architectures that achieve competitive accuracies, highlighting its efficiency and effectiveness in neural archi- tecture generation.
CIFAR-100, and ImageNet- 16-
In order to verify the effectiveness and generability of the architecture generated by PG-NAG on different visual tasks, we validate it on the MeanNNAS-Bench-a
Results on TransNAA-Benchh-l
Table Performance comparison with state-of-the-art architec- tures in NATS Bench search space.
search space. We compare PG-NAG with four NAS search methods (Duan et al., 2021) and one NAS acceleration method (He et al., 2024b). As shown in Table 3, PG-NAG shows strong performance in Class Object and Room Layout. In order to use a metric to measure the combined performance of the above methods on the seven tasks, we add an average rank column the right of Table 3. Specifically, we compute the ranking of architectures for each task. These rankings then averaged to evaluate the
METHOUS
COST (S)
CIFAR-10
CIFAR-100
IMAGENer16-120
70.6710.62
BOIBB
12,000
93.9410.28
72.00
45.70.1..66
7.387
91.0520.66
68.2610.96
40.6911.33
DARTS
90.02
65.3827.84
60.49:+..9
36.7947.59
93.2320.58
39.40:0.000
SETN
31,010
92.72+0.73
69.36++1.72
39.51=0.33
4,147
93.91=0.08
70.96+0.14
44.73+0.35
6

---

Prior Knowledge Guided Neural Architecture Generation
Table Performance comparison with state-of-the-ar architectures in TransNAS MA--AS---enc-1OO0 search space. Room layout's 1.2 loss is mul- tiplied by factor of 100 for better readability. The rightmost column reports the average percentile rank of searched networks (ang. the benchmark, averaged across target tasks.
CLASS
CLASS SCENE
SURFACE NORMAL
SEMANTIC SEGMENT
AVERAGE RANK
TASK
JIASAW
OBJECT
ENCODING
LAYOUT
METRIC
Acc.t
Acc.t
SSIMT
SSIM
MloUt
L2LOSS
ACC.1
DT
42.03
49.80
51.20
55.03
22.45
66.98
88.95
42.53
RS
45.16
54.41
55.94
56.85
25.21
61.48
94.47
13.70
PPO-TES
45.19
54.37
55.83
56.90
25.24
61.38
94.46
13.53
PPO-TRANSEER
44.81
54.15
55.70
56.60
24.89
62.01
14.26
RoBoT
45.59
54.87
55.42
57.44
26.27
61.16
94.82
16.43
PG-NAG
46.32
54.38
54.12
57.30
25.03
60.09
94.40
12.23
generalization and effectiveness of architectures searched or generated by different methods. Although other methods perform well specific tasks, PG-NAG shows competi- tive performance across the tasks, demonstrating its excellent generalization capabilities across different tasks.
Table 5. Statistics the generated architectures. Each method generates 1, 000 architectures.
DATASET
STATES.
RANDOM
DIFFUSIONNAG
PG-NAG
MAX
94.37
94.37
94.36
CIFAR-10
MEAN
87.12
94.13
94.15
MIN
10.00
86.44
87.08
Table 4. Comparation of test validation analysis on NAS-Bench-
ASR and NAS- MAS-Bench-NLP
MAX
72.74
73.51
73.51
CIFAR-100
MEAN
61.59
70.34
70.79
MIN
1.00
58.09
66.08
TEST PER (8)
TEST LoG PERPLEXITY
METHODS
AvG.
MIN.
NASBOT
21.89
22.02
21.9323881
4.38
4.67
1.61:0.03
NPENAS
21.88
21.89+10.6
4.55
4.67
4.58+0.03
BANANAS
21.86
21.99
21.8944..79
4.6
4.67
4.63++0.02
4.3. Distribution of generated architectures
REA
21.90
22.25
21.9510.10
4.62
4.67
4.602
PG-NAG
21.58
21.86
21.78.1013
4.53
4.66
4.5710.02
We further explore the distribution of the accuracy of archi- tectures generated by PG-NAG. Firstly, we compare the dis- tribution of 1, 000 generated architectures with 1, 000 ran- domly selected architectures NAS- Bench-201, as shown in Figure 3. Then, we compare the stability the generated architectures with random architecture sampling and Dif- fusionNAG (An et al., 2024) Table 5. The results show that the distribution of the generated architectures has low variance, indicating concentrated distribution characteris- tic with tendency for clustering in the nigh-performance part. Meanwhile, both the mean and the minimum of the accuracy of the generated architectures are higher than that of NAG. This means that PG-NAG can target the generation of perforrrreee architectures when deal- ing with specific tasks. To make the comparison clearer, delete some low oowpeeffrrmance architectures obtained by random selection, the original figures are in Appendix C.6.
Results on formance of PG-NAG on the unseen task of speech recog- nition, compare it with four different NAS methods the MAS-Bench-AARR search space. The architecture gen- erated by PG-NAG will be validated on the TIMIT speech recognition dataset (Garofolo et al., 1993), using Phoneme Error Rate (PER) as the performance metric. As shown in Table 4, architectures generated by PG-NAG have lower eтror rates than other methods, consistently outperforming them in terms of minimum, maximum, and average values. Therefore, PG-NAG can generate architectures with more stable results and lower speech recognition error rates.
order to validate the per-
Results on NAS-Bench-NLP. To evaluate the effectiveness of the architectures generated by PG-NAG on the unseen language modeling task, we conduct a comparison with four other NAS methods within the MAS--Benc-NNL search space. The architectures are tested on the Penn Tree Bank dataset, with test log perplexity serving as the performance metric. As illustrated in Table 4, PG-NAG achieves the lowest test log perplexity. The variance in the performance of architectures generated PG-NAG is also the lowest com- pared to other methods, further suggesting that it generates stable and reliable architectures.
4.4. Ablation Study
Components of PG-NAG. We evaluate the impact of three core components of PG-NAG: prior knowledge guidance, operation feature extraction method, and connection feature extraction method. The experimental results on CIFAR- 10, CIFAR-100, and ImageNet16 120 in NAS-Bench-201 are shown in Table 6. The prior knowledge denotes the

---

Prior Knowledge Guided Neural Architecture Generation
Table Ahalation study about components of PG-NAG NAS-Bench-201.
PRIOR KNOWLEDGE
OPERATION FEATURE
CONNECTION FEATURE
CIFAR-10
CIFAR-100
IMAGENETI6-120
VALID
TEST
VALID
TEST
VALID
TEST
X
GCN
90.45
93.31
71.17
71.13
45.63
45.77
GIN
91.50
94.37
73.31
73.09
45.59
46.33
GAT
91.34
94.00
72.98
72.48
45.23
46.20
GCN
X
91.35
94.30
72.77
72.30
45.53
46.44
GCN
91.55
94.36
73.49
73.51
46.37
46.34
500, 800, and 1, 1000 steps. While larger time steps generally improve the accuracy of the sampling process, larger time step usually results in higher computational cost. Therefore, in order to ensure the efficiency of architecture generation, we strike a balance between computational efficiency and the performance of architectures. These findings confirm that the diffusion model designed in PG-NAG is highly effective and well-suited for architecture generation tasks.
CIFAR-10
CIFAR-100
Figure 3. The distribution of generated architectures.
Table 8. Compare the quality and quantity prior knowledge on NAS-Bench-22
generation of architectures without the guidance of prior knowledge. The operation feature involves replacing the original graph convolution network for operation extrac- tion with a graph isomorphic network or a graph attention network. The connection feature represents the learning method of connections used by MIP for learning one-hop subgraphs is replaced with new MLP that learns for con- nections only. These results highlight that prior knowledge guidance leads to the greatest performance drop, indicating its critical role in PG-NAG.
CIFAR-I0
CIFAR-100
IMAGENET 16
METHOD
TEST
TEST
k=10
89.68
92.31
68.84
69.49
43.22
42.60
k=20
91.55
94.36
73.49
73.51
46.37
46.34
k=30
94,22
73.13
73.17
46.32
k=50
91.61
94.37
72.75
73.22
45.56
46.71
RANDOM 20
82.42
87.11
57.06
56.95
30.70
30.57
Prior knowledge. To verify the significance of prior knowl- edge, conduct ablation experiments the quality and quantity of the selected architectures in NAS- Bench-201. Firstly, we evaluate the impact of varying the number of high-performanee architectures i used as prior knowledge to guide the generation model. We evaluate PG-NAG with k == 10, 20, 30, and 50 architectures. As shown in Table the results indicate that k 50 yields the highest valid and test accuracy in CIFAR-10 and test accuracy in ImagNet- 16-120. These results suggest that A: 20 is practical and effective choice, as it achieves near-optimal performance across datasets. Increasing A to 30 or 50 provides an ad- ditional benefit, causing unnecessary computational cost when using larger prior knowledge sets. Instead, we select a lower but effective number learn prior knowledge.
Table 7. Comparison with different diffusion models on NAS- Bench-201.
METHODS
NOISE SCHEDULE
NUMBER OF TIMESTEP
CIFAR-10
CIFAR-100
IMAGENET
DDPM
1,000
93.28
70.23
41.88
SIGMDID
500
92.73
69.
42.83
PG-NAG
SIGMDID
500
94.22
73.17
46.48
1,000
45.33
SIGMDID
1,000
94.36
Diffusion model. To evaluate the effectiveness of the diffu- sion model PG-NAG, we conduct a comparative analysis with the standard DDPM diffusion model. Additionally, we examine the impact of key components in our diffusion model, i.e., the noise schedule and the number of time steps. As shown in Table 7, PG-NAG significantly outperforms DDPM on CIFAR-IO, CIFAR-100, and ImageNet 120. For the noise schedule, PG-NAG utilizes a sigmoid sched- ule, which we compare to the linear schedule used in DDPM. Additionally, we test different time step settings, including
Secondly, to evaluate the quality of selected architec- tures, select 20 random architectures instead of high- performance architectures. We find that the perfor- mance of architectures declines compared with using high- performance architectures as prior knowledge. These comparisons demonstrate that the quality of prior knowl- edge is crucial for guiding the architecture generation pro-
8

---

Prior Knowledge Guided Neural Architecture Generation
Higgh-perfommance architectures provide meaningful insights that enable PG-NAG to achieve superior results, whereas random architectures fail to provide such guidance.
Chen, X., Xie, L.,, Wu, J., and Tian, Q. Progressive dif- ferentiable architecture search: Bridging the depth gap between search and evaluation. In ICCV, pp. 1294-1303, 2019.
5. Conclusion
Chen, X., Wang. R., Cheng, M., Tang, X., and Hsieh, C.- J. Drnas: Dirichlet neural architecture search. arXiv preprint arXiv:2006. 10355, 2020.
In this paper, we propose a method to automatically generate segh-performance architectures guided by prior knowledge. It leverages an analysis of high-performance architectures to formulate prior knowledge. Then the knowledge serves as guiding principle for the diffusion model, enabling it to automatically generate architectures optimized for various tasks. Extensive experimental results on new search spaces show that PG-NAG generation process is highly efficient on unseen tasks without costly and time-ccnsuming training. In addition, the effectiveness of the components in PG-NAG is verified by the ablation study.
Chen, Z., Zhan, Y., Yu, B., Gong, M., and Du, B. Not all operations contribute equally: Hierarchical operation- adaptive predictor for neural architecture search. In ICCV, pp. 10508-10517, 2021b.
Chrabaszez, P., Loshchilov, I., and Hutter, F. A downsam- pled variant of imagenet as an alternative to the cifar datasets. arXiv preprint 17Xiv:1007.000 7778988199....... 2017.
Chu, X., Wang, X., Zhang. B., Lu, S., Wei, X., and Yan, J. Darts-: robustly stepping out of performance collapse without indicators. arXiv preprint arXiv:2009.01027, 2020a.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chu, X., Zhou, T., Zhang, B., and Li, J. Fair darts: Eliminat- ing unfair advantages in differentiable architecture search. In ECCV, pp. 460. Springer, 2020b.
Acknowledgments
Chu, X., Zhang, B., and Xu, R. Fairna:: Rethinking evalua- tion fairness of weight sharing neural architecture search. In ICCV, pp. 12239-12248, 2021.
This work was supported by National Natural Science Foun- dation of China under Grant 62276175 and Innovative Re- search Group Program of Natural Science Foundation of Sichuan Province under Grant 2024NSFTD0035.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 008-255. Ieee, 2009.
References
Dong, X. and Yang, Y. Searching for robust neural ar- chitecture in four gpu hours. In CVPR, pp. 1761-1770, 2019.
An. S., Lee, H., Jo, J., Lee, S., and Hwang, S. J. Diffusion- nag: Predictor Predictor-guided neural architecture generation with diffusion models. In ICLR, 2024.
Dong, X. and Yang. Y. Nas-hench-201: Extending the scope of reproducible neural architecture search. arXiv preprint 2001:000000036 2020a.
Cai, H., Gan, C., Wang, T., Zhang, Z., and Han, S. Once- for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv: /rXiv:/908.000 097919 2019.
Dong, X. and Yang. Y. Nas-bench-201: Extending the scope of reproducible neural architecture search. In ICLR, 2020b. URL https: wwwpreeieeeeeeeeeeeeeeeeeeeeeeee net/ forum? id=HJxyZkBKDD:
Cai, Z., Chen, L., Liu, P., Ling, T., and Lai, Y. Eg- Eg-nas: Neu- ral architecture search with fast evolutionary exploration. In AAAI, volume 38, pp. 11 11159-11161 1167, 2024.
Dong, X., Liu, L.,, Musial, K., and Gabrys, B. Nats-bench: Benchmarking nas algorithms for architecture topology and size. IEEE transactions on pattern analysis and machine intelligence, 44(7):3634-3646, 2021.
Chen, W., Gong, X., and Wang. Z. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2 1535, 2021a.
Duan, Y., Chen, X., Xu, H., Chen, Z., Liang, X., Zhang, T., and Li, Z. Transnas-bench- bench-1011 Improving transferabil- ity and generalizability of cross-task neural architecture search. CVPR, pp. 0251-5260, 2021.
Chen, X. and Hsieh, C.-J. Stabilizing differentiable archi- tecture search via perturbation based regularization. In ICML., pp. 1565-1565. PMLR, 2020.

---

Prior Knowledge Guided Neural Architecture Generation
Dudziak, L.,, Chau, T., Abdelfattah, M., Lee, R., Kim, H., and Lane, N. Brp-nas: rediction-based nas using gens. NeurlPS, 33: 13:10080-10490 2020.
Liu, Y., Tang, Y., and Sun, Y. Homogeneous architecture augmentation for neural predictor. In ICCV, pp- 12249 12258, 2021.
Elsken, T., Metzen, J. H., and Hutter, F. Neural architecture search: A survey, Journal of Machine Learning Research, 20(55):1-21, 2019.
Liu, Y., Tang, Y., Lv, Z., Wang, Y., and Sun, Y. Bridge the gap between architecture spaces via cross-domain predictor. NeurIPS, 15:13355-133666 2022.
Garofolo, J. S., Lamel, L. Fisher, W. M., Pallett, D. S., Dahlgren, N. I., Zue, V., and Fiscus, J. G. Timit acoustic- phonetic continuous speech corpus. (No Title), 1993.
Lu, S., Hu, Y., Wang, P., Han, Y., Tan, J., Li, J., Yang, S., and Liu, J. Pinat: A permutation invariance augmented transformer for nas predictor. In AAAI, volume 37, pp. 8957-8965, 2023.
He, H., Liu, L., Zhang, H., and Zheng, N. Is-darts: Stabiliz- ing darts through precise measurement candidate im- portance. In AAAI, volume 38, pp. 12367-12375, 2024a.
Lukasik, J., Jung, S., and Keuper, M. Learning where to look-generative nas is surprisingly efficient. In ECCV, pp. 257-273. Springer, 2022.
He, Z., Shu, Y., Dai, Z., and Low, B. K. H. Robustifying and boosting training-freee neural architecture search. In The Twelfth International Conference on Learning Repre- sentations, 2024b.
Mehrotra, A., Ramos, A. G. C., Bhattacharya, S., Dudziak, L., Vipperla, R., Chau, T., Abdelfattah, M. S., Ishtiaq, S., and Lane, N. D. Nas-bench-asr: Reproducible neural architecture search for speech recognition. ICLR, 2021.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12558, 2022.
Mills, G., Han, F. X., Salameh, M., Changiz Rezaei, S., Kong, L., Lu, W. Lian, S., Jui, S., and Niu, D. L2nas: Learning optimize neural architectures via continuous- action reinforcement learning- In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 1284-12953 2021.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. NeurlPS, 33:6840-6851, 2020.
Huang, S. Y. and Chu, W.-T. Searching by generating: Flex- ible and efficient one-shot nas with architecture generator. In CVPR, pp. 983-992, 2021.
Mills, K. G., Fred, X. H., Salameh, M., Lu, S., Zhou, C., He, J., Sun, F,, and Niu, D. Building optimal neural architectures using interpretable knowledge. CVPR, June 2024.
Ji, H., Feng, Y., and Sun. Y. Cap: A context-aware neural predictor for nas. In Larson, K. (ed.), Proceedings of the Thirly-Third International Joint Conference on Artifi- cial Intelligence, IJCAI-24, pp. 4219-4223 International Joint Conferences on Artificial Intelligence Organization, 2024. doi: 10.24963/ijcai.i 2024/466. URL https: //doi.orgg / 24 /63 ijcai. 2024 466. Main Track.
Peng. Y., Song, A., Ciesielski, V., Fayek, H. M., and Chang, Pre-nas: wredictor-assisten evolutionary neural archi- tecture search. In GECCO, pp. 1066-1074 1074, 2022.
Peng. Y., Song, A., Fayek, M., Ciesielski, V., and Chang, X. Swap-nas: Sample-wise activation patterns for ultra- fast nas. arXiv preprint arXiv:2403 240.204401...1 2024.
Jing, K., Xu, J. and Li, P. Graph masked autoencoder enhanced predictor for neural architecture search. In IJCAL, pp. 3114-3120, 2022.
Real, E., Aggarwal, A., Huang, and Le, Q. V. Regular- ized evolution for image classifier architecture search. In AAAI, volume 33, pp. 4780-4789, 2019.
Klyuchnikov, N., Trofimov, I., Artemova, E., Salnikov, M., Fedorov, M., Filippov. A., and Burnaev, E. Nas-bench- nlp: neural architecture search benchmark for natural language processing. IEEE Access, 10:45736-45747, 2022.
Shi, H., Pi, R., Xu, H. Li. Z., Kwok, J., and Zhang, T. Bridging the gap between sample-based and one-shot neural architecture search with bonas. NeurlPS, 33:1808- 1819, 2020.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Sun, Y., Wang, H., Xue, B., Jin, Y., Yen, G. G., and Zhang, M. Surrogate- evolutionary deep learning using an end-to-end random forest-based performance predictor. IEEE Transactions on Evolutionary Computation, 24(2): 350-364, 2019.
Liu, H., Simonyan, K., and Yang, Y. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.099 2018.
10

---

Prior Knowledge Guided Neural Architecture Generation
Tang, Y., Wang, Y., Xu, Y., Chen, H., Shi, B., Xu, C., Xu, C., Tian, Q., and Xu, C. semi-supervised assessor of neural architectures. In CVPR. pp. 1810-1819, 2020.
Yang, Y., Li, H., You, S., Wang. F., Qian, C., and Lin, Z. Ista- Efficient and consistent neural architecture search by sparse coding, NeurIPS, 33:10503-10511 2020.
Vignac, C., Krawezuk, I., Siraudin, A., Wang, B., Cevher, V., and Frossard, P. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209. 14734, 2022.
Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., and Hutter, Nas-bench-101: Towards reproducible neural architecture search. In ICML,, pp. 7105-71 14. PMLR, 2019.
Wang. R., Cheng, M., Chen, X., Tang, X., and Hsieh, C.-J. Rethinking architecture selection in differentiable nas. In ICLR, 2021.
Yuan, G., Wang, B., Xue, B., and Zhang, M. Particle swarm optimization for efficiently evolving deep convolutional neural networks using an autoencoder-based encoding strategy. IEEE Transactions Evolutionary Computa- tion, pp. 1-1, 2023. doi: 10. 101109/7EVC.222335255522
Wei, C., Niu, C., Tang, Y., Wang, Y., Hu, H., and Liang, J. Npenas: Neural predictor guided evolution for neural ar- chitecture search. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Zela, A., Siems, J., Zimmer, L., Lukasik, J., Keuper, M., and Hutter, F. Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks. arXiv e-prints, art. ar 2XXx:2000.0097770 10.4 000485500arxiv.2208.09999
White, C., Neiswanger, W., and Savani, Y. Bananas: Bayesian optimization with neural architectures for neu- ral architecture search. In AAAI, volume 35, pp. 10293- 10301, 2021.
August 2020. doi:
Zhang, M., Su, S. W., Pan, S., Chang, X., Abbasnejad, E. M., and Haffari, R. idarts: Differentiable architecture search with stochastic implicit gradients. In ICMI, pp. 12557-12566. PMLR, 2021.
Xiao, H., Wang, Z., Zhu, Z., Zhou, J., and Lu, J. Shapley- nas: Discovering operation contribution for neural archi- tecture search. In CVPR, pp. 11892-11901, 2022. 2022.
Zhang, M., Pan, S., Chang, X., Su. S., Hu, Haffari, G. R., and Yang, B. Balenas: Differentiable architecture search via the bayesian learning rule. In CVPR, pp. 11871- 1 11880, 2022.
Xie, S., Zheng, H., Liu, C., and Lin, Snas: stochastic neu- ral architecture search. arXiv preprint arXiv:1812.09926, 2018.
Zhou, Y., Xie, X., and Kung, S.-Y. Exploiting operation importance for differentiable neural architecture search. IEEE Transactions on Neural Networks and Learning 6248, 2021.
Xu, Y., Xie, L., Zhang, X., Chen, X., Qi, G.-J,, Tian, Q., and Xiong, H. PC-DARTS: Partial channel connections for memory-efficient architecture search. ICLR, 2020.
Systems,
Xu, Y., Wang, Y., Han, K., Tang, Y., Jui, S., Xu, C., and Xu, C. Renas: Relativistic evaluation of neural architecture search. In CVPR, pp. 441 1-400, 2021.
Xue, Y., Chen, C., and Stowik, A. Neural architecture search based multi-objective evolutionary algorithm with probability stack. IEEE Transactions on Evolutionary Computation, 2023.
Xue, Y., Han, X., and Wang. Z. Self-adaptive weight based on dual-attention for differentiable neural architecture search. IEEE Transactions on Industrial Informatics, 20 (4):6394-6403, 2024. doi: 10.1109/T1.22223333433
Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., and Yang, M.-H. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023a.
Yang. T., Yang, L., Jin, X., and Chen, C. Revisiting training- free nas metrics: An efficient training-bass method. In WACV, pp. 4751-4760, 2023b.

---

Prior Knowledge Guided Neural Architecture Generation
A. Details of PG-NAG.
A.I. Architectures Encoding for Different Search Spaces
Since we have chosen architectures from three different search spaces for learning prior knowledge, we first need a unified representation of the architectures. To unify the architectures in different search spaces, we design generalizable encoding for different search spaces following the convention. Specifically, we encode all operations from different search spaces into vectors using the one-hot encoding method. For operations in different search spaces but with similar we use the same encoding. For instance, we encode max-pooling in NAS-Bench-100 and avg- org-pooling in functionality, with the same encoding. Additionally, MAS-Bench- operations on edges. Therefore, we transfer operations on nodes to standardize. This encoding approach allows us to encode architectures from three search spaces in a uniform standard.
has operations on nodes while NAS-Benche Bench-201 and DARTS have
A.2. Dependence on Prior Knowledge
The success of PG-NAG relies on the quality of the prior knowledge. As we all know, different operations in architecture are not equally important (Chen et al., 2021b). Therefore, some methods have tried to analyze the operations in architecture, such as (Wang et al., 2021; Xiao et al., 2022). Furthermore, by analyzing the effect of different operations in DARTS, DARTS-(Chu et al., 2020a) discovers that skip connections have a clear advantage over other candidate operations and treats it prior knowledge. Therefore, this method adds decaying auxiliary skip connection. Similarly, iDARTS (Zhang et al., 2021) analyzes the preferences of different operations as prior knowledge. Based on prior knowledge, iDARTS interposes static BN layer between the node input and the operations to maintain the balance between different operations. Therefore, prior knowledge has been used in the process of NAS to search for optimal architectures.
In this paper, we recognize that PG-NAG relies on carefully selected architectures to obtain prior knowledge. This knowledge derived from three search spaces of different sizes, while the architectures these search spaces have been validated on three different datasets, which provides us with broad reference for generating new architectures. By learning from these diverse architectures, we obtain representative prior knowledge, which enables our method to not only perform well in these three search spaces but also to achieve good results in other unseen search spaces. This shows that the prior knowledge we learned based on these architectures is highly generalizable and flexible.
B. Experimental Details
B.1. Search Spaces
In this section, we describe the search spaces that are used verify the effectiveness of PG-NAG.
DARTS (Liu et al., 2018) search space is one of the most popular search space. It contains total of 10 unique architectures. These architectures are always evaluated on three datasets: CIFAR-10, C00, and ImageNet.
(Dong & Yang, 2020a) is a widely used benchmark that includes a total of 15, 625 unique architectures. These architectures are trained and evaluated on CIFAR-10, CIFAR-100, and ImageNet1 6- 6-20 datasets.
TrunsNAS- MannsNAS-Beech-101 (Duan et al., 2021) is a relatively recent addition to NAS benchmarks. evaluates architectures across seven vision tasks: Scene Classification, Object Classification, Autoencoding, Surface Normal, Semantic Segmentation, Room Layout, and Jigsaw Puzzle. A total of 51, 464 architectures are evaluated across these tasks with comprehensive training details.
NAS-Bench- 10/ (Ying et al., 2019) is the first public architecture benchmark for NAS research. It consists of 423, 624 unique architectures, all of which are trained and fully evaluated multiple times on the CIFAR-10 dataset.
NAS-Bench-ASR (Mchrotra et al., 2021) is a NAS benchmark for Automatic Speech Recognition (ASR). It contains 8, 242 unique models trained on the TIMIT audio dataset for three different epochs, with each model starting from three different initializations.
NAS-Bench-NLP (Klyuchnikov et al., 2022) is a NAS benchmark focused the language modeling task in natural language processing. It includes 14, 000 RNN-like architectures, each trained and validated on the Penn Tree Bank (PTB) dataset.
12

---

Prior Knowledge Guided Neural Architecture Generation
B.2. Datasets
In this section, we describe the validation datasets that are used to verify the effectiveness of PG-NAG.
For computer vision validation datasets, utilize three widely used datasets: CIFAR-IO, CIFAR-100, and ImageNet, which are all image classification datasets. The number of images in CIFAR-10 and CIFAR-100 is 60, 000. CIFAR-10 has ten classes, each containing 6, 000 images. CIFAR-100 has 100 classes, each containing 600 images. ImageNet is massive dataset comprising over 12, 197, 122 images. ImageNet has 1000 classes, each contains images between 732 and 1, 1,300. ImageNet-16 16-120--0000 downsamples ImageNet to 16x 16 pixcls, from which selects all images with label E [1, 120] to construct ImageNet- 16- 16-100. ImageNet-16, 120 contains 151, 700 training images, 3, 000 validation images, and 3, 000 test images with 120 classes.
For the automatic speech recognition task, valid architectures TIMIT dataset. It consists of recordings of 630 speakers of 8 dialects of American English each reading 10 phonetically rich sentences. It also comes with the word and phone-level transcriptions of the speech.
For the natural language processing task, we valid architectures on PTB dataset, which is one of the most known and used corpus for the evaluation of models for sequence labelling. It consists of annotating each word with its Part-of-Speech tag. has 38, 219 setences and 912, 344 tokens for training, 5, 527 sentences and 131, 768 tokens for validation, and 5, 462 sentences and 129, 654 tokens for testing.
B.3. Implementation Details
In the search process in NAS- MAS-Bench100, WAS--ench-2200 and TransNAS-Benchh.. Bench- 101, we generate a cell and stack it into an architecture according to the benchmark rules. In the DARTS search space, generate cell and set the number of initial convolutional channels 36. We optimize the architecture weights using stochastic gradient descent with initial learning rate of 0.025 and a single Consine annealing learning rate schedule. All experiments were done on Linux Ubuntu 18.04, using Nvidia 3090 GPUs.
C. Additional Experimental Results
C.1. Results on NAS-Bench-201
Table 9. Performance comparison with state-of-the-art architectures on CIFAR-10, CIFAR-100, and ImageNet16-120 in NAS-Bench-201
search space.
CIFAR-10
CIFAR-100
IMAGENETI6
METHODS
VALID
TEST
VALID
TEST
VALID
TEST
DARTS
39.77
$4.30
38.57
38.97
18.87
18.41
SGNAS
90.18
93.53
70.28
70.31
44.65
44.98
PC-DARTS
89.96
93.41
67.12
67.48
40.83
41.31
DARTS
89.96
93.58
70.57
70.83
40.38
40.89
DRNAS
91.55
94.36
73.49
73.51
46.37
46.34
FAIRNAS
90.97
93.23
70.94
71.00
41.09
42.19
BANANAS
94.37
73.51
ANGEI.LSSS
90.19
93.16
71.70
70.48
41.93
43.04
EG-NAS
90.17
93.58
70.90
70.98
45.18
46.59
L2NAS
91.47
94.28
73.02
73.09
46.58
47.03
DIFFLSIONNAG
94.37
73.51
PG-NAG
91.55
94.36
73.49
73.51
46.37
46.34
We applied PG-NAG to MAS-Bench-201 automatically generate hegh-performanc architecture. We compare PG-NAG against various popular NAS algorithms and one NAG method: DARTS and its variants (Liu et al., 2018; Xu et al., 2020; Zhang et al., 2021; Chu et al., 2021), SGNAS (Huang & Chu, 2021), DrNAS (Chen et al., 2020), BANANAS (White et al., 2021), AngelLoss (Yang et al., 2023b), EG-NAS (Cai et al., 2024), L2NAS (Mills et al., 2021), and DiffutionNAG (An et al., 2024). As shown in Table 9, all searches or generates are performed on CIFAR-10, 10, and the output architecture is then trained and evaluated on each of CIFAR-10, 10, CIFAR-100, and ImageNet16-120, These comprehensive experimental results
13

---

Prior Knowledge Guided Neural Architecture Generation
indicate that our method surpasses the other baselines.
C.2. Results on NAS-Bench-101
Table 10. Comparison with other NAS methods on CIFAR-I0 using the NAS-Benchh 101 search space.
METHODS
RANKING ( (%)
TEST ACC ( (%)
REA
0.384
93.64
ANGELLOSS
0.242
93.70
E2EPP
0.132
93.77
HAAP
0.004
94.09
RENAS
0.011
94.01
SEMINAS
0.011
94.01
NPENAS
0.002
94.14
AG-NAS
0.002
94.18
BRP-NAS
0.001
94.22
BANANAS
0,004
94.08
GMAE-NA -AS
0.002
94.14
BONAS
0.002
94.14
PG-NAG
0.001
94.23
We evaluate the architecture generated by PG-NAG on outlined in Table 10, the generated architecture demonstrates a performance improvement over previous methods such as REA (Real et al., 2019), AngelLoss (Yang al., 2023b), and E2EPP (Sun et al., 2019). To set up a basic benchmark, we also compare PG-NAG with HAAP (Liu et al., 2021), ReNAS (Xu et al., 2021), SemiNAS (Tang et al., 2020), NPENAS (Wei al., 2022), AG-NAS (Lukasik et al., 2022), BRP-NAS (Dudziak al., 2020), BANANAS (White al., 2021). GMAE- NAS (Jing al., 2022), and BONAS (Shi et al., 2020). Moreover, the proposed method PG-NAG achieved the highest accuracy architecture 94.23% in this search space.
101. We compare PG-NAG with 12 NAS methods. As
C.3. Architecture Generated for DARTS and NAS-Bench-201 Search Space
skip-commect
dip. SWWW!!
orp Dw0.
nor-Conv--::3
nor-conv-
K
nox.
nor-Conv-3x3
nor-Conv-3
Figure 4. Cell architecture found by PG-NAG on CIFAR-10 Figure 5. Architecture found by PG-NAG for NAS-Bench-201 Net datasets.
search space.
We use our PG-NAG to generate high perforreffom architecture in the DARTS search space based on the CIFAR-10 dataset at first. Then we transfer this architecture to the ImageNet dataset the DARTS search space. Figures demonstrate the neural architectures generated by PG-NAG for the CIFAR-10 and ImageNet datasets. The generated architectures contains three dil__onv__ 3 and two dil_comvy 5, and only one kki__cnnnect. This is consistent with the fact that the high-performance architectures searched by existing NAS methods contain large number of convolutional operations.
Similarly, Figures 5 demonstrates the neural architectures generated by PG-NAG for the CIFAR-10, CIFAR-100, and ImageNet datasets. The generated architecture consists of four nor_conv 3 x 3 and one skip_ cotp_connet, which closely resembles the high-performing architectures observed in MAS-Bench-200, Specifically, these architectures often include multiple 3 convolutions, highlighting the potential for similar performance patterns in the generated design.
14

---

Prior Knowledge Guided Neural Architecture Generation
C.4. Architecture Stability Analysis
97.5
PG-NAG (ours)
CDP
97.0
DARTS
Test Accuracy
96.5
96.0
95.5
95.0
300
500
600
Epoch
Figure 6. Comparison results existing methods CIFAR-10 on DARTS search spaces.
To verify the stability of the architectures generated by PG-NAG, we designed five different architectures for CIFAR-10 dataset the DARTS search space. We run these architectures five times and calculate their performance variance. In addition, utilize the architectures generated by DARTS and CDP method (Liu et al., 2022), which performs well in the DARTS search space. All experiments Figure 6, it can be seen that the PG-NAG achieves the best performance and demonstrates remarkable stability.
five times independently to ensure the reliability of the results. As shown in
C.5. Visualization for Architecture Accuracy Drop
2.50%
6.24%
2.5
6
2.0
1.84%
5.
Drop accuracy
4.
1.5
Drop
U
1.0
0.80%
2.06%
0.5
1.
0.0
conv convix1-tbn-rete
cony 3x3 bn-relu
maxpool3x3
nor, por_cony__u 1x1
nor ra_.coov_u _o33v_333 333
avg_poot pool_oo__3xs 3x3
(a) NAS-Bench-100
(b) NAS-Bench-201
Figure Visualization operation influence on architetcure performance in different search space.
To better understand how different operations affect the performance of architectures, which can help explain the selection of operations in our generated architectures. We remove operations in NAS-Bench- and NAS-Bench-201 to observe the impact of different operations on the performance of the architecture. Specifically, for each architecture, we compute performance by removing one operation at a time, progressively eliminating each operation individually (c.g.. first removing the first operation, then the second, and so on).
15

---

Prior Knowledge Guided Neural Architecture Generation
As shown in Figure 7(a) for NAS-Bench-101, the Conv_3x3 operation has the most significant impact on performance, underscoring its importance in achieving high accuracy. A similar trend is observed in NAS-Bench-22 in Figure 7(b), where the 3 3 convolution operation enhances model performance. These findings help us make informed decisions in selecting the most effective operations for generating perh-perroriing architectures.
C.6. Architecture Distribution of Generated Architectures
To demonstrate the effectiveness of PG-NAG in generating high- seethrrrffrmmece architectures stably, we conducted an experiment where 1, 000 architectures were generated using PG-NAG within the we randomly selected 1, 000 architectures from the same search space for comparison. Then we evaluate these architecutes on CIFAR-I0 and CIFAR-IO0 100 datasets. As shown in the Figure 8, the two figures are more complete distribution map than the Figure 3 in the main text Section 4.3. Architectures generated by PG-NAG consistently exhibit high accuracy, indicating that PG-NAG is capable of generating a large number of architectures with competitive performance stably. In contrast, the randomly selected architectures demonstrate broader range of performance, with some architectures performing significantly worse. This demonstrates the ability of PG-NAG to stably generate high-performance architectures.
search space. Additionally,
CIFAR-10
CIFAR
Random PG-NAG
250
Random PG-NAG
350
300
100
150
100
40
10
20
Accuracty
(a) CIFAR-I0
(b) CIFAR-100
Figure 8. The distribution of generated architectures on CIFAR- 10 and CIFAR- 100 in NAS- Bench-201.
16


Reviews:
[{"review_id": "2r1qap16XQ", "reviewer": "Reviewer_WwRR", "summary": "This paper proposes a neural architecture generation method called Prior Knowledge Guided Neural Architecture Generation (PG-NAG), which aims to generate high-performance neural architectures without the need for search and evaluation processes. By quantifying the contribution of each component within an architecture to its overall performance, the method identifies valuable prior knowledge and uses it to guide a diffusion model to generate architectures for various tasks. Extensive experiments demonstrate that PG-NAG achieves superior accuracy with minimal computational resources (e.g., generating architectures with 76.1% top-1 accuracy on ImageNet and 97.56% on CIFAR-10 in just 0.004 GPU days). The method also shows strong generalization across unseen search spaces like TransNAS-Bench-101 and NATSBench.\n\n## Update after rebuttal\n\nThe authors' rebuttal looks great to me. I am finally happy to raise the recommendation to clear accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our good performance, effectiveness, and efficiency. We are also grateful for the valuable and constructive feedback. \n## Weaknesses\n>An analysis of how the method works effectively\n\nThe effectiveness of PG-NAG can be attributed to prior knowledge guidance, operation feature extraction, and connection feature extraction. As shown in Table 6 in the main text, skipping or replacing any of these results in a decline in performance. Specifically, prior knowledge measures the contribution of each operation or connection in an architecture with Shapley values. Moreover, the operation feature extraction effectively learns representations of operations that contribute significantly to performance. These learned features are then integrated into the generation process. Similarly, the connection feature extraction learns the connections that are useful to high-performance architectures, ensuring the generation of high-performance architectures.\n>An explanation of metrics used in PG-NAG\n\nThe explanations of the six metrics used are in Table (_https://anonymous.4open.science/r/PGNAG/metrics.md_). We added this table in the new manuscript. \n>How does the selection of benchmarks impact the effectiveness of generated architectures\n\nWe leverage NAS-Bench-101, NAS-Bench-201, and NAS-Bench-301 to ensure the generated architectures perform well in basic image processing and large-scale complex tasks, exhibiting strong generalization capabilities across other tasks.\n* NAS-Bench-101 focuses on image classification, ensuring the generated architectures perform well on basic image tasks. The experiment shows that prior knowledge without NAS-Bench-101 has lower performance (comments 3 of VjZy).\n* NAS-Bench-201 contains simple but effective operations and has been the basis for various search spaces. Therefore, it can guarantee the generated architectures have generalization capabilities across different tasks. \n* NAS-Bench-301 supports complex architecture design for large-scale tasks, ensuring that the generated architectures are suitable for complex tasks like ImageNet.\n\n>The motivation for Shapley values \n\nThe motivation behind using Shapley values lies in the cooperative relationship between operations and connections in architecture. Specifically, the operations and connections in architecture are not independent of each other, and they interact as a whole to determine the overall performance. Shapley value can quantify the contribution in a cooperative game, thus helping us identify the contribution of operations and connections to the architecture performance [1]. By incorporating this, PG-NAG can focus on generating architectures that emphasize these critical components, leading to higher-performance designs.\n\n>Highlight the best-performing results\n\nWe have highlighted the best results in the new manuscript.\n## Questions\n>How to ensure the quality and diversity of prior knowledge?\n\nThe quality and diversity of prior knowledge are ensured by the benchmark selection, diversity, and sampling method. \n* Selection: We select high-performance architectures from three widely used NAS benchmarks, including NAS-Bench-101, NAS-Bench-201, and NAS-Bench-301. These architectures achieve good performance on well-known image classification datasets such as ImageNet. This allows us to learn how components are comprised in high-performance architectures and apply this knowledge to other search spaces.\n* Diversity: To illustrate the diversity of selected benchmarks, Figure (_https://anonymous.4open.science/r/PGNAG/prior.png_) visualizes the distribution of $20$ architectures sampled from NAS-Bench-101, confirming that our selected architectures cover a diverse distribution in terms of both accuracy and complexity.\n* Sampling method: We conduct ablation studies on the number of selected architectures in Table 8 and the selection method (weakness 1 in Reviewer PwDK). The results confirm that PG-NAG utilizes an effective sampling method to generate high-performance architectures.\n\n>How does PG-NAG balance computational efficiency and accuracy?\n\nPG-NAG addresses this trade-off by using a sigmoid noise schedule and prior knowledge to reduce diffusion steps.\nFirstly, we conduct experiments on CIFAR-10 in NAS-Bench-201 about how diffusion steps impact performance. Results below show that beyond $1,000$ steps, performance gains are minimal compared to the added computational cost.\n\n|time steps|CIFAR-10|\n|:-:|:-:|\n|$800$|$94.22$|\n|$1,000$|$94.36$|\n|$1,200$|$94.37$|\n\nSecondly, to achieve high performance in fewer steps, we use a sigmoid noise schedule for a smoother and more controlled noise removal process. Additionally, by integrating prior knowledge, PG-NAG effectively reduces the search space and enables the diffusion model to focus on high-performance designs without added computational cost.\n\n**Reference**\n\n[1] Shapley-NAS: Discovering operation contribution for neural architecture search, CVPR'22", "claims_and_evidence": "The claims made in the submission are supported.", "methods_and_evaluation_criteria": "Methods and evaluation criteria are appropriate.", "theoretical_claims": "There are no explicit theoretical proofs or claims that require verification. The focus of the manuscript is primarily on the empirical evaluation of PG-NAG rather than on theoretical analysis.", "experimental_designs_or_analyses": "The experimental setup and analyses appear to be well-structured and appropriate for assessing the claims made.", "relation_to_broader_scientific_literature": "The paper is well-aligned with recent literature, like neural architecture search and diffusion models.", "essential_references_not_discussed": "No significant references are missing.", "other_strengths_and_weaknesses": "Pros:\n\n1. The manuscript introduces a new approach to neural architecture generation by leveraging prior knowledge to guide the diffusion model, eliminating the need for traditional search and evaluation processes.\n2. This work achieves high performance with extremely low computational costs (0.004 GPU days).\n3. Extensive experiments and comparisons with baseline, thoroughly validating the effectiveness and efficiency of PG-NAG.\n\nCons:\n\n1. Providing deeper analysis and insights into why the proposed method works effectively can enhance the quality of the manuscript.\n2. The manuscript involves evaluations across multiple tasks and benchmarks, which strengthens the robustness of the work.  However, this also introduces a multitude of metrics.  A detailed explanation of these metrics would be beneficial for understanding the contributions of the manuscript.\n3. The prior knowledge extracted from existing benchmark datasets may not always represent the target tasks or domains. Further exploration is needed to understand how the selection of benchmark datasets impacts the effectiveness of the generated architectures\n4. The motivation behind using Shapley values to quantify the contributions of components within an architecture needs to be better articulated.\n5. Highlighting the best-performing results would make it easier for readers to quickly identify the strengths of their method.", "other_comments_or_suggestions": "Please see Strengths And Weaknesses.", "questions_for_authors": "1. How can the quality and diversity of prior knowledge be ensured when selecting high-performance architectures from benchmark datasets?\n2. How does PG-NAG handle the trade-off between computational efficiency and the accuracy of generated architectures, especially when increasing the number of diffusion steps?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ldJQ8GjYlf", "reviewer": "Reviewer_PwDk", "summary": "This paper proposes a method, Prior Knowledge Guided Neural Architecture Generation, to efficiently generate high-performance neural architectures without the need for an exhaustive search and evaluation process. The key idea is to leverage prior knowledge extracted from high-performance architectures to guide a diffusion model for architecture generation. The method is validated on several search spaces, including DARTS, NATS-Bench, TransNAS-Bench-101, NAS-Bench-ASR, and NAS-Bench-NLP, achieving state-of-the-art performance with significantly reduced computational cost (0.004 GPU days for ImageNet).\n\n## update after rebuttal\n\nAfter the rebuttal, I feel my concerns have been well-discussed by the authors, and some of weaknesses has been fixed. Therefore, I choose to raise my positive scores. As we see, the majority of reviewers are lean towards to accept this submission, maybe we ask the Reviewer xYiC who holds the only negative score to further discuss if his/her concerns have been addressed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our efficiency, novel use of Shapley values, and generality. We are also grateful for the valuable feedback.\n\n## Weakness\n\n> Limited discussion on prior knowledge\n \nA potential limitation is that the quality of prior knowledge affects the accuracy of generated architectures. Specifically, we compare different strategies for obtaining prior knowledge. In our main text, we select each top-20 high-performance architectures from the three benchmarks. To compare with it, we use the following sample strategies: architectures uniformly sampled across a performance range of 0%–100%, architectures uniformly sampled from two narrower ranges (i.e., 90%-100% and 80%–90%). The architecture accuracy we used is the validation accuracy in CIFAR-10. To see the distribution of architectures more intuitively, we take the benchmark NAS-Bench-101 as an example and visualize the performance and parameter distribution of architectures under different strategies in the Figure (_https://anonymous.4open.science/r/PGNAG/prior.png_). As shown in the table below, architectures randomly sampled in a narrow range of high performance can also generate high-performance architecture. This demonstrates that the quality of prior knowledge is crucial for the effectiveness of PG-NAG. \n\n|Method|CIFAR-10|CIFAR-100|ImageNet-16-120|\n|:-:|:-:|:-:|:-:|\n|sample in 0%-100%|87.11|56.95|30.57|\n|sample in 80%-90%|88.37|60.22|35.30|\n|sample in 90%-100%|93.23|70.06|43.02|\n|top-20|94.36|73.51|46.34| \n\n> Fixed noise schedule\n\nThe sigmoid noise schedule is fixed because it is effective when different search spaces and architecture complexities are adopted. Specifically, we conduct experiments in NAS-Bench-201 and DARTS search spaces, comparing the performance of sigmoid, linear, and cosine noise schedules. We evaluate PG-NAG on CIFAR-10, CIFAR-100, and ImageNet-16-120 in NAS-Bench-201, and evaluate PG-NAG on CIFAR-10 in DARTS. The table below shows the results that the sigmoid schedule achieves the best performance regarding different search spaces and architecture complexities.\n\n| Noise Schedule | CIFAR-10 | CIFAR-100 | ImageNet-16-120 | CIFAR-10 in DARTS |\n|:-:|:-:|:-:|:-:|:-:|\n|linear|93.42|   70.90   |      45.33      |       97.40       |\n|cosine| 93.50 |   70.67   |      44.53      |       97.54       |\n|sigmoid|  94.36   |   73.51   |      46.34      |       97.56       |\n\n> Size and performance trade-off\n\nWe would like to clarify that PG-NAG can handle the trade-off between model size and accuracy. This is because the prior knowledge already covers architectures with both high performance and diverse parameter counts, as visualized in Figure (_https://anonymous.4open.science/r/PGNAG/generated%20architectures%20in%20NAS-Bench-201.pdf_). We control the model size of the generated architectures when needed. As shown in the Table below, in the first three lines we control the model size and the last one is the results we reported in the main text, From the results, we can find that when the model size becomes smaller, the performance of PG-NAG does not drop significantly, demonstrating its ability to balance model size and performance effectively.\n\n| FLOP (M) | Params (MB) | Latency (ms) | CIFAR-10 | CIFAR-100 | ImageNet-16-120 |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|121.82|0.858|21.41|93.50|70.67|44.53|\n|149.34|1.045|19.97|94.02|72.99| 45.44 |\n |  153.27  |    1.073    |    20.22     |   94.37     |   73.22   |      46.71      |\n |  184.73  |   1.289     |    20.59     |    94.36     |   73.51   |      46.34      |\n\n> Improve interpretability\n\nTo interpret why the generated architectures perform well, we add two visualizations showing the impact of operations on architecture performance and the correlation between operations, respectively. Furthermore, we analyze that prior knowledge can effectively ensure the generation of high-performance architectures.\n* Operation-wise Influence on Performance: In Appendix Figure 7, we visualize the impact of different operations on architecture performance across various search spaces. This helps illustrate the contribution of each operation to overall accuracy.\n* Relevance of Operations in Benchmarks: We provide visualizations of operation relevance in NAS-Bench-101 (_https://anonymous.4open.science/r/PGNAG/Shapley%20values/NAS-Bench-101.pdf_), NAS-Bench-201(_https://anonymous.4open.science/r/PGNAG/Shapley%20values/NAS-Bench-201.pdf_), and NAS-Bench-301 (_https://anonymous.4open.science/r/PGNAG/Shapley%20values/NAS-Bench-301.pdf_), which are used to obtain prior knowledge. Shapley values quantify the marginal contribution of each operation, while heatmaps highlight the correlations between different operations for better interpretability.\n* Analysis of the effectiveness of PG-NAG:\nTo further explain why our method performs well, we provide a detailed discussion in response to Reviewer WwRR’s Weakness 1, analyzing the interactions between architecture components and their impact on performance.", "claims_and_evidence": "Some claims (e.g., efficiency and accuracy) are well-supported by experimental results.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria generally make sense for the problem of neural architecture generation.", "theoretical_claims": "Yes, I examined the theoretical claims related to the use of Shapley values(Equations 3 and 4) and the diffusion model (Equations 5, 6, and 7).", "experimental_designs_or_analyses": "Yes, I reviewed the soundness and validity of the experimental design and analysis in the paper, focusing on the benchmark selection, comparison methods, performance metrics, and ablation studies.", "relation_to_broader_scientific_literature": "The submission provides the discussion of the relevant literature. Prior work required iteratively evaluating a large number of architectures, and the manuscript provides a more efficient way to automatically generate architectures.", "essential_references_not_discussed": "This paper already has a comprehensive literature review.", "other_strengths_and_weaknesses": "Other Strengths,   \n**Efficiency.** PG-NAG eliminates the need for costly architecture evaluations, requiring only 0.004 GPU days to generate architectures with competitive performance.   \n**The use of Shapley values.** Shapley values to quantify the contribution of each operation and connection is new and grounded in cooperative game theory.   \n**Generality.** PG-NAG shows strong performance across diverse search spaces and tasks, including vision, speech, and language.   \n\nOther Weaknesses,   \n**Limited Discussion on Prior Knowledge.** The manuscript does not provide enough detail on the potential limitations or biases introduced by relying on prior knowledge from existing benchmarks.   \n**Fixed Noise Schedule.** The noise schedule in the diffusion model is fixed (sigmoid). However, different search spaces and architecture complexities might require different noise schedules.   \n**Size and Performance Trade-off.** The paper focuses heavily on accuracy but does not explore the trade-off between model size and accuracy.   \n**Improve Interpretability.** The manuscript should Include analysis or visualization of why the generated architectures perform well.", "other_comments_or_suggestions": "Refer to the Weaknesses section", "questions_for_authors": "Refer to the Weaknesses section", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5swzHvxWpB", "reviewer": "Reviewer_VjZy", "summary": "This paper presents a novel method to enhance neural architecture generation using diffusion models. Instead of relying on predictor-based approaches, the authors train a diffusion model on graph representations of high-performing architectures. They further integrate explicit prior meta-knowledge extracted through Shapley value analysis to quantify the contribution of each component. This guidance steers the model to focus exclusively on generating high-quality architectures. Experimental results demonstrate that the method achieves promising performance improvements.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our efficiency, high-quality outputs, and robust experiments. We are grateful for the constructive feedback.\n## Claims and Evidence\n>Potential limitations and diverse generation\n\nWe discuss potential limitations regarding the quality of prior knowledge (weakness 1 in Reviewer PwDk). Then we prove PG-NAG can generate diverse architecture (weakness 3 in Reviewer PwDk). \n>Effectiveness of guidance mechanism\n\nWe conduct experiments that indicate the guidance mechanism indeed plays a critical role. Specifically, we replace prior knowledge with architectures of 40%-50% and 80%-90% accuracy. PG-NAG generates architectures whose performance matches the provided architectures, demonstrating its effectiveness.\n\n|Architectures|CIFAR-10|CIFAR-100|ImageNet-16-120|\n|:-:|:-:|:-:|:-:|\n|40%-50%|75.81|47.13|15.66|\n|80%-90%|88.37|60.22|35.30|\n|top-20|94.36|73.51|46.34| \n\n## Weakness\n>Dependency on prior knowledge quality\n\nThe dependency is discussed in weakness 1 in Reviewer PwDk.\n>Limited architectural diversity\n\nWe discuss the diversity in claims and evidence 1.\n>Lack of Dataset Conditioning\n\nIndeed, PG-NAG has performed the dataset conditioning. Specifically, we incorporate constraints to guide the process for different datasets to achieve a similar effect as dataset conditioning. Unlike DiffusionNAG, PG-NAG successfully applied to a wider range of tasks such as speech recognition in Table 4 of the main text.\n>Attribution of performance gains\n\nThe performance gains of PG-NAG are mainly attributed to the guidance mechanism instead of the high-quality training data. The effectiveness of the guidance mechanism is demonstrated in claims and evidence 2.\n\n## Other Comments\n>Diversity of generated architectures\n\nWe investigate the diversity regarding the parameters and distribution of generated architectures. A visualization of the accuracy and params of 20 generated architectures is in Figure(_https://anonymous.4open.science/r/PGNAG/generated%20architectures%20in%20NAS-Bench-201.pdf_), it demonstrates that PG-NAG can generate diverse architectures. \n> Include DiffusionNAG\n\nWe added the mean accuracy of DiffusionNAG on CIFAR-10 (97.39%$\\pm$0.01), which is lower than PG-NAG (97.48%$\\pm$0.08). \n>Potential benefits of incorporating dataset-specific knowledge\n\nIncorporating dataset-specific knowledge could enhance accuracy. PG-NAG achieves top performance on NAS-Bench-101 in Appendix Table 1. To compare, we remove architectures from NAS-Bench-101 in prior knowledge and the results can be seen below. This shows that dataset-specific knowledge could make PG-NAG more precise.\n\n|Prior knowledge|Ranking|Acc(%)| \n|:-:|:-:|:-:|\n|no NAS-Bench-101|0.004|94.08|  \n|PG-NAG|0.001|94.23|\n\n## Questions\n>Architecture retrieval\n\n* Differences: A retrieval method needs to learn the target dataset to obtain the conditions. In contrast, PG-NAG learns from existing benchmarks and then transfers the learned design principles across different tasks, without the learning process on the target dataset.\n* Ensure the novelty: PG-NAG introduces noise to explore variations beyond the benchmarks. Figure(_https://anonymous.4open.science/r/PGNAG/differences%20in%20generated%20architecture%20in%20TransNAS-bench-101%20and%20prior%20knowledge.png_) illustrates the generated architectures for TransNAS-Bench-101 are different from the learned architectures in NAS-Bench-201.\n\n>Variance and diversity\n\nPlease see weakness 2.\n>Comparison with DiffussionNAG\n\n* Ensure generating high-performance architecture: We replace the predictor with a guidance mechanism, whose effectiveness has been discussed in claims and evidence 2.\n* Reason for performance gains: The performance gains cannot be attributed to the decent performance of training architectures on unseen datasets, it should be attributed to the effectiveness of the guidance mechanism based on prior knowledge. For example, NAS-Bench-NLP includes linear operations absent in prior knowledge, showing generated architectures adapt to new tasks. Additionally, prior knowledge is used for image classification, our tasks include speech recognition and natural language processing. \n\n>Incorporating data information\n\n* Challenges brought by incorporating dataset include reduces in generalization and the need for encoding. PG-NAG aims to generalize across datasets rather than overfitting to a specific one. While dataset-specific information can improve accuracy, it may reduce generalizability. Additionally, PG-NAG learns architecture design principles from different datasets, these architectures need to be represented in a unified format.\n* The difficulties preliminary come from the need to choose appropriate datasets and ensure effective learning across them. First, we need to select which dataset to be incorporated. To enhance the generalization, we choose three widely used NAS benchmarks. Second, to ensure effective learning across datasets, we encode architectures using a unified DGL graph representation.", "claims_and_evidence": "Most of the paper’s claims are supported by extensive empirical evidence. However, a couple of points warrant further scrutiny:\n\n- As noted in Lines 96-99, the authors train their generative model solely on high-performing architectures. This approach could limit the diversity of the derived prior knowledge. A deeper discussion or additional experiments examining the potential limitations and diversity issues of this focus would strengthen the claim.\n- Although the authors leverage prior knowledge to guide architecture generation with experimental results supporting this approach the evidence (see Table 8 in Section 4.4) suggests that the performance gains may primarily stem from extracting knowledge from high-performing architectures. This raises the question of whether the observed improvements are due to the superior quality of the training data rather than the guidance mechanism itself.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited to address the challenge of efficient neural architecture generation.\n\n**Methodology**\n\n Unlike existing diffusion-based architecture generation methods that rely on accuracy predictors derived from graph representations, the authors introduce a novel approach that integrates explicit prior knowledge. This prior knowledge is extracted from high-performing architectures using Shapley value analysis, which quantifies the contribution of each neural network operation. By combining this information with the graph representation of architectures, the diffusion model is trained to focus exclusively on generating high-quality designs.\n\n**Evaluation Criteria**\n\nThe authors evaluate their method across several well-established NAS benchmarks, including DARTS, NAS-Bench-201, TransNAS-Bench-101, NAS-Bench-ASR, and NAS-Bench-NLP. They use standard metrics such as top-1 accuracy, computational cost (measured in GPU days), parameter counts, and FLOPs. This comprehensive evaluation framework provides a robust comparison against state-of-the-art approaches and demonstrates the efficiency and effectiveness of the proposed method.", "theoretical_claims": "The proposed method builds upon conventional diffusion models, as seen in previous architecture generation approaches, and its claims are consistent with established principles in the conditional diffusion literature.", "experimental_designs_or_analyses": "Overall, the experimental framework is robust and well-aligned with established practices in NAS research. The authors evaluate their method on popular NAS datasets and benchmarks, comparing it with several existing approaches. However, one concern is that DiffusionNAG which also employs a diffusion-based approach but relies on an accuracy predictor is not included in Table 1 and only appears in later results. Including DiffusionNAG in the initial comparison would provide a clearer and more comprehensive evaluation of the proposed method relative to similar diffusion-based techniques.", "relation_to_broader_scientific_literature": "By integrating prior knowledge into the generation process, the authors extend and enrich the current scientific literature in neural architecture design. Specifically, they replace a task-conditioned accuracy predictor with a prior derived from high-performing architectures. However, this approach raises the question of whether training exclusively on high-performing architectures is sufficient to overcome the benefits offered by dataset-conditioned accuracy predictor-based models. While the contribution may appear marginal at first glance, the underlying idea holds promise and could pave the way for further improvements.", "essential_references_not_discussed": "The authors provide a comprehensive review of the most relevant work in this field, effectively contextualizing their contributions. This paper", "other_strengths_and_weaknesses": "**Strengths:**\n- **Efficiency in Generation:** By incorporating prior knowledge, the method reduces the need for iterative evaluation or reliance on accuracy predictors during the architecture generation process.\n- **High-Quality Outputs:** The generated architectures achieve competitive performance, demonstrating that the guidance from high-performing designs is effective.\n\n**Weaknesses:**\n\n- **Dependency on Prior Knowledge Quality:** The method’s success heavily relies on a highly curated set of high-performing architectures. If the source data is biased or unrepresentative, the performance could be adversely affected.\n- **Limited Architectural Diversity:** Training solely on high-performing architectures may constrain the model’s ability to generate diverse and innovative designs, potentially limiting the exploration of novel architectures. This lack of diversity is shown in Figure 3 and Table 5. since the method focused only on high performing architecture.\n- **Lack of Dataset Conditioning:** Unlike DiffusionNAG, which uses dataset-conditioned sampling to tailor generation for unseen datasets, this method operates via blind generation, potentially limiting its adaptability to new tasks.\n- **Attribution of Performance Gains:** There is a concern that the observed performance improvements might primarily stem from the high-quality training data rather than the inherent advantages of the proposed method itself.", "other_comments_or_suggestions": "- Conduct a thorough investigation of the diversity of the generated architectures relative to the pretrained architectures.\n- Include DiffusionNAG in Table 1 for a direct performance comparison.\n- Explore the potential benefits of incorporating dataset-specific knowledge into the generation process.", "questions_for_authors": "1. **Architecture Retrieval:**  \n   The sampling process appears to resemble a retrieval of high-performing architectures, since the conditioning is fixed based on a set of already high-performing designs. Could you clarify how this method differs from a simple retrieval mechanism, and what ensures that truly novel architectures are generated?\n\n2. **Variance and Diversity:**  \n   How do you explain the observed low variance in the generated architectures? Could this be an indication that the method is overly focused on high-performing designs, thereby potentially limiting diversity and innovation?\n\n3. **Comparison with DiffusionNAG:**  \n   DiffusionNAG uses a dataset-conditioned accuracy predictor, while your approach removes this predictor. How does your method ensure that it generates high-performing architectures for unseen datasets? Is it possible that the performance gains are primarily due to the fact that the training architectures perform similarly well on unseen datasets?\n\n4. **Incorporating Dataset Information:**  \n   What are the challenges in integrating dataset-specific information into your generation process? It seems that blindly generating architectures may be effective only if the pretrained set is highly representative. Could you discuss the potential difficulties or limitations in incorporating dataset conditioning into your approach?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x7YpgrQevP", "reviewer": "Reviewer_xYiC", "summary": "The authors propose a Neural Architecture Generation (NAG) method based on Prior knowledGe (PG, for PG-NAG). NAG techniques extend Neural Architecture Search (NAS) to discover the features such as operations and subgraphs that contribute to high performance. This is achieved using Shapley values. Specifically, importance's are initialized as the prior knowledge of the diffusion model and the Shapley values from the top-20 architectures are used to initalize prior knowledge. PG-NAG is evaluated on several benchmarks such as NAS-Bench-101, NATS-Bench, TransNAS-Bench, etc.\n\n**Post-Rebuttal Response**\nI thank the authors for their detailed response. While my score will remain as weak reject, I quote and emphasize the direct language of how ICML perceives such a score: \"Weak reject (i.e., leaning towards reject, but could also be accepted)\"\n\n*Reasons to raise score*\n- The rebuttal is pretty well-detailed and provides good factual clarification on the workings of the paper. \n- The justification for the performance drop of PG-NAG compared to other techniques is mostly that PG-NAG is faster/more efficient and the authors emphasize this. They are encouraged to place further emphasis on this advantage in the paper, e.g., BOHB taking 12k seconds per Table 2.\n \n*Reasons to lower score*\n- The rebuttal's key weakness is the response regarding Table 1. To quote the rebuttal \"We apologize for the confusion. We report the best performance, selecting the top architecture from five runs of PG-NAG.\" That the authors are showing the best result for **their** method yet the average result for **other** methods is bizarre and conspicuous. As the main result for the paper, it is literally an apples-to-oranges comparison that cannot be overlooked.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the recognition of our effective evaluation. We are also grateful for the valuable feedback. \n## Theoretical Claims\n>Visual examples\n\nWe add a Flowchart (_https://anonymous.4open.science/r/PGNAG/flowchart.png_).\n>Concerns about prior knowledge\n\nPG-NAG aims to train high-performance architectures from benchmarks and transfer the knowledge to generate architectures in other datasets. Experiments show that if we select architectures from a narrow performance range, the generated architectures still achieve competitive results (weakness 1 in Reviewer PwDK). This demonstrates that PG-NAG can learn the design principles.\n\n## Experimental Designs\n>Table 1\n\nWe apologize for the confusion. We report the best performance, selecting the top architecture from five runs of PG-NAG.\n>Table 2\n\nPG-NAG only lose to BOHB on CIFAR-100 and ImageNet16-120. However, it is important to note that PG-NAG runs $2.89$x faster than BOHB, i.e., BOHB requires about $12,000$ seconds yet PG-NAG only takes $4,147$ seconds to generate the high-performance architecture.\n>Table 3\n\nThough PG-NAG does not outperform other methods on every task, it achieves the highest average rank among all methods. Meanwhile, PG-NAG also shows significant gains on Class Object and Room Layout. The results demonstrate the generalization capability of it across diverse tasks.\n>Table 5\n\nAlthough PG-NAG shows a marginal 0.01% lower max performance compared to Random and DiffusionNAG, it is important to note that DiffusionNAG requires querying an architecture on the target dataset, while PG-NAG does not. Thereby enhancing the generalizability of PG-NAG. Compared to random, PG-NAG has higher minimum and mean values, and can consistently generate high-performance architectures.\n## Supplementary Material\nIt's important to note that DiffusionNAG and L2NAS require querying additional architectures on the target dataset, while PG-NAG does not. Specifically, DiffusionNAG queries one architecture, and L2NAS queries $1,000$ architectures. The extra querying demands additional computational resources and reduces the generalization capabilities. To compare, we utilize the same number of queried architectures with the two methods in PG-NAG. The accuracies are 94.31% for NAS-Bench-101, 94.37%, and 47.31% for CIFAR-10 and ImageNet-16-120 in NAS-Bench-201, which are higher than that of DiffusionNAG and L2NAS.\n## Essential References\nWe added the two methods in the new manuscript. GA-NAS has the same accuracy as PG-NAG, while NAS-Bowl underperforms at 94.2% accuracy compared to PG-NAG. However, GA-NAS requires an additional 150 architectures in NAS-Bench-101 compared to PG-NAG.\n## Weakness\n>Presentation quality\n\nThank you and we will thoroughly polish the manuscript. Subsequently, we provide a detailed explanation of the computation process for Shapley values and how the PG-NAG works.\n* Shapley values: Shapley values are used to quantify the importance of each operation to overall performance. We provide a detailed flowchart of this computation process in Figure (_https://anonymous.4open.science/r/PGNAG/shapley%20value.png_). Specifically, we remove a specific operation and measure the performance difference compared to the original architecture. This performance drop reflects the marginal contribution of the operation, the view of the performance drop is visualized in Appendix Figure 7. After computing Shapley values, we further visualize the relationships among these operations through heatmaps for NAS-Bench-101 (_https://anonymous.4open.science/r/PGNAG/Shapley%20values/NAS-Bench-101.pdf_), NAS-Bench-201 (_https://anonymous.4open.science/r/PGNAG/Shapley%20values/NAS-Bench-201.pdf_), and NAS-Bench-301(_https://anonymous.4open.science/r/PGNAG/Shapley%20values/NAS-Bench-301.pdf_). \n\n* Visualization of PG-NAG: We add a flowchart (https://anonymous.4open.science/r/PGNAG/flowchart.png). It begins with constraints of search space. For example, architectures are constrained to have six operations and four nodes in NAS-Bench-NLP. Then a diffusion model is trained for architecture generation. This diffusion model begins with extracting prior knowledge from the three benchmarks using Shapley values, then incorporates this guidance into the model to learn the architecture design principles.\n\n>Format of the manuscript\n\nWe modified all the charts in the manuscript.\n## Question:\nThe influencing factors are as follows:\n* Prior knowledge: The quality of prior knowledge influences the architecture design principles learned by the PG-NAG (weakness 1 of Reviewer PwDK).\n* Noise schedule: Different noise schedules in the diffusion model affect the generation of architectures (weakness 2 of Reviewer PwDK).\n* Learning methods for operations and connections: We utilize GCN to learn the features of operations in the diffusion model. Cause the features of connections are hard to learn, we learn the features of subgraphs instead. The ablation study is in Table 6 in the main text.", "claims_and_evidence": "A claim of this paper is that search spaces are large, necessity smarter techniques for performance evaluation. While it is indeed true that large search spaces prohibit evaluation, there are existing techniques that overcome this burden.", "methods_and_evaluation_criteria": "The method is trained and evaluated on several standard benchmarks like DARTS, etc. The evaluation experimental setup raises no alarms.", "theoretical_claims": "This is difficult to evaluate as the paper's methodology is not well-written and difficult to understand, even after performing multiple passes over the methodology section. Specifically, better visual examples would help a lot. \n\nOne idea the reviewer takes issues with is that this method relies on knowing which architectures are good ahead of time, and which are bad. PG-NAG trains using only a small subset of cherry-picked architectures.", "experimental_designs_or_analyses": "Experimental design is sound but not noteworthy.\nA primary weakness of this paper is that its results are not even incremental. The method does not provide clear and convincing proof of its efficacy to merit acceptance. Some examples:\n- Table 1: Line 260-263 (right column) states \"To validate the effectiveness of PG-NAG in five new search spaces, we design five architectures for each search space\", yet Table 1 only features one result from PG-NAG for ImageNet or CIFAR-10. Is it a mean value? A max?\n- Table 2 method loses on CIFAR-100 ad ImageNet16-120.\n- Table 3 the method loses on every task except Class Object and Room Layout\n- Table 5 the method loses compared to Random and DiffusionNAG in terms of max performance.", "relation_to_broader_scientific_literature": "PG-NAG extends the idea of directly generating or building high-performance architectures from a search space as opposed to using a search algorithm. There are several related works in this field such as DiffusionNAG [1], a primary related work that is cites. There is also AutoBuild [2], which is cited but not compared to (different benchmarks). This paper also lightly deals with the concept of finding the good/bad architecture subcomponents, thus falls under the banner of interpretable NAS, like AutoBuild and others.", "essential_references_not_discussed": "For NAS-Bench-101 comparison in Table 10, a missing related work would be GA-NAS [3] which also obtains 94.23%. \nFor the idea of finding relevant architecture subcomponents you have NAS-Bowl [4] which is not cited.", "other_strengths_and_weaknesses": "Presentation quality of this paper is definitively below the bar for an A* conference like ICML. As stated before the method is hard to understand and grasp in key sections, e.g., computation of Shapley values and `visualizing` how the method actually works.\n\nAdditionally, float formatting leaves much to be desired. Fig. 1/2 and Tab. 1 in the main manuscript are decently formatted, but the rest are not, e.g., either too small scalebox, too small text.", "other_comments_or_suggestions": "References: \n\n[1] \"DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models\" - ICLR'24\n\n[2] \"Building Optimal Neural Architectures using Interpretable Knowledge\" - CVPR'24\n\n[3] \"Generative Adversarial Neural Architecture Search\" - IJCAI-21\n\n[4] \"Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels\" - ICLR'21", "questions_for_authors": "What factors influence architecture selection criteria exactly?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Three reviewers recommend acceptance (2 Accept and 1 Weak Accept) and one rejection (Weak Reject). The reviewers acknowledge the effectiveness, efficiency, and generality of the proposed method, and the extensive experiments. However, they raised some concerns about the justification of the method and the chosen metrics, the limited discussion of potential limitations, the fairness of the comparisons in Table 1. This latter point was raised by Reviewer xYiC, but the authors further clarified it in a private answer to the AC because they were at that point unable to respond to the reviewer directly. Ultimately, despite recommending a Weak Reject, Reviewer xYiC acknowledges that this paper could be accepted. The balance leans even further towards acceptance considering the authors' explanations of the results in Table 1 in the private comment to AC. As such, the AC recommends acceptance but strongly encourages the authors to include elements of their feedback, including the clarification of the comparison fairness, in the final version.
Decision: Accept (poster)