Multi-conditioned Graph Diffusion for Neural Architecture Search
Authors: Rohan Asthana
Joschua Conrad
Youssef Dawoud
Maurits Ortmanns
Vasileios Belagiannis
Venue: Accepted by TMLR ()

Abstract: Neural architecture search automates the design of neural network architectures usually by exploring a large and thus complex architecture search space. To advance the architecture search, we present a graph diffusion-based NAS approach that uses discrete conditional graph diffusion processes to generate high-performing neural network architectures. We then propose a multi-conditioned classifier-free guidance approach applied to graph diffusion networks to jointly impose constraints such as high accuracy and low hardware latency. Unlike the related work, our method is completely differentiable and requires only a single model training. In our evaluations, we show promising results on six standard benchmarks, yielding novel and unique architectures at a fast speed, i.e. less than 0.2 seconds per architecture. Furthermore, we demonstrate the generalisability and efficiency of our method through experiments on ImageNet dataset.

Published in Transactions on Machine Learning Research (03/2024)
Multi-conditioned Graph Diffusion for Neural Architecture Search
Rohan Asthana
rohan.asthana@fou..d
Priedrich- Aleindhe-eedde----eeee Universität Brlangen- Belangen-Nümmberg Brlangen, Germany
Joschua Conrad
joschua. cow..oammmmmmmmmmmmmmmde de
Universität Ulm
Ulm, Germany
Youssef Dawoud
youssef.dwwoul@@oo..d
Meedrich-Alezandee Universität Brlangen-NW berbbby Brlangen, Germany
Maurits Ortmanns
maurits..rrtmarnn@@mmiii-
Universität Ul
Ulm, Germany
Vasileios Belagiannis
musilcios.belagiannniii de
Priedrich-Alezander Universiläl Eetangen-Nürmbeyy
Erlangen, Germany
Reviewed OpenReview: https: openreview. net/ str/orrm?? id= id=5totyS
Abstract
Neural architecture search automates the design of neural network architectures usually by exploring a large and thus complex architecture search space. To advance the architecture search, we present graph diffusion-based NAS approach that uses discrete conditional graph diffusion processes to generate high-performing neural network architectures. We then propose multi-conditioned classifier-fres guidance approach applied to graph diffu- sion networks to jointly impose constraints such as high accuracy and low hardware latency. Unlike the related work, our method is completely differentiable and requires only single model training. In our evaluations, we show promising results on six standard benchmarks, yielding novel and unique architectures at a fast speed, i.e. less than 0.2 seconds per ar- chitccture. Furthermore, we demonstrate the gencralisability and efficiency of our method through experiments on ImageNet dataset.
1
Introduction
The design of neural network architectures has been normally manual and time-consuming task, re- quiring domain expertise and trial-and-error experimentation EEssee et. al. 2019). Neural Architecture Search (NAS) addresses this limitation by leveraging data-driven methods to automatically search for well-
performing neural network architectures (Liu al. 2019: Howard et al.] 2019 Pham et al.) (2088). Existing works in NAS mostly represent the architectures graphs and include search based methods (Li & Talwalkar 2020
White et al. 2021b), reinforcement learning (Zoph & Le 2017 Tian et. al.] 2020), and evolution-basee

---

Published in Transactions on Machine Learning Research (03/2024)
G1
Architacture latent space
Archilecture latent space
Performance metrics (e.g. Accuracy, Latency)
Performance conditions (e.g. Accuracy-top5% Latency=2)
Valid architectures
GN
Training
Sampling
Figure 1: Overview of our approach. We train discrete graph diffusion model (denoted (D) valid architectures from the architecture search space along with their performance metries (eg. accuracy, latency). After training, we sample architectures given the required conditions (eg. accuracy>top5%, latency<2).
approaches (Real et al. 2019: Chu et al. 2020 However, the large size of the architecture search space makes challenging for these methods to search for high-performing topologies.
To accelerate the architecture search, generative methods reduce the search queries by learning the archi- tecture search space and optimising the latent space from which generator network draws architectures (Rezaei et al.] 2021 Huang & Chu 2021). These methods not only enhance the efficiency but also capture intricate architecture distributions, generating novel architectures. However, the choice of graph generative model significantly impacts the NAS search time. The existing methods employ complex GAN-based gener- ators [Rezaci et al. 2021) or use computationally intensive supernets (Huang & Chu 2021). . More recently, An et al. (2023) employs conditional diffusion processes guided by a classifier while Lukasik et al. (2022) uses a simple generator paired with a surrogate predictor. However, these methods require separate pre- dictor networks for the generated architecture performance. Hence, we present a diffusion- based generative approach that is completely differentiable and thus training involves only single model. As result, we reach promising performance with much smaller search lime.
Denoising difffusion probabilistic models (DDPMs) (Ho et al. 2020) have recently gained attention because of their ability to effectively model complex data distributions through an iterative denoising process. DDPMs offer precise generative control, improving distribution coverage compared to other generative models (Dhari- wal & Nichol 2021). This characteristic makes diffusion models particularly appealing for NAS, as they fulfil the requirement to generate neural network architectures and eventually facilitate the exploration of the search space. In addition to their superior performance, diffusion models excel in conditional generation through the classifier-free guidance technique (Ho & Salimans 2021 This technique enables the condition- ing of diffiusion models on a specific target class, allowing the model to generate samples belonging to that class without utilizing the gradients of an external classifier. Previous studies show that along with image synthesis, classifier- free guidance works well in molecule (graph) synthesis using graph diffusion networks (Hoogeboom et al. 2022). Morcover, Giambi & Lisanti (2023) demonstrated the capability of classificr-free guidance approach using multiple conditions. Motivated by this idea, we present a mulli-conditioned graph diffusion model in which constraints such as high model accuracy and low latency jointly contribute to architecture sampling.
We introduce a graph diffusion-based NAS approach (DiNAS), depicted in Figure | that utilises discrete conditional graph diffusion processes to generate high-perforning neural network architectures? We leverage classifier-free free (CF) guidance, initially developed for image tasks, and extend its application to graph models. Additionally, to impose multiple constraints, we utilize a multi-conditioned CF guidance technique, and apply it within our graph diffusion framework. To demonstrate the effectiveness of our proposed method, perform extensive evaluations on six standard benchmarks, including experiments ImageNet (Deng
The code for our paper available httpss///gtthbb..cmmm/ http://ghhh...c///hchaaaaaaaaaaa
2

---

Published in Transactions on Machine Learning Research (03/2024)
et al. 2009], and ablation studies to demonstrate state-of-the-art performance and faster generation rate lless than 0.2 seconds per architecture knowledge, this is the first formulation of NAS using contii-ondd graph-based diffusion models. In summary, we claim that guided graph diffusion, specifically discrete graph diffusion with multi-conditioned NAS approach, should work better than previous generative and tradit NAS methods due to the model's ability to perform architecture generation in a controlled guided manner without the need of an external predictor. Our claims are supported by empirical evidence detailed in Section Our contributions are as follows:
single GPU) compared to the prior work. To the best of our
classfferr-rre-ggiddan----eee
We introduce a differentiable generative NAS method, which employs discrete conditional diffusion processes to learn the architecture latent space by training a single model.
We propose a multi-conditioned diffusion guidance technique for graph diffusion networks, effectively applied within our NAS approach.
We demonstrate promising results in six standard benchmarks while using less same number of queries with rapid generation of novel and unique high-performing architectures.
2
Related Work
Neural Architecture Scarch (NAS) Automating the neural network architoctural design has gained
substantial interest in the past. few years (Liu et al.) 2019 Jin et al. 2019 Zoph al. 2018: Bender et. al., Shala et al. )023). A straightforward approach is to randomly select and evaluate architectures from the scarch space Li & Talwalkar 2020). However, the lack of optimisation in the scarch space makes this approach inefficient. To address this limitation, carlier works rely on reinforcement learning (Zoph & Le. 2017 Baker et al. 2017 Franke et al.] [022]] to discover well-performing architectures. Gradient- approaches (Brock et al. 2018 Chen et al. 2021b; Yang et al. 2020) employ gradient- bbaeed optimisation, while evolutionary methods (Real et al. 2019 2017) deploy evolutionary algorithms to perform the search. Although these approaches exhibit faster search pace than random search due to the optimisation, they are still regarded slow in scarching high-performing architectures (Liu et al. 2019 Another major challenge with bearch-based methods is the requirement to train networks at each iteralion Luo et al. 2022). This becomes particularly problematic when NAS approaches require substantial mumber of iterations to generate well- performing architectures, which is often the case with reinforcement learning-basse
2018
methods. This issue is solved by the recently developed generative mothods (Lukasik et al., 2021 Rezaci et al. 2021 Lukasik eet al. 2022 An et al.] (223),, which reduce the search time by learning the architecture search space. The generative NAS method by Lukasik et al. (2022) utilises generator, a surrogate predictor, paired with a latent space optimisation technique to generate high performing architectures represented as graphs. This generation is performed node by node, which requires multiple passes of graph neural network, with each pass to generate a node. In contrast, our approach learns and generates all the nodes and the edges of the whole graph (as a representation of architectures) together using a diffusion model, effectively reducing the generation time. While the method proposed by An et al. (2023) also utilises diffusion models for neural architecture synthesis, there are several key differences to note. First,, unlike their approach, we employ discrete graph diffusion instead of continuous graph diffusion. This allows our model to retain the structural information and sparsity of the graphs (Vignac et al., 2023). Second, we eliminate the need of an external predictor to predict the accuracies from noisy data, as done by An et al. (2023), which allows our model to omit the dependency on noisy data classification (Ho & Salimans, 2021). Following the same direction, we present a generative model that reduces the search time compared to the prior work, while minimising the performance loss.
Diffusion Models Although the original idea of data generation through diffusion goes back several years Dichl-icktttineeec et al. 22015. diffusion models later gained popularity for image (Ho et al.) 2020 Rombach 2022 Saharia et al. 2022)), text (Austin et al., (021)) and more recently graph generation (Wang et al. 2022). The ability of diffusion models effectively synthesise graphs motivates our work to formulate NAS as graph generation problem. Nevertheless, the generation of well-performing architectures requires
ct al.
3

---

Published in Transactions on Machine Learning Research (03/2024)
conditional generation through guidance, e.g. by specifying the minimum architecture accuracy. Hence utilize formulation of classifier-free guidance for graph-diffusion networks. Then, we introduce multi- conditioned graph-diftusion approach that accounts for several constraints in the architecture generation.
3
Background
3.1 Denoising Diffusion Probabilistic Models
Denoising Diffusion Probabilistic Models (DDPMs) (Ho et. al. 2020) comprise two fundamental processes, namely, forward and reverse processes. The forward process sequentially corrupts the data sample x using noise model g that follows Gaussian distribution until x reaches state of pure noisc. The noisy variants of x are denoted as xx,,x?,.... x²)), where represents the total number of corruption steps. Subsequently, the reverse process involves learning a denoising model represented as a deep neural network do with parameters to estimate the noise state of sample x at time step t- 1, 1, iee. xf=1 given the current state x'. This is achieved using scoring function that maximises the likelihood of x Formally, the scoring function is defined SF x log Po (x(xx=((xx) which corresponds to the gradient of the log-likelihood with respect to state _ Following the network training, another data point can be sampled from noisy prior (denoted as z. )²), and by iterative denoising the data point (i.e. predicting z from z²), sample z is obtained, which corresponds to the original data distribution. This process is referred to as the sampling process.
Diffusion models can generate high-quality samples from complex data distributions. However, in our task, we do not intend lo sample from the entire distribution but a subset of containing low latency architectures. Hence, diffusion models need be modified incorporate conditioning. This can be achieved using conditional diffusion models.
and/or or
3.2 Conditional diffusion models with guidance
The conditional diffusion model estimates the distribution (x (ax"=([p' ()) From Bayes rule, we have:
po pxxx po(x", ,u|x (xxx==))((="> (x(x
(1)
To ensure the balance between sampling diversity and quality, generative models can incorporate guidance scale 7, modifying the Eq. to:
(a(x 1x",,) v) x px(x', poxx""lx'"> -
(2)
Specifically, increasing 7 sharpens the distribution which favors enhanced sample quality at the expense of sample diversity during the sampling process, referred to as guidance in diffusion models. To guide a diffusion model for labelled data, the model is conditioned on the classification target and the score function Vx--1 (x' y) is computed. Dhariwal Nichol (2021) approach this problem using an external classifier (parameterised by v) where the score function SF modified to include the gradients of the classifier. The reformulated score function is then the weighted sum of the unconditional score function and the conditioning term obtained by the classifier, defined as:
V x Log |x' v) xx--1 lx + V 1oge-loogp (y/x'
While we have successfully expressed the reverse denoising process as a weighted sum of two score functions, estimation of the conditional score function requires training a separate classifier. Moreover, calculating log (y/px requires inferring y from noisy data x'. Although feeding noisy data the classifier yields decent performance, it disrupts the robustness of the model since it ignores most of the original input signal. To address this issue, Ho & Salimans (2021) came up with the classifier-fre freffe guidance, which develops the classifier using the generative model itself. In this case, the score function is defined as:
x 1o--loggoo 2o, (x' (x'-1 [x", (x',)) V log 1- (x V log 1o''p(x'-1
(4)
4

---

Published in Transactions on Machine Learning Research (03/2024)
Eq. demonstrates that it is possible to achieve the same behaviour as the classifier-based guidance without explicitly using classifier. This is achieved through weighted sum, specifically barycentric combination, of the conditional and unconditional score functions.
3.3 Discrete Graph Diffusion
Diffusion models typically work in continuous space and apply Gaussian noise to the data (Ho et al. 2020;
Saharia el al.] 2022). Training diffusion model to generale graphs in the same manner, however, leads lo the loss of graph sparsity and structural information. DiGress, a discrete diffusion approach proposed by Vignac et al. (2023), addressos this problem with a Markov processes as discrete noise model. In this case, the graph comprises of nodes and edges, both being categorical variables, and the goal to progressively add or remove edges as well as change graph node categories. Hence, the diffusion process applied on the node categories X and edges E. Eventually, this model solves a simple classification task for nodes and edges instead of complex distribution learning task, normally performed in generative models like VAEs [Kingma & Welling, 2013) or DDPMs (Ho et al.] 2020). Our approach, in principle, follows DiGress to generate graphs which correspond to neural network architectures.
At each forward step, discrete marginal noise is added to both X and E using the transition probability matrices Qx and QE respectively, which incorporate the marginal distributions m'y and m'g- We select the noisy prior distribution such that it close the original data distribution. Then, the transition matrices are defined follows:
(1= Q' = (1=
(5)
where 7 is the identity matrix, 1, and 1 are the indicator functions, t is the time-step, and a' is the cosine schedule defined as a² = cos(0..5mt/ (t/T + s1((1 +1++1++000 s))² with close to O.
Training For the reverse (denoising) step, a Graph Transformer network Φv, parameterised by 0, is em- ployed. This network learns the mapping between the noisy graphs G and the corresponding clean graphs G. During training, thg can take noisy graphs at any time step / E (1, 7) to predict the clean graph. The loss functions for and E are based on the cross-entropy between their respective predicted probabilities g = (fx, ,²) and the ground-truth graph G (X, E). The total loss then, weighted sum of node-level and edge-level losses, which is given by:
La(px. x. x. E)=
Σ (E Σ CE(e,,
(6)
ISiS
where CE is the cross-cntropy loss function, is parameter to weight the importance of nodes and edges and, is the number of nodes.
Sampling Let the posterior distribution be pa. We start from noisy prior distribution GT ~ (ex (nr) x ge(wr)) where ny is sampled from the node distribution in the training data. Then, we estimate the node and edge distributions po po(x/-1/'') and po(et- using the predicted probabilities fx and p This can written as:
(
G GG)
Σ po(x 12, G'))(()); pele xeX
Σ Pole e eEE
e, G))
(7)
Finally, sampling new graphs can be seen as iteratively estimating the distribution po G') until a clean graph G obtained. Po po(G"=1(G') can be seen as the product of the node and edge distributions marginalisod over prodictions from the network Φo:
Po G G
II p
II po(c)"" 15t.5n
1545n
5

---

Published in Transactions on Machine Learning Research (03/2024)
The above model successfully handles sparse categorical graph data in discrete manner, synthesising graphs from complex data distributions. Therefore, it is suitable for our problem. helesth in our task, we seek to introduce conditioning in discrete graph diffusion models through classifier-free (CF) guidance. To that end, we propose next a multi-conditioned graph diffusion formulation for NAS.
4 Method
Consider the diffusion model qD comprising of a neural network Φe paramterised by 0. During training, the model @o takes the directed acyclic graph G as input and learns to reconstruct G from the noisy version G³, where & E (I) is the number of diffusion time steps. This reconstruction is essentially performed by learning to estimate the actual data distribution 9 from the noisy version of G, which we denote as PN. through iterative denoising. Following the training of Φe, we aim generate DAGs representing high- performing neural network architectures using samples from PN, where we denote sample as z.
Our directed acyclic graph (DAG) representation of architectures follows the standard cell-based NAS search spaces (Liu et al., 2019: Klyuchnikov et al., 2020), where each cell is a DAG. G consists of a set of nodes and edges. The sequence of nodes in G is represented by X= [11, 02..... U,l, where the number of nodes is n, and the edges as the adjacency matrix E of shape (n. Hence, each DAG represented by G EX, Each node is a categorical variable, describing operations, e.g., 1xl convolution, while each edge is a binary variable, specifying the presence or absence of the connection between nodes. In addition, G maps to the ground-truth performance metrics P e.g., the accuracy and latency of each DAG.
Our objective is twofold, namely to generate valid cells C. = Ev) from the latent variable z, sampled from the noise distribution PN and, second, to learn the mapping between the valid cell C, and its corre- sponding performance metrics P. The learned mapping is then used to generate high-performing cells with accuracy close to the maximum achievable accuracy or cells with latency below certain latency constraint. Note that a coll is valid when the corresponding DAG is connectod and includes a realistic sequence of nodes.
4.1 Diffusion based NAS
We consider the unconditional and conditional graph generation. First, we present the unconditional model that learns to generate valid cells. Since some of the generated cells might have poor performance, we propose the single conditioned and multi-conditioned graph diffusion models to generate just the high-performing cells based metrics like the model accuracy and latency.
4.1.1 Unconditional model
Our unconditional model is based on the discrete denoising graph diffusion model (Vignac et al, 2023 outlined in Section 3.3. The forward process involves adding discrete marginal noise Q' and Q (Eq. 5 to both nodes X and edges E respectively. To perform denoising, we employ the Graph Transformer network do, which is trained to predict clean graphs G from noisy graphs G While this model effectively captures the data distribution for undirected graphs, it lacks the ability to incorporate directional information of DAGs. This directional information depicts the flow of data from input to output in the cells and hence is crucial for generating valid cells. To address this limitation, we integrate into our model the positional encoding technique by Vaswani et al. (2017b). In detail, we add sinusoidal signals of different frequencies to the node features X before passing them through the Graph Transformer da, thereby enhancing the network's capability to consider sequential information.
Despite the ability of our unconditional model in forming valid cells necessary for complete network archi- tectures, our goal lies in generating a particular subset of the learned architecture distribution comprising high-performing cells. To that end, we first condition our model on the accuracy metric.
4.1.2 Conditional model
To achieve the generation of higf-perf architectures, we propose guidance approach, inspired by the classifier-free guidance (Ho & Salimans, 2021), and integrate it to our unconditional graph diffusion model.
6

---

Published in Transactions on Machine Learning Research (03/2024)
Unlike the unconditional model, our conditional model estimates the distribution po( ,y), essentially by computing the score function VG--1 log (G' v) as:
G log log Po,
(G' lo y) = (1-1) G log ( + log
(9)
where y is the guidance scale and y is the target variable. The first term of Eq. 9 corresponds to the unconditional distribution pe Po G') learning, while the second one corresponds to the conditional distri- bution (G y) learning. Following Ho & Salimans (2021), we remove the conditioning information for some forward passes determined by the control parameter This leads to the unconditional training of the network. For the rest forward passes, we keep this information to enable conditional training.
4.1.3 Discretisation of the target variable
Our guidance approach assumes that the target variable y, e.g. accuracy or latency, belongs to finite set such that y E {yi, 32. ] S R, where w the number of possible values of y. However, in our case, y takes continuous values from the real number domain, R+. To address this issue, we split y into d discrete classes based on their value. The choice of the split affects the balance of the class data distribution. In our implementation (Sec. 5.1. we provide information on how we select the number of classes and splits
according to the problem.
4.2 Incorporating Multiple conditions
Next, we introduce multiple conditions to the dilfusion guidance to impose several constraints. Con- sider the unconditional noise model that corrupts the data progressively for time steps. Our objec- tive is to estimate the reverse conditional diffusion process ((G'-I|G* .2.. Mls), given the k indepen- dent conditions V1,32;-... .., Assuming po GG /1, 9/, M.)))))) approximates ((G' G+ yı V....., Uk)); we perform the estimation of reverse conditional diffusion process by computing the score function VG-- logpe, po, (G' U1, ykk as:
V logpo, (G' G yl, m. 9h) (1 7) G log (G
(10)
+ +7V logno 3/ ,..,,..,.
where y is the guidance scale. The derivation is provided in Appendix A. Similar to the standard single- conditioned guidance (Eq. 9), the conditional score function for multi-conditioned guidance can be expressed as weighted sum of conditional and unconditional score function. These score functions can be computed using two forward passos of our network, the unconditional and conditional forward pass.
4.3 Training and Sampling
Training procedure Let ci....,ck denote the metries, which we want to constrain c.g. accuracy or hardware latency. The training procedure starts by randomly selecting the time-step t from the range (1, Subsequently, the performance metrics P = (C1,..., ck) undergo a substitution with a null token for a probability instances. Then, marginal noise is introduced to both and E for a duration time-steps. Next, each of C1, cg is individually processed through distinct embeddings, with the resultant embeddings being included to both X and E. We then apply positional encoding X. Finally, the resultant graph is provided input to Graph Transformer network Φg- This network then generates the denoised graph (X, E) which is used to calculate the loss (Eq. G
We present the training algorithm in Alg. 1
Sampling procedure Let (x) be the constraints desired to be imposed (e.g. = 5%). The sampling procedure is initiated with sampling random noisy graph Gt from the prior distribution (ax (mT) x (r(r))). Next, we apply the positional encoding to X and perform two forward passes of our trained network
Φe. namely the unconditional and conditional pass. In the unconditional pass, (c1 = 0, C2 = 0.... .. C = 0). where 0 is mull token, whereas for the conditional pass, (c1 = é .... Ck ēạ). Then, the score estimates are computed for both functions (f. for conditional and P.. for unconditional). Lastly, calculate the resulting
7

---

Published in Transactions on Machine Learning Research (03/2024)
score by linear combination of the score estimates and sample a less noisy graph G with Eq. and This is iteratively performed to produce the clean graph GO. The sampling algorithm is presented in Alg and implementation details in Appendix Sec. A.2
Algorithm Training DINAS
Input: G =X, EX, C1,,,,,, and 6 t (I,,.,?)
b Sample randomly from (1, (1,...") Conditional dropout to train unconditionally Apply marginal noise for t time steps Append embeddings to nodes and edges b. Add sinusoids to X for positional encoding
Ch, ck f- 0 with probability
G + (XQxx EQ'g C1, Ck..
G' + (G', Emb(ci), Emb(ee))
X - x (X) Φx f + po (G')oq.... CK)
Forward pass D. Calculate loss and optimise (Eq. G
oplimiser.slep((cor X, (,
Algorithm 2 Sampling from DINAS
Input: guidance scale y, and conditions é1, Cx
Sample by number of nodes from training data distribution Sample random graph GO ~ (ax (nT) x (e(Tr)
> Sample from prior distribution
for t= T to 1 do
X + X + Pos
b Add sinusoids to for positional encoding
Px p Ge C1 0,.., Ck
Unconditional forward pass
xx f (G' c1 -k 2x)
Conditional forward pass
x = (1 7)1/ + the
Linear combination of score estimates
8F = (1 -)p + +
Linear combination of score estimates
Calculate Po( and pa (e (1-1 G²)
> Eq.
13
G'-l~ ~ II, Po(x G II, (e 1- G') )
Sample Gt-1 (Eq.
end for
return GO
5
Experiments
We evaluate our approach on six standard benchnarks- encompassing (abular, surrogate, hardware aware benchmarks, and the challenging ImageNet image classification task (Deng al. 2009
5.1
Experimental Setup
Tabular Benchmarks We first consider the tabular benchmarks- NAS-Bench-101 (Ying al. 2019) and NAS-Bench-201 (Dong & Yang 2020) for our experiments. Tabular benchmarks list unique architectures with their corresponding accuracy. We utilise the validation accuracy as performance metrics P. The evaluation protocol follows the established standard (Yan et al. 2020 Wu et al.] 2021) of conducting a search for the maximum validation accuracy within a fixed number of querics and reporting the corresponding test accuracy, both as a mean over runs. Although there are different other factors (like the nature of the algorithm) affecting the search time for different approaches, generally search times are directly proportional to the number of queries, and is thus used
efficiency metric by previous approaches by Lukasik et al.
(2022). Yan et al. (2020) and Wu et al. (2021)
For NAS-Bench-101, we compare our approach with Arch2Vec (Yan et al. 2020 NAO Luo et al. 2018
BANANAS (White et al.) (021aa,, Bayesian Optimisation (Snock et al. 2015). Local Search (White et al.
2021b), Random Search (Li & Talwalkar 2020), Regularised Evolution
Real et al. 2019 WeakNAS (Wu
TTh detailed evaluation protocol for each benchmark can found Appendix A.5

---

Published in Transactions on Machine Learning Research (03/2024)
et al. 2021 and AG-Net
Lukasik et al. 2022 For JAS-Bench-20D our approach is evaluated against
SGNAS (Huang & Chu
2021), GANAS RRezaei et al. 2021 BANANAS, Bayesian Optimisation, Random
Search, AG-Net, TNAS
(Shala et al., 2023), MetaD2A (Lee et al. 2021), 3-DARTS Ye et al. 2022) and
DiffusionNAG (An et al.
2023). The corresponding results are reported in Tables
and
Surrogate Benchmarks Next, we evalu- ate our method on surrogate benchmarks. Surrogate benchmarks operate on signifi- cantly larger search spaces like DARTS (Lin et al. 2019 NAS-Bench-NT.P (Klynch- nikov et al. 2020 and therefore use a sim- ple surrogate predictor to estimate the ground truth accuracy. We perform our experi- ments on two surrogate benchmarks, the MAS-Bench-301 (Siems et al. ( (trained CIFAR-10 (Krizhevsky et al. 2009) DARTS search space and
Table 1: Comparison of results NAS-Bench-101. 'Val' represents the maximum validation accuracy and "Test" rep- resents the corresponding test accuracy, both as a mean over 10 runs. Queries are the number of retrieval attempts for accuracy from the benchmark.
Methods
Val(%)
Test (%)
Queries |
Optimum
95.06
94.32
Arch2vec + RI
94.10
400
Arch2vec + BO
94.05
400
NAO 1
94.66
93.49
192
We report the maximum validation accuracy as a mean over 10 runs, along with the mum- ber of queries and compare our method to the prior work as with NAS-Bench-100 The re- sults are presented in Table
BANANAS
94.73
94.09
192
Local Search
94.57
93.97
192
Random Scarch
94.31
93.61
192
Bayesian Optimisation
94.57
93.96
192
WcakNAS
94.18
200
Regularised Evolution
94.47
93.89
192
Hardware Aware Benchmark Our next evaluation is the Hardware Aware Bench- mark (HW-NAS-Bench) (Li et al.) (221)). HW-NAS-Bench provides hardware informa- tion (e.g. latency) along with the accuracy
AG-Net
94.90
94.18
192
DiNAS (ours)
94.98
94.27
150
for multiple edge devices. We follow the slandard protocol (Lukasik el al. 2022) and report the accuracy of best found architectures for ImageNet classification task given the latency constraint (in milliseconds) mcan over 10 runs along with the number of queries. We also report the fcasibility, which indicates the percentage of generated architectures following the given latency constraint. We compare our approach to Random Search and AG-Net as strong baselines for multiple devices each in multiple latency constraints.
The results are available in Table [I]
Experiments on ImageNet Lastly, we conduct experiments on the large-scale image classification task ImageNet (Deng et al. 2009), following the protocol from Liu et al. (2019) Chen al. (2021a). This involves training and evaluating the best generated architecture from NASBench301 (trained on CIFAR10 image classification task) the ImageNet dataset. We report the top-1 and top-5 errors along with the number of queries, comparing our method to several robust. baselines (e.g. DARTS, TENAS (Chnn et al. 2021a 2018) DrNAS (Chen et al. 2021b). (-DARTS (Yo et al.) 2022), Sweetimator
NASNET-A (Zoph et al.] (Yang et al.) 2023). DARTS ++ Soro & Song 2202),, and AG-Net. To ensure a fair comparison, we report the results of methods with search on CIFAR-10 and evaluation on ImageNet, wherever applicable. We
summarise the results in Table
The discretization of the target variable is essential for our task as it is not sufficient to train our model solely high-peeforming samples due to the low number of high-performing architectures. We empirically found that in our task, d 2 for the accuracy metric has slightly superior performance over other values of d (sce ablation study in Appendix Sec. 5.3.4| and thus, we discretisc the accuracy into two classes. One class includes > fth percentile of accuracy values, while the remaining values belong lo the other class. Using higher values of f for accuracy generates better- pertteriprforinngggg architectures, but they also lead class imbalance, thereby reducing the model performance. We address this issue by modifying f depending on the data availability of the specific benchmark. For generating high performing samples, the model is conditioned to generate the samples belonging to > SsA percentile class for accuracy during
Implementation
9

---

Published in Transactions on Machine Learning Research (03/2024)
Table 2: Comparison of results on NAS-Bench-201 for different datasets. 'Val' represents the maximum validation accuracy and "Test' represents the corresponding test accuracy, both as a mean over 10 runs. Queries/ Gen, are the number of retrieval attempts for accuracy from the benchmark or number of architec- ture generations
Methods
CIFAR-10
CIFAR-100
ImageNett16-122
Queries/ /Gen.
Val(%)
Test (%)
Val(%)
Test (%)
Val(%)
Test(%
Optimum
91.61
94.37
73.49
73.51
46.77
47.31
SGNAS
90.18
93.53
70.28
70.31
44.65
44.98
BANANAS
91.56
94.30
73.49*
73.50
46.65
46.51
192
Bayesian Opt.
91.54
94.22
73.26
73.22
46. 13
46.40
192
Random Search"
91.12
93.89
72.08
72.07
45.97
45,98
192
GANAS
94.34
73.28
46.80
444
AG-Net
91.60
94.37*
73.49*
73.51*
46.64
46.43
192
TNAS
94.37*
73.51
MetaD2A
94.37*
73.34
500
8-DARTS
91.55
94.36
73.49*
73.51
46.37
46.34
DiffusioNNAG
94.37*
73.51
DINAS (ours)
91.61*
94.37*
73.49*
73.51 *
46.66
45.41
192
Table 3: Comparison of results on NAS-Bench-301 (left) and NAS-Bench-NLP (right). 'Val' represents the maximum validation accuracy as a mcan over 10 runs. Querics are the number of retricval attempts for accuracy from the benchnark.
Methods
NAS-Bench-301
NAS-Bench-NI.P
Val(%)
Queries
Val(%)
Querics
BANANAS
94.47
192
95.68
304
Bayesian Opt.
94.71
192
Random Scarcht
94.31
192
95.64
304
Regularised Evolution!
94.75
192
95.66
304
AG-Net
94.79
192
95.95
304
DiNAS (ours) 94.92
100
96.06
304
the sampling process. Specific values for each benchmark can be found the Appendix Sec. A.5 For the metric of latency in the hardware-aware benchmark (Li et al. (2021), we wish to generate high-performing architectures lower than the given latency constraint. To achieve this, we discretize latency into two discrete classes- one below the constraint value and one above the constraint.
5.2 Discussion of Results
Tabular Benchmarks Tables and 2 present empirical evidence of the superior performance of our pro- posed method in the context of tabular benchmarks. Across both tabular benchmarks, our approach con- sistently outperforms the SOTA or converges to optimal validation accuracy. Notably, for NAS-Bench-101, our method concurrently reduces query count by 25%, demonstrating its effectiveness. GANAS exhibits a slightly better test accuracy on ImageNet. in MAS-Bench-201 experiments, which can be explained by the fact that our method scarches for the best architocture in terms of validation accuracy and the best validation accuracy does not necessarily imply best test accuracy. To that end, we found that some of the generated architectures with high validation accuracy correspond to comparatively low test accuracy for the case of ImageNet. This discrepancy is also reflected in the standard deviation values for ImageNet, reported in Table 10
Surrogate Benchmarks Furthermore, the results in Table 3 demonstrate that DINAS excels surrogate benchmarks as well. Our method achieves the SOTA in nearly 50% reduction in queries for NAS-Bench-301 and the same query count Random Search, Bayesian Optimisation, and AG-Net. The results from NAS- -eS-Bench--N.P experiment also
surpassing the performance of previous methods such
10

---

Published in Transactions on Machine Learning Research (03/2024)
Table 4: Comparison of results on HWW-NASBench.. 'Val' represents the maximum validation accuracy as a mean over 10 runs and 'Feas' represents the feasibility considering generations of all the runs. Querics are the number of retrieval attempts for accuracy and latency from the benchmark.
Device
Lat. (ms)
DiNAS
AG-Net
Random
Queries
(%)(%)
Feas. (%)]
Val(%)
Feas. (%)
Val(%)
2
39.44
92.60
39.70
29.00
37.20
200
EdgeGPU
4
43.91
93.20
42.80
29.00
41.70
200
6
45.03
66.35
45.30
64.00
44.90
200
2
34.67
92.80
34.60
28.00
33.90
200
Raspi4
4
43.25
77.80
42.00
47.00
41.90
200
6
44.72
57.70
44.00
56.00
43.20
200
EdgeTPU
1
45.31
48.37
46.40
74.00
45.40
200
2
40.01
97.30
40.90
48.00
38.80
200
Pixel3
4
44.74
82.50
45.30
69.00
43.8
200
6
45.95
78.50
45.70
77.00
45.1
200
Eyeris
1
44.67
78.12
44.50
49.00
43.30
200
FPGA
1
44.53
91.65
43.30
65.00
42.90
200
prove that our approach is not only effective in image classification tasks but also in NLP tasks, proving our approach to be lask-independent.
Hardware-Aware benchmark From Table 11 we can observe that our approach outperforms Random method excels in
Search in most cases and surpassing AG-Net in over half of them. Additionally, feasibility across diverse devices and latency constraints while using the same number of queries compared to AG-Net, proving that our multi-conde conditoondd guidance was indeed able to replicate the behaviour of multiple independent predictors (for accuracy and latency) using single set of hyperparameters.
Image.Net. Lastly, we can observe from Table 5 that our approach demonsstrates competitive performance with low top-1 and top-5 error rates on ImageNet, outperforming robust basclines such DARTS, NASNET. A and TENAS. However, the generations from AG-Net DrNAS (Chen al. 2021b), DARTS ++ Soro & Song 2023), 3-DARTS (Ye et al., 2022), and Sweetimator (Yang et al., 2023 are better performing ImageNet et than our method, with the best pest-performing methods being DrNAS and 3-DARTS. The comparison of our approach to non-generative methods gives our approach a disadvantage due to the unavailability of a dataset with DARTS-slyle normal-reduced cell architectures and ImageNet accuracies. Although the transferability works well in this case, it does not beat the state-of-the-art unfortunately. This experiment thus highlights that the generated architectures from our method possess generalisation capabilitics across different dalasets.
5.3
Ablation studies
5.3.1
Comparison search times
To prove the efficiency of our method, report the training (one-time cost) and maachhgeneration times for the search on CIFAR-100 and ImageNet datasets in the Table G The search times correspond to gen- erating all the architectures in one single run of an experiment. We observe that our approach requires a significantly less search lime compared to previous approaches on ImageNet. On CIFAR100 dataset, our approach outperforms DiffusionNAG An et al. (2023) by a significant margin. It can thus be proven that our approach, without an external classifier, more efficient than prodictor-basee approaches, due to its reduced computational requirement for each generation.
The search time comparisons are concurrent with the protocol from other generative NAS methods by Lukasik et al. (2022) and Huang & Chu 2021 where the scarch times are taken as the generation times of
Note that Lukasik all 2022 refers validation accuracy of NAS-Bench-NLP validation perplexity i "Rssults from Lukasik al. 2022
11

---

Published in Transactions on Machine Learning Research (03/2024)
Table 5: Comparison of results for top-1, top-5 errors on ImageNet.
Methods
Top-1 +
Top-5 +
Queries
NASNET-A
26.0
8.4
20000
DARTS
26.7
8.7
DrNAS
23.7
7.1
TENAS
26.2
8.3
AG-Net
24.1
7.2
304
Sweetimator
24.1
DARTS* aug
24.8
7.8
B-DARTS
23.9
7.0
DINAS (ours)
24.8
7.4
100
Table 6: Comparison of search times in GPU seconds and training times of the pre-trained generator (if applicable) in GPU hours ImageNet and CIFAR 100 (using NAS-Bench-20L) for different approaches. The scarch times correspond to generating all the architectures in one single run of an experiment. Note that we report the mean time over different runs for DINAS.
Dataset
Methods
Search/ Search/Gen. Time (GPU sec.)
Pre-training cost (GPU hrs) +
ImageNet
NASNET-A
1.7x 10
ImageNet
DARTS
345600
ImageNet.
DrNAS
397440
ImageNet
TENAS
4320
ImageNet
AG-Nct
1728
21.6
IetgeNet.
3-DARTS
34560
ImageNet
DARTS
25920
ImageNet
DiNAS (ours)
53.76
16.6
CIFAR100
DiffusionNAG
261
3.43
CIFAR100
DiNAS (ours)
15.36
0.25
the proposed method. The training times Table 0 are considered as one-time cost because the search time of a pre-trained model is the inference cost of the model. Upon training, the pre-trained model can then be used to generale architectures, which also generalise differrnt datasets (see Table 5
5.3.2 Novelty and Uniqueness Analysis
We conduct an analysis of novelty and uniqueness for the generated architectures. We start by generating 2000 and 100,000 architectures respectively based on our proposed method. To assess novelty, we calculate the percentage of generated samples absent in the training data whercas to assess uniqueness, (Zaaang the ratio of architectures present just once in the generations to the total number of generations et al.) 2019) (An et al. 2023). Given the enormous size of DARTS and NAS-Bench-NI.P search spaces, we consider NAS- LeS-Bench-301 and NAS-Bench-NLP for our analysis. Furthermore, to examine the efficiency of our method, we record and report the training times for our method (for 100 epochs), along with the sampling times per architecture using a single NVIDIA A6000 GPU on five different benchmarks. We can observe the results of our ablation studies Table 7 Note that for benchmarks involving multiple cases (e.g. HWNAS and NAS-Bench-201)), we take the mean of the training times for all cases.
calculato
We observe from the results of the novelty analysis for the case of 2000 samples and 100,000 samples that in both the datasets, all the generated samples are novel and most of them are unique, proving that our method is not just selecting the best-performing architectures from the training set. We can also observe that while the novelty for all the cases remain at the top, the uniqueness suffers a bit when sampling high number of architectures, due limited number of possible high-performing architectures. Morcover, we can observe that the sampling rates of each benchmark are milliseconds, proving the rapid generation capabilities of our method.
12

---

Published in Transactions on Machine Learning Research (03/2024)
Table 7: Ablation study on novelty analysis (left) and efficiency analysis (right). Note that 'Nov' represents the novelty ratio, 'Uni' represents the uniqueness ratio and 'Gen' represents the number of architectures generated. The training time is reported in hours and sampling time per architecture in seconds.
Benchmark
Train (hrs))
Sample (sec)
Benchmark
Gen.
Nov.(%)
Uni.(%) (%) t
NB101
0.96
0.09
2000
100
97.37
NB201
0.25
0.08
NAS-Bench-301
100,000
100
91.10
NB301 (Normal)
8.3
0.14
2000
100
97.57
NB301 (Reduced)
8.3
0.14
NAS-Bench-NLP
100,000
100
96.78
NRNLP
0.95
0.15
HWNAS
1.5
0.08
5.3.3 Required number of training samples
This section analyses the performance of our method on DARTS search space (Liu et al. 2019) when trained on different number of samples. In particular, we consider training in three scenarios- on 100,000 samples, 10,000 samples and 1,000 samples, for which, we randomly sample the given number training samples from DARTS search space and follow the same training protocol as our experiments on NAS-Bench. 301. Upon training in cach case, 192 architectures are generated for total of 10 runs and the maximum validation accuracy is reported as mean over these runs. In addition, we caleulate and report the novelty and uniqueness of the generated architectures, calculated using the same methodology considering generations from all the 10 runs. Finally, we report the number of queries. The results for this ablation study are reported in Table
Section 5.3.2
Table 8: Peformance on NAS-Bench-301 (Siems et al. 2021) with different number of samples. The Val. represents the maximum validation accuracy; reported mean over 10 runs, whereas novelty and uniqueness are calculated considering the generations from all the runs. Queries are the number of retrieval attempts for accuracy from the benchmark.
Training Samples 100,000
Val (%). f Novelty(%)
Uniqueness (%) f
Queries
94.92
100
97.37
192
10,000
94.89
100
100
192
1,000
85.29
100
100
192
We observe from Table that our method learns the architectural representation and finds the high- performing architectures with one tenth of the number of training samples as well. However, the mean maximum validation accuracy drops when we further reduce the training samples to 1000. We found out that the reason for this decline was the unavailabilityy of any valid generations in one of the runs. This resulted in the maxinum validation accuracy for that run to be 0, which influenced the mean. From this, we can conclude that the data capture capabilities of our model are compromised when training on a very small number samples. Morcover, we observe that the novelty and uniqueness ratios do not suffer at all when reducing the training data availability.
5.3.4 Number of classes for guidance
The goal of this ablation study is analyse the effect of the number of classes d for accuracy present in the training data our approach. We train and evaluate our method on NAS-Bench-200 (Dong & Yang, 2020) for the task of CIFAR-10 image classification involving four different cases: d = {2, 3, 4, 5). We use the same training and evaluation protocol as experiments in Table 2 The split of the data into classes, denoted of the accuracy. For instance,
as 8T, is performed depending on specific percentiles ʃ J for two classes, f = 95 and the split sT = [95mh - 100g.. On 95,x). while for three classes, f = [80, 95] and ST = [95, 100,,, 80t, 95,,, Ot 80,,). all the cases, we generate architectures belonging to the class of >95h percentile. The choice of empirical as does not affect the samples in the class we want to condition our model (i.e. > 95th percentile). We report the maximum validation accuracy and the
13

---

Published in Transactions on Machine Learning Research (03/2024)
corresponding test accuracy, both as mean over 10 runs, along with the number of queries. Moreover, we report the percentiles f used for splitting. The results for this study are reported in Table 9
Table 9: Comparison of results NAS-Bench-201 for CLEAR-10 when using different number of classes, Here, f represents the percentiles used for splitting the data into classes, 'Val' and 'Test' represent the maximum validation accuracy and the corresponding test accuracy, both represented as a mean over 10 runs. Queries are the number of retrieval attempts for accuracy from the benchmark.
Number of classes (d)
Val(%))
Test(%) )
Queries
2
195
91.61
94.37
192
3
[80, 95]
91.60
94.00
192
4
[50, 80, 95]
91.52
93.79
192
5
300, 50, 80, 95]
91.57
93.89
192
We observe from Table 9 that discretising the accuracy into two classes results in a slightly superior perfor- mance over other values of d. However, the differences are marginal.
6 Conclusion
We presented a generative method to facilitate the search process for neural architectures. Our approach uses conditional graph diffusion model to rapidly generate novel, unique and high-performing neural network architectures. In this context, first formulated the classifier-free guidance for graph diffusion models and then proposed multi-conditioned classifier-free guidance for diffusion models. Unlike the related work, method does not require external surrogate predictor and is thus differentiable. In the experiments, we demonstrated state-of-tae of-tth-arttttto performance in tabular, surrogate and hardware-aware evaluations by consid- ering six standard benchmarks. Furthermore, we have shown the search efficiency of our approach compared to the previous work using ablation studies. We observed that our method samples architectures two orders of magnitude faster than other generative NAS approaches and at least three orders of magnitude faster than classic approaches.
References
Sohyun An, Hayeon Lee, Jachyeong Jo, Seanie Lee, and Sung Ju Hwang. Diffusionnag: Yask-guided neural architecture generation with diffusion models. arXiv preprint ar arXim:2305.1694 2023.
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In M. Ranzato, A. Boygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, Curran Associates, Inc., 2021. URL https: pdf
pp. fatp://000111100000000r008686222
heurips.cc/ppeee httpe//ppppppeseee//////pp
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures
using reinforcement learning. In International Conference Learning Representations, 2017. URL https:
Gabriel Bender, Picter-Jan Kindermans, Barrct Zoph, Vijay Vasudevan, and Quoe Le. Understanding and simplifying one-shot architeclure search. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 550-559. PMLR, 10-15 Jul 2018. URL htttss:/pproceeddngg mlr presss/880/eendr1125 18a.hmml
Andrew Brock, Theo Lim, J.M. Ritchie, and Nick Weston. SMASH: One-shot model architecture search through hypernetworks. In International Conference on Learning Representations, 2018. URL https: //openreview.netee www.:eeeeettootmrririereeeee
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference knowledge discovery and data mining, pp. 785 -94, 2016.
14

---

Published in Transactions on Machine Learning Research (03/2024)
Wuyang Chen. Darts evaluation: Train from scratch for architectures from darts space, 2022. URL https: http:/cco/cccnyy3 Satpeuuyyyn/ARRSS_ealaaaaaaaaa evaluation
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. In International Conference Learning Representations, 2021a.
Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. Drnas: Dirichlet neural architocture search. In International Conference on Learning Representations, 2021b. URL https:
//openreviev.o wttp::/www...e.ooomm.mdda/aaaaaa
Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Multi-objective reinforced evolution in mobile neural archilecture search. In European Conference on Computer Vision, pp. 99 113. Springer, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 20t9//CVR..000.5522..
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 118778--8794 2021.
Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020.
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. AAAI Workshop on Deep Learning on Graphs: Methods and Applications, 2021.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019. URL http:://mmm http:///ww...rr.pppppe////////// html
Jörg K.H. Franke, Gregor Koehler, André Biedenkapp, and Frank Hutter. Sample-efficient automated deep reinforcement learning. In International Conference on Learning Representations, 2021. URL https:
//openreview // wttp://tttttooooormiddsssasasaa
Nico Giambi and Giusoppe Lisanti. Conditioning diffusion models via attributes and semantic masks for face generation. arXiv preprinl urxi:2206. 00914, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in newral information processing systems, 33:6840-688 2020.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pp. 8867-8887. PMLR, 2022.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chigh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Scarching for mobilenetv3. Proceedings of the IEEE/CVF international conference on computer vision, pp. 1314-1324, 2019.
Sian- Yao Huang and Wei-Ta Chu. Searching by generating: Flexible and efficient one-shot nas with architec- ture generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 983-9992-9 03-9992,44 2021.
Haifeng Jin, Qingquan Song, and Xia Hu. Auto-keras: An efficient neural architecture search system. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1946-1956, 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1r16::1312.6111, 2013.
15

---

Published in Transactions on Machine Learning Research (03/2024)
Nikita Klyuchnikov, Πya Trofimov, E. Artemova, Mikhail Salnikov, Maxim Fedorov, and Evgeny Burnaev. Nas-bench-nlp: Neural architecture search benchmark for natural language processing. IEEE Access, PP: 1 1-1, 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Hayeon Lee, Eunyoung Hyung, and Sung Ju Hwang. Rapid neural architecture search by learning to generate graphs from datasets. In International Conference on Learning Representations, 2021.
Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, Cong Hao, and Yingyan Lin. (HW)-(nas)-bench Hardware-awer aware aware neural architecture search benchmark. In International Conference on Learning Representations, 2021.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in artificial intelligence, pp. 367-377. PMLR, 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARIS: Differentiable architecture search. In Interna- tional Conference on Learning Representations, 2019.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. are preprint Ait::1711.05001. 2017.
Jovita Lukasik, David Friede, Arber Zela, Frank Hutter, and Margret Keuper. Smooth variational graph embeddings for efficient neural architecture search. In 2021 International Joint Conference Newral Networks (IJCNN) pp. 1-8. IEEE, 2021.
Jovita Lukasík, Steffen Jung, and Margret Keuper. Learning where to look generative nas is surprisingly efficient. In European Conference on Computer Vision, pp. 257-273. Springer, 2022.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural archilecture optimization. Advances in neural information processing systems, 31, 2018.
Xiangzhong Luo, Di Liu, Hao Kong, Shuo Huai, Hui Chen, and Weichen Liu. Surgenas: comprehensive surgery on hardware-aware differentiable neural architecture search. IEEE Transactions on Computers, 72(4):1081 1094, 2022.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated cor- pus of English: The Penn Treebank. Compulational Linguislics, 19(2):313 33030,,, 1993. URL https:
//aclanthology.oo hrwpchhoooogoooo/J//// Jr3/
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficientt neural architecture search via parameters sharing. In International conference machine learning, pp. 4095 4104. PMLR, 2018.
Esteban Real, Sherry Moore, Andrew Sclle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2902-22222 2911. PMLR, 06-11 Aug 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the anai conference on artificial intelligence, volume 33, pp. 4780- 4789, 2019.
Seyed Saeed Changiz Rezaci, Fred X Han, Di Niu, Mohammad Salameh, Keith Mills, Shuo Lian, Wei Lu, and Shangling Jui. Generative adversarial neural architecture search. arXio preprint orXin:2105.0080 05300356 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esscr, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
16

---

Published in Transactions on Machine Learning Research (03/2024)
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 11334799-34994 2022.
Gresa Shala, Thomas Elsken, Frank Hutter, and Josif Grabocka. Transfer NAS with meta-learned bayesian surrogates. In The Eleventh International Conference on Learning Representations, 2023. URL https:
//openreviex. peepareviee.me//forumnn htte://rrmmmiipparrrr1aaaa.
Julien Niklas Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. {NAS}- bench-301 and the case for surrogate benchmarks for neural architecture search, 2021.
Jasper Snock, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International conference on machine learning, pp. 2171- 2180. PMLR, 2015.
Jascha Sohl-Dickstein, Eric Weiss, Niru Mahoswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265.
PMLR, 2015.
Bedionita Soro and Chong Song. Enhancing differentiable architecture search: A study small number of cell blocks in the search stage, and important branches-basee cells selection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 1253-1261, October 2023.
Yuan Tian, Qin Wang, Zhiwu Huang, Wen Li, Dengxin Dai, Minghao Yang, Jun Wang, and Olga Fink. Off-policy reinforcement learning for efficient and effective gan architecture search. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part VII 16, pp. 175-192. Springer, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In 1. Guyon, U. Von Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017a. URL https::/prooceeding neurips.
Paper. .pff
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017b.
Clement Vignac, Tgor Krawezuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. In The Eleventh International Conference on Learning Represenlations, 2023.
Junxiang Wang, Junji Jiang, and Liang Zhao. An invertible graph diffusion neural network for source localization. In Proceedings of the ACM Web Conference 2022, WWW '22 pp. 1058-1069, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450390965. doi:
URL https://doi.org org/10. 1145/3485447 33121555
Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural archilectures for neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10293-10301, 2021a.
Colin White, Sam Nolen, and Yash Savani. Exploring the loss landscape in neural architecture search. Tn Uncertainty in Artificial Intelligence, pp. 654 664. PMLR, 2021b.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardwaro-aware efficient convnct design via differentiable neural architecture search. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 10734-10744 2019.
17

---

Published in Transactions on Machine Learning Research (03/2024)
Junru Wu, Xiyang Dai, Dongdong Chen, Yimpeng Chen, Mengehen Liu, Ye Yu, Zhangyang Wang, Zicheng Liu, Mei Chen, and Lu Yuan. Stronger nas with weaker predictors. Advances in Neural Information Processing Systems, 34:28904 28918, 2021.
Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture representation learning help neural architecture search? Advances in neural information processing systems, 33:12486 12498, 2020.
Shen Yan, Colin White, Yash Savani, and Frank Hutter. Nas-bench-xI1 and the power of learning curves. Advances in Newral Information Processing Systems, 34:22534 22549, 2021a.
Shen Yan, Colin White, Yash Savani, and Frank Hutter. Nas-bench-x11 and the power of learning curves. 2021b.
Advances in Newral Information Processing Systems,
Longxing Yang, Yanxin Fu, Shun Lu, Zihao Sun, Jilin Mei, Wenxiao Zhao, and Yu Hu. Sweet gradient matters: Designing consistent and efficient estimator for zero-shot architecture search. Newral Networks, 168:237 255, 2023. ISSN 0893-6080. doi: https:/ http://oooooooo..1/1 10.16////1111// hat.:////nun....m
Yibo Yang, Hongyang Li, Shan You, Fei Wang, Chen Qian, and Zhouchen Lin. Ista-nas: Efficient and consistent neural architecture search by sparse coding, Advonces in Neural Information Processing Systems, 33:10503 10513, 2020.
Pong Ye, Baopu Li, Yikang Li, Tao Chen, Jiayuan Fan, and Wanli Ouyang. beta-darts: Beta-docay regu- larization for differentiable architecture search. In 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10864-1000 10873. IEEE, 2022.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS-bench- 101: Towards reproducible neural architecture search. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 Proceedings of Machine Learning Research, pp. 7105 7714,, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Garnett, and Yixin Chen. D-vae: A variational autoencoder for directed acyelic graphs. Advances in Neural Information Processing Systems, 32, 2019.
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. In International Confer- ence on Learning Representations, 2017. URL httpss//opearrrree...e wttp:///wwwrtr...ooooooooorooooa
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scal- able image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8697 8710, 2018.
A Appendix
A.1 Proofs and Derivations
Derivation Let q be the unconditional Markovian noise model and be the conditional noising process similar to g. We define aim as decomposing ( G yı, 32, yk) and then deriving the score function
18

---

Published in Transactions on Machine Learning Research (03/2024)
for the multi-conditioned diffusion process. We start by expanding the term:
(G' GO UI, ,/,,..,, y8k
(G' U1, V2. , vb) v2))
(11)
GG' y1, 122 ..... Ukk
((G' G¹ V2.............................. 3/h
(12)
Ky), y2, ..... yh G (G+)
(Vi...... yk G G') G*)
(13)
4(11, M2
Mk G*)
(1v1 ......... yk G G G (G))
(14)
((y) 1/: , ".
((1 yk G GG))
G
(15)
(1V1 V2. ,2...., . (kk)'"))
Dhariwal & Nichol (2021) prove that the classification term (in our case 4(y1,..., yk G does not depend on the noisier version of G (i.e. GO) and be rewritten as ((y)...., ya G Furthermore, they
also show that n behaves the same when not conditioned on the classification targets ",...., 9/. We use these findings to further simplify Eq. 15 to:
Uk) = Alvr,..., yk G - (y"
q( G GG' U1 U2. Uk)
(16)
( V2..... ................................ yk Gt)
We assume that the generative model pa G approximates (G' and the classifier Po (VI....., Mk G approximates ((),,..,, yk G. By substituting the distributions with their approx- imations, taking the logarithm and calculating the gradients w.r.t. Gt-1, we obtain the following score function:
V log.oo p o yi.... yke V log po + V log (y1...., ( G
(17)
where Go and G represent the DAG G at time step t and t t- respectively, vi are the classifier parameters and 8 are the generative model parameters. Similar to the standard classifier- based guidance (Dhariwal & Nichol 2021), we multiply the conditioning term by a factor of y Thus, we can express the reverse denoising process as a weighted sum of the unconditional score function Va log po po (G') and the conditioning term G log (y), (y....., yk (x-1), given by:
Ve-- log v G G GG Mi.... Uk) = V logpo + 1o- log Vhl G
(18)
where > is the guidance scale. Then, by substituting the conditioning term V log Me G
from Eq. I7 and removing the classifier parameters v, we obtain:
V logro, LogPo,(G' (G' G1, Vi,,.,, yk) = (1 - V log Po (G
yi,
(19)
+ + log yy,,..,, Mk).
Hence, as we can observe from Equation 19 we have successfully derived the score function of multi- conditioned diffusion guidance.
A.2 Implementation Details
Our proposed method involves training a Graph Transformer network, proposed by Dwivedi & Bresson (2021), for denoising. This network comprises of an input node/edge wise MLP layer, followed by 5 Graph Transformer layers and, node/edge wise MT.P the output layer. Each Graph Transformer layer consists
19

---

Published in Transactions on Machine Learning Research (03/2024)
Table 10: Mean validation and test accuracies with standard deviation for all runs of JAS-Bench-101. NAS-
Bench-201, NAS-Bench-301 and NASS experiments. StD represents the standard deviation the table.
Experiment
Dataset
Val.
acc.
Test acc.
Mean
StD
Mean
StD
WAS-Bench-LDe
CIFAR-10
94.98
0.169
94.27
0.197
NAS-Bench-201
CIFAR-10
91.61
0.000
94.37
0.184
MAS-Benches
CIFAR-100
73.49
0.000
73.51
0.000
ImageNet
46.66
0.092
45.41
0.589
NAS- AAS-BBnnh-301
CIFAR-10
94.92
0.072
TAS-Bench-NIA.
Penn Tree Bank
96.06
0.173
of three main parts: a self-attention module similar to the one found in the standard Transformer model (Vaswani et al., 2017ah, fully connected layer, and layer normalisation. All the models were trained for 100 epochs with learning rate of 0.0002, batch-size of 16, weight dccay of 10-12, guidance scale of -4 and using Adamn optimiser (Loshchilov & Hutter 2017) single NVIDIA A6000 GPU. For noising, use cosine noise schedule for T' 500 time-steps. The hyperparameters used in our method were derived from the code from Vignac et al. (2023).
For the evaluation on ImageNet, we employ the same training pipeline and code as AG-Net (Lukasik el al. 2022) and TENAS (Chen et al., 2021a), taken from Chen (0022). We train the best generated architecture in terms of validation accuracy from NAS-Bench-301 on ImageNet for 250 epochs. The initial learning rate is set to 0.5 with cosine learning rate scheduler and the batch size is set to 1024. The ImageNet training is performed on 3 NVIDIA V100 GPUs parallelly in a distributed manner.
A.3 Additional Results
Table 10 provides some additional results, mainly reporting the mean and standard deviations over different runs for NAS-Bench-101, NAS-Bench-201, NASSBench-301 and NAS-Bench-NT.P experiments. It. can be observed that the standard deviation (StD.) for test accuracy is generally higher than the StD. for validation accuracy, particularly for NAS-Bench-201 on ImageNet. The reason for this difference is that the model is trained to generate architectures with high validation accuracy and has no test accuracy information. This also explains the discrepancy of the validation accuracy values and test accuracy values for the case of ImageNet in Table 2
A.4 Additional ablation studies
A.4.1 Effect of the guidance scale parameter
We perform an additional ablation study to analyse the effect the guidance scale parameter y on the performance of our model. We train our model on two differently sized search spaces: NAS-Bench-101 (size of 423k samples) and NAS-Bench-NLP (size of 1003 samples) search space using four guidance scales (-4, -2, and 4) in this study and report the mean and standard deviation of the validation accuracy (and test accuracy for (AAS-eench-10)) over 10 runs. The evaluation protocol is identical to the main experiments (Section
Table I1 presents the results of this ablation study. Interestingly, we can observe that negalive values of guidance scale perform better than positive values, unlike Ho & Salimans (2021). This can be attributed to the difference in formulation of Eq. in Ho & Salimans (2021), where the guidance parameter is w, instead of y (w ~ - We found that ? -4 setting produces best results for both, small and large search spaces, and is thus used in our experiments.
20

---

Published in Transactions on Machine Learning Research (03/2024)
Table 11: Comparison of performance of DiNAS using different guidance scales on NAS-Bench-101 and AAS-Bench-NLP search spaces. Val. represents the mean validation accuracy, "Test' represents the mean test accuracy and StD, represents the standard deviation over 10 runs.
7
NAS-Bench 101ch-1011
NAS-Bench-NLP Val(%) #StD.1
(al(%) +St..1
Test(%) SSt..1
-4
94.98
94.27=0.20
96.06=0.17
-2
95.06=00.14 94.16+t0.20
95.97++0.12
2
92.6011..43 92.05:10.58
95.42.10008
4
91.44+0047 90.86++0.59
95.36+00.29
A.5 Evaluation protocols
WAS-Bench 101 and NAS-Bench 201 We generate fixed number of architectures (equal to the respective number of queries) and query them on both the benchmarks to find the maximum validation accuracy and its corresponding test accuracy. This process is repeated 10 times to calculate the mean maximum validation accuracy and mean corresponding test accuracy. Note that we use f = 99 for NAS- Bench-101 experiments and f 95 for NAS-Bench-22 201 experiments.
NAS- AAS-Bench-301 To evaluate our approach on MAS-Bench-301, a random subset of 100,000 architectures is selected from the DARTS search space. As surrogate benchmarks do not provide accuracy, the accuracy of the selected architectures are calculated using pre-trained surrogate predictor XG-Boost (Chen & Guestrin 2016) provided with NAS-Bench-301. Next, the network is trained using normal cells from this dataset, producing 10 normal cells from > fuh class, Next, this process is repeated to produce 10 reduction cells. The ovaluation involves 100 queries, considering all possible combinations of the 10 generated normal and reduction cells. For each query, the highest validation accuracy and its corresponding test accuracy are recorded. This entire process is iterated 10 times, yielding mean values for these recorded accuracies. We use 99 for this benchmark.
NAS-Bench-NLP
Given the enormity of this search space, employ NAS-Bench-X11 Yan et al. 2021b) surrogate prodictor to obtain accuracy for those architectures, trained specifically the Penn TreeBank dataset (Marcus et al. 1993). However, it should be noted that NAS-Bench-XI1 is only capable of handling graphs with up to 12 nodes, which filters our dataset to include the total of 7,258 architectures. After training, we generate 304 architectures and estimate their accuracy using NAS-Bench-X11. The process is repeated 10 times. We set f = for this benchmark.
HW-NAS-Bench For this evaluation, we consider 12 distinct cases for latency and device constraints. Upon each training, our method generates 200 architectures which are then queried on the benchmark. We adopt two conditions simultaneously, namely the accuracy should be in fal(= 95) percentile class and the latency should satisfy the given constraint. We repeat the generation process for 10 runs and report the mean of the validation accuracy, along with the feasibility and number of queries.
ImageNet We start by generating 100 architectures through our approach trained on NAS-Bench-301.
Then, we select the best architecture in terms of validation accuracy. Next, we train the network using the same training pipeline and code as TENAS (Chen et al. 2021a). Finally, we save the weights from the epoch where the top-1 and top-5 validation errors are minimum and report the top-1 and top-5 errors in Table
A.6
Benchmark Descriptions
A.6.1
NAS-Bench-101
NAS-Bench-101 (Ying et al. 2019 is cell-based tabular benchmark, comprising a large collection 423,624 distinct architectures represented cells. These architectures are also mapped their respective validation and test accuracy metries, evaluated on CIFAR-10 image classification task. In this benchmark, the cells
21

---

Published in Transactions on Machine Learning Research (03/2024)
are constrained to have maximum of 7 nodes and 9 edges. Specifically, the first and last nodes within these cells serve as input and output nodes. Intermediate nodes within the cells can take on one of three possible operations: Ixl convolution, 3x3 convolution, or 3x3 max-pooling. Furthermore, it is important to note that each convolutional operation is precoded by batch normalisation, followed by a Rectified Lincar Unit. (ReLU) activation function.
A.6.2 NAS-Bench-209
Another cell-based tabular benchmark is NAS-Bench-201 Dong & Yang. 2020J. which contains data for 15,625 architectures (cells) trained on 3 datasets- CIFAR-10, R-00 Krizhevsky et. al. 2009) and ImageNet16- 120 (Deng et al.) 2009). In contrast to NAS-Bench- 101, each edge of a cell in NAS MAS--Bnch--211 is associated to an operation drawn from a predefined operation set O {1x1 convolution, 3x3 convolution, 3x3 avg pooling, skip, zero}. In our training and experiments on NAS-Bench-201, we convert the edge based representation to node-based representation, where each node is associated with operation, similar to NAS-Bench-101. This conversion is in line with the conversion in Arch2Vec (Yan et al. 2020). Each cell comprises 4 nodes and 6 edges and the adjacency matrices are identical to one another. The existence of operations like zero and skip enforces the structural diversity in different architectures.
A.6.3 NAS-Bench-301
NAS-Bench-301 (Siems al. 2021 is a surrogate benchmark that trains and evaluates several performance predictors on 60,000 sampled architect ures from DARTS search space (Liu el. al., 2019). These learned performance (surrogate) predictors are then able to predict the accuracy of architectures in DARTS search space (comprising 10 architectures). The architectures in DARTS comprise of normal cell and a reduction cell. Each cell has a maximum 3x3 sep. conv., 3x3 dil. conv., 5x5 sep. conv, 3x3 average pooling, identity, zero }.
nodes and 12 edges with each edge associated with an operation drawn
from the set O = { We utilise pretrained XGBoost (Chen & Guestrin 2016) provided by [Siems al. (2021) as the surrogate predictor for our experiments.
A.6.4 NAS-Bench-NLP
NAS- Bench-NLP is the first NAS benchmark designed for Natural Language Processing tasks (Klyuchnikov et al. 2020). While its search space is extremely large with the total of 10a3 architectures, NAS-Bench-NLP provides 14,322 architectures trained on Penn TreeBank dataset (Marcus al.] 1993). Each cell in the scarch space has maximum of 24 nodes, 3 hidden states and linear input vectors. The nodes in each cell depict the operations drawn from the set O = {Linear, blending, product, sum, tanh, sigmoid, LeakyRELU J. We utilise the surrogate predictor provided by NAS-Bench-XL (Yan et al. 2021a for this benchmark.
A.6.5 HW-NAS-Bench
HW-NAS-Bench is unique benchmark that provides hardwarc-speeific details, including latency and energy cost, across various devices along with their respective accuracy. These devices encompass diverse set of hardware platforms, including EdgeGPU, Raspi4, Pixel3, TPg, Fyeriss, Pixe13, and FPGA. Crucially, TN-NASSBeende operates within two distinct search spaces: NAS-Bench-201 (Dong & Yang 2020) and FB- Net (Wu et al. 2019). In our experiments, we utilise latency information as hardware constraint, within the context. of the NAS- BAS-Bench-201 search space.
A.7 Examples of generated cells
Here, we demonstrate some examples the generated normal cells on DARTS search space using our proposed method trained on NAS-Bench-301 (Siems et al. 2021 (demonstrated figure 2 and indicate structural differences compared to cells generated by DARTS (Liu et al., 2019 and TENAS (Chen et al., 2021a) (demonstrated in figure 3 It can be observed that DiNAS generations have a marginally higher 5x5 convolutional connections compared to the other approaches.
22

---

Published in Transactions on Machine Learning Research (03/2024)
sep_comv
3x.3
Akip_comeest
scp Sap_soNV_55 5
max__ool 3x
Xp__omy__S0 Su5
2
(k)
[_-22
skip cocmect
Sep Sep__com coon 5x5
[b1l
skip crnnect
cony 5x5
skip_conmer connect
wpp
[_
sep comy .3x
Figure 2: Examples of high performing generated cells by method DINAS DARTS search space using NAS- -0S-ennc--
sep_comv___ 3x3
[_(k]]
_x33_333
skip_coneect
scp_s0en 333
sep_ conv,
9op stp_com 263
sep_2001_5
epp_comv_333 3x3
[_(k]
sep_comw__ 3x3
conv
1ol
skip_comm kii__connect
di.cans
Figure 3: Examples of high performing generated cells by DARTS Liu al. 2019 (left) and TENAS (Chen et al. 2021a) (right)
23


Reviews:
[{"review_id": "offIrcnlH3", "reviewer": "Reviewer_3zRw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed approach shows marginal improvements over prior work on multiple NAS benchmarks.\n- The novelty and uniqueness analysis is interesting.\n\nWeaknesses:\n- The primary weakness is the lack of a concrete claim in this paper. It is unclear to this reviewer why we need to learn the density of architectures using a diffusion model. What conceptual or practical limitation of existing density models for the same (e.g., VAE or its variants) is the proposed approach addressing.\n- The multi-condition idea is interesting in principle. However, the particular multi-condition used here in terms of different accuracy quantiles is strange. Under what circumstances should quantiles corresponding to low accuracy of interest? Wouldn’t we always want models in the top say 95th percentile?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a diffusion based generative model for neural architecture search. The architectures are represented as a graph, a diffusion model is trained over the distribution of nodes and edges of graphs corresponding to architectures. The proposed diffusion models is a small variation of an existing diffusion model over graphs. The trained diffusion model can be used in a search process to find models with high accuracy.\n\nExtensive experiments are conducted on multiple tabular NASBench datasets, and ImageNet. The proposed approach offers marginal benefits over prior work.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please state the concrete claims being made by the paper and what evidence (if any) has been provided to validate the claims. The claims are not apparent from the paper.\n- For the novelty and uniqueness of architectures, how was the similarity of a pair of generated graphs measured? Was this through a WL-test? This information is missing.\n- How would the novelty and uniqueness vary as you sample lot more architectures? I would presume, there are only a small set of “good” architectures, and once those are sampled, you will start seeing lots of repetitions. \n- For the results, can you also provide standard deviations of accuracy? Only mean values have been provided. Same for the number of queries (if it changes from one run to another).\n- In section 4.1.2, what is $a$? It has not been defined as far as I can tell.\n- The purpose of discretization of the target variable in section 4.1.3 is unclear. As in, why do we need this discretization? Through discretization the diffusion model will have to learn the density over regions with low accuracy networks. Almost always, our goal is to maximize accuracy. So in this case, isn’t it sufficient to learn the density over networks above a certain accuracy threshold and ignore the rest? This goes back to the weakness mentioned above, in terms of, lack of clarity on what limitations of existing density models is the proposed approach designed to overcome?", "broader_impact_concerns": "There are no perceived ethical concerns for this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E8JuyDh8vf", "reviewer": "Reviewer_kwHT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1.\tAlthough there have been some very recent methods proposing diffusion approaches for network architecture search, these are contemporaneous with the submitted paper. The novel approach will be of significant interest to multiple researchers who are exploring the topic of network architecture search. \n2.\tThe proposed method appears to offer advantageous performance (although there is a need for clearer reporting of execution time to better support some of the main claims of the paper).\n3.\tThe methodology is clearly described and the experiments are thorough. The reporting of novelty/uniqueness is a particularly welcome and interesting inclusion. \n\nWeaknesses\n1.\tSome closely related work is not discussed in detail. Although some of this related work is very recent, it is cited in the paper, so the authors were clearly aware of it. While it may be unreasonable to expect a detailed experimental comparison, a more thorough qualitative discussion is important, and if experimental results are available, then it seems reasonable to include them in the tables. \n2.\tThe results in the paper do not include any indication of variability over different runs. \n3.\tIn some cases, the choice of baselines could be improved. There is a multitude of recent techniques – some of the studies compare to only one recent method. \n4.\tThe reported execution times are difficult to understand, making it challenging to compare the proposed method with the baselines. This is important, considering that a major claim of the paper is that the required time is dramtically reduced.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the task of neural architecture search and proposed a graph-diffusion approach. Existing techniques from the literature are employed for performing discrete conditional graph diffusion. The paper introduces an extension involving multiple conditions to encourage satisfaction of multiple constraints such as accuracy and latency. The proposed approach is differentiable and thus permits end-to-end training and only requires the training of a single model. The paper reports the results of experiments on commonly studied benchmarks and shows that the proposed approach offers an attractive trade-off between speed and accuracy.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tThe authors mention the papers by An et al. (2023) and Lukasik et al. (2022) in the introduction but An et al. (2023) is not discussed in the Related Work section and Lukasik et al. (2022) is only mentioned in passing. These seem to be the most closely related works, so the relationship with these works should be discussed in detail. In particular, the paper claims in the introduction that because the approach “is completely differentiable and thus training involves only a single model … we reach promising performance with much smaller search time.” The paper demonstrates this to some degree for AG-Net (although I think the reporting of timing results needs to be much clearer, as requested below). However, I do not believe there is any experimental comparison to the method proposed by An et al. (2023). This is a recent work, so there could be an argument that it is concurrent research. On the other hand, the paper by An et al. (2023) was posted on arXiv in May, and that is the version that the authors are citing, so presumably they were aware of the work. There are comparable experimental results that could be taken from An et al. (2023), even if the experiments were not reproduced. Moreover, if there is no experimental comparison, the paper should not claim that there is a “much smaller search time”. It should be made clear that this claim refers only to Lukasik et al. (2022). In general, I would like to see a considerably more detailed discussion of why the authors believe that an approach involving a single model is so important.\n\n2.\tTables 1 and 2 report only the number of queries. Are the search times directly proportional to the number of queries? Or does it vary substantially for different methods? In particular, does the proposed diffusion approach require significantly more time per query?\n\n3.\tThe paper does not provide any intervals or deviations for the averaged results. The reader would have better insight into the performance of the method if such information were included. For example, in Table 2, what is the variability over different runs for ImageNet for both the validation and test sets? (Perhaps this would explain \n\n4.\tWith regard to the results in Table 2, can the authors provide a more detailed explanation of \nwhy there is a significant discrepancy between the validation and test results for ImageNet? The current explanation is that the proposed approach optimizes over validation accuracy and this does not necessarily correlate with test performance. While this is true, it is odd that the only technique that is affected in a significant way is the proposed method. The variability over different runs would be helpful here – is the proposed method identifying exactly the same architecture, irrespective of instantiation? Or is one of the high-scoring validation architectures particularly poor in the test set?\n\n5.\tTable 5 presents important ImageNet experimental results. However, the selection of the baselines is surprising, with only AG-Net being a recent comparable baseline. There are multiple other recent baselines that could be included (if only to provide the reader with a better understanding of the competitive landscape). Some suggestions might be [R1] (although this is a very recent publication, it was posted on arXiv in 2022), [R2] and [R3]. There is also [R4], but this is a very recent publication, so it is perhaps unfair to ask for inclusion/comparison – perhaps the authors could at least cite and discuss. Arch2VEC (Yan et al., 2020) [R5] and DrNet (Chen et al., 2021) [R6] are also relevant baselines and should be cited and discussed (possibly with some experimental comparison). \n\n6.\tIn Table 5, why is the training time not included in the execution time? The paper refers to this as a “one-time cost”. For a fair comparison, it would seem more reasonable just to have a column that reports the total execution time. (This can be broken down in other columns to provide more information about how the time is being used). Is there a demonstration of transferability to multiple searchers if this the training time is to be considered a “one-time cost”. (I struggled to understand this from the paper – perhaps the authors could add a sentence or two to highlight how the training applies to multiple different searches). At the moment, the value for T1 isn’t even reported, making it very hard to interpret this table and compare the times for different architectures. This is important, because a key claim of the paper is that the method is faster than AG-Net (the accuracy performance in Table 5 is inferior). \n\n7.\tThere is a concern that the metrics for DARTS, TE-NAS, NASNET-A were all taken from respective papers. This means that only AG-Net follows the protocol specified in the paper. Since the gpus are different and the experimental procedure has changed, it seems unreasonable to have a table that reports numbers as though they can be directly compared. \n\n[R1] Yang, Longxing, et al. \"Sweet Gradient matters: Designing consistent and efficient estimator for Zero-shot Architecture Search.\" Neural Networks 168 (2023): 237-255.\n\n[R2] Ye, Peng, et al. \"b-darts: Beta-decay regularization for differentiable architecture search.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[R3] Soro, Bedionita, and Chong Song. \"Enhancing Differentiable Architecture Search: A Study on Small Number of Cell Blocks in the Search Stage, and Important Branches-Based Cells Selection.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[R4] Li, Muchen, et al. \"GraphPNAS: Learning Probabilistic Graph Generators for Neural Architecture Search.\" Transactions on Machine Learning Research (2023).\n\n[R5] Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture representation learning help neural architecture search? In Advances in Neural Information Processing Systems (NeurIPS), 33:12486– 12498, 2020.\n\n[R6] Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. Dr{nas}: Dirichlet neural architecture search. In Proc. International Conference on Learning Representations, 2021.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XDRe5wriPP", "reviewer": "Reviewer_uVxU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n* Single model training is more efficient than methods requiring separate predictor networks. Most generative NAS methods need an additional predictor model to estimate performance of generated architectures. Training two models is less efficient. By using classifier-free guidance, this method trains only the generator model itself, reducing total training costs.\n\n* Multi-conditioned guidance technique is flexible for imposing multiple constraints, an important capability. Guiding the search with multiple objectives like accuracy and efficiency is crucial for real applications but not well supported in previous generative NAS techniques. The proposed multi-conditioned guidance provides an elegant way to flexibly impose multiple conditions, enabling robust multi-objective search.\n\n* Fast architecture generation rate enables rapid exploration of broad search spaces. Generating just under 0.2 seconds per architecture is orders of magnitude faster than previous approaches that require full architecture evaluation. This rapid rate allows searching much larger spaces, finding high-performing architectures faster than other methods.\n\n* Achieves state-of-the-art or comparable results across several NAS benchmark tasks. Solid performance matching or exceeding prior state-of-the-art on major NAS benchmarks demonstrates this method's effectiveness. Competitive accuracy to more complex techniques shows the power of the proposed graph diffusion approach.\n\n### Weaknesses\n\n* The method does not always achieve absolute state-of-the-art accuracy, slightly underperforms some other methods. While overall results are highly competitive, top accuracies fall just short of 1-2 other approaches in some benchmarks. There may still be room to improve architecture quality further through guidance tuning or alternate diffusion formulations.\n\n* Choice of guidance scale parameter affects diversity/quality tradeoff and the optimal values are likely dataset-specific. The hyperparameter $\\gamma$ balances diversity and quality during sampling. Best setting probably depends greatly on factors like search space size and conditioning constraints. More analysis could better characterize how to set $\\gamma$ robustly.\n\n* The top-1/5 errors on ImageNet are slightly higher than some strong baselines. Specifically, ImageNet results were not superior to all baselines as with CIFAR experiments. Factors such as architecture conditioning strengths may differ when generalizing to very large datasets.\n\n* Does not provide intuitive case studies to fully demonstrate advantages. While numerical results show accuracy and efficiency gains, the paper lacks detailed case studies to more intuitively showcase the benefits versus other methods. Providing examples that compare architectures found across different search techniques could give better insights into real-world improvements. More qualitative analysis would strengthen overall motivation for the proposed approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a neural architecture search (NAS) approach called Multi-conditioned Graph Diffusion for NAS (DiNAS). It uses discrete conditional graph diffusion processes to generate high-performing neural network architectures. It introduces a multi-conditioned classifier-free guidance technique to impose multiple constraints (e.g. accuracy, latency) jointly when generating architectures. This allows searching for architectures that meet multiple objectives. The method is completely differentiable, requiring only a single model training. It achieves promising results on six NAS benchmarks, yielding novel architectures very quickly (less than 0.2 seconds per architecture). Experiments demonstrate the approach generates novel, unique architectures. The method also generalizes well, as shown by experiments on ImageNet.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were unanimous that this paper should be accepted as is by TMLR, congratulations! 

The original reviews outlined many suggestions and questions. The authors replied to almost all of them (very quickly) and provided new comparisons and ablation studies. The authors also clarified several important aspects of the empirical results. 

The reviewers' recommendations did not surface new elements.
Decision: Accept (poster)