POMONAG: Pareto-Optimal Many-Objective Neural Architecture Generator

## Abstract

Abstract Neural Architecture Search (NAS) automates the design of neural network architectures, minimising dependence on human expertise and iterative experimentation. While NAS methods are often computationally intensive and dataset-specific, employing auxiliary predictors to estimate architecture properties has proven extremely beneficial. These predictors substantially reduce the number of models requiring training, thereby decreasing overall search time. This strategy is frequently utilised to generate architectures satisfying multiple computational constraints.
Recently, Transferable Neural Architecture Search (Transferable NAS) has emerged, generalising the search process from being dataset-dependent to task-dependent. In this domain, DiffusionNAG stands as a state-of-the-art method. This diffusion-based method streamlines computation, generating architectures optimised for accuracy on unseen datasets without the need for further adaptation. However, by concentrating exclusively on accuracy, DiffusionNAG neglects other crucial objectives like model complexity, computational efficiency, and inference latency – factors essential for deploying models in resource-constrained, real-world environments.
This paper introduces the Pareto-Optimal Many-Objective Neural Architecture Generator (POMONAG), extending DiffusionNAG through a many-objective diffusion process. POMONAG simultaneously considers accuracy, the number of parameters, multiply-accumulate operations (MACs), and inference latency. It integrates Performance Predictor models to estimate these secondary metrics and guide the diffusion gradients. POMONAG’s optimisation is enhanced by expanding its training Meta-Dataset, applying Pareto Front Filtering to generated architectures, and refining embeddings for conditional generation. These enhancements enable POMONAG to generate Pareto-optimal architectures that outperform the previous state-of-the-art in both performance and efficiency.
Results were validated on two distinct search spaces – NASBench201 and MobileNetV3 – and evaluated across 15 image classification datasets.

## 1 Introduction

Deep learning has become indispensable in various domains by enabling models to learn intricate patterns from large datasets. The architecture of neural networks plays a pivotal role in their performance, traditionally requiring expert knowledge and extensive experimentation. Neural Architecture Search (NAS) automates this design process, aiming to discover optimal architectures without human intervention. However, conventional NAS methods often involve significant computational costs and are typically tailored to specific datasets, limiting their scalability and general applicability.
Transferable Neural Architecture Search (Transferable NAS) addresses these limitations by generalising the search process across different tasks. DiffusionNAG stands out in this domain, utilising diffusion processes to generate neural architectures optimised for accuracy on unseen datasets. While effective in reducing computational overhead, DiffusionNAG focuses solely on maximising accuracy, neglecting other crucial performance metrics such as model size, computational cost, and inference latency.
In practical applications, especially within resource-constrained environments like mobile devices and embedded systems, it is essential to consider multiple objectives simultaneously. Existing Multi- and Many-objective NAS methods often face scalability challenges or require extensive computational resources, making them less practical for widespread adoption.

This work introduces the Pareto-Optimal Many-Objective Neural Architecture Generator (POMONAG), which extends the capabilities of DiffusionNAG by performing a many-objective optimisation. POMONAG simultaneously considers accuracy, the number of parameters, multiply-accumulate operations (MACs), and inference latency during the architecture generation process. Auxiliary Performance Predictor models, trained to estimate these metrics, are integrated into the diffusion process, guiding exploration towards regions of the search space offering optimal trade-offs among these objectives. This approach facilitates the generation of architectures that are both highly accurate and efficient in computational resources and speed.
To enhance POMONAG’s performance and adaptability, several key improvements have been implemented. The training Meta-Dataset has been significantly expanded, incorporating a diverse set of architectures and tasks to enhance the Performance Predictors’ capabilities across different domains. Pareto Front Filtering and Stretching techniques are applied to balance the multiple objectives effectively, ensuring that the generated architectures are Pareto-optimal. Additionally, the embeddings used for conditional generation have been refined to facilitate more accurate and dataset-aware architecture synthesis.
Extensive experiments validate POMONAG’s effectiveness. Evaluations conducted on two prominent search spaces—NASBench201 and MobileNetV3—and tested across 15 diverse image classification datasets demonstrate that POMONAG outperforms existing state-of-the-art methods, including DiffusionNAG. The generated architectures achieve superior accuracy while satisfying various computational constraints and require significantly fewer trained models, highlighting the efficiency of the proposed technique.

The key contributions of this work include:

- •
Diffusion-based Many-Objective Optimisation. POMONAG introduces a many-objective diffusion approach within Transferable NAS, optimising neural architectures for accuracy, number of parameters, MACs, and inference latency.
- •
Pareto-Optimal Architecture Generation. By computing Pareto Fronts, POMONAG effectively navigates trade-offs among multiple objectives, generating architectures that offer balanced compromises suitable for diverse deployment scenarios.
- •
Enhanced Meta-Datasets. New Meta-Datasets have been developed to improve the Performance Predictors’ ability to predict architecture performance across various tasks and datasets.
- •
Refined Performance Predictor Models. The Performance Predictor models have been enhanced to increase prediction accuracy and effectiveness in guiding the architecture generation process.
- •
State-of-the-Art Transferable NAS Model. POMONAG establishes a new benchmark by generating architectures that are high-performing and adaptable to various computational constraints.

The POMONAG model’s code and the accompanying Meta-Datasets will be made publicly available upon publication of this paper at the following link.

## 2 Related Works

Neural Architecture Search. Neural Architecture Search (NAS) seeks to automate neural architecture design, removing the need for manual trial-and-error processes. Early methods, such as reinforcement learning [, ], evolutionary algorithms [, ], and gradient-based techniques [, ], are computationally expensive as they require full training of numerous architectures.

One-shot NAS. To mitigate computational costs, one-shot NAS methods employ weight sharing among candidate architectures. ENAS [] uses an RNN controller to generate sub-networks; DARTS [] relaxes the search space into a continuous one; OFA [] trains a single large network with many architectural choices, and OFAv2 [] extends this approach with an enriched search space. While these methods reduce costs, they may face challenges in optimisation bias, training stability, or performance across diverse sub-networks.

Multi- and Many-Objective NAS. As application complexity grows, NAS methods increasingly consider multiple objectives. Approaches like MONAS [] and DPP-Net [] balance criteria such as accuracy, latency, and energy consumption. NSGANetV2 [] handles up to 12 objectives. Methods such as POPNASv3 [], NAT [], and NATv2 [] offer Pareto-optimal solutions, addressing scalability and computational challenges inherent in multi-objective optimisation.

BO-based NAS. To further reduce computational costs, Bayesian Optimisation (BO) predictor-based methods [, ] employ surrogate models to estimate architecture performance without full training. BANANAS [] uses an ensemble of neural networks, while NASBOWL [] combines graph kernels with Gaussian processes. However, these methods often play a passive role, mainly filtering architectures generated by other strategies.

Transferable NAS. Transferable NAS methods leverage knowledge from previous tasks to accelerate searches on new datasets. MetaD2A [] employs meta-learning to generate dataset-specific architectures. TNAS [] enhances predictor adaptability to unseen datasets using BO with a deep-kernel Gaussian process. While promising in reducing search times, these methods may still face inefficiencies in exploring the architecture space.

Diffusion Models. Diffusion models [, ] have shown outstanding generative performance by learning to reverse the process of gradually adding noise to data. In graph generation, GDSS [] applies diffusion models to undirected graphs. However, their application to neural architecture generation, involving directed acyclic graphs with specific constraints, remains unexplored.

DiffusionNAG. DiffusionNAG introduces a conditional Neural Architecture Generation framework based on diffusion models [], addressing inefficiencies in traditional NAS methods that require sampling and training numerous irrelevant architectures. It models neural architectures as directed graphs and employs a graph diffusion process for their generation. The forward diffusion process perturbs architectures with Gaussian noise, mapping the architecture distribution to a known prior. The reverse process then refines noise into valid architectures using a learned score function.
A key component is a specialised Score Network designed to capture dependencies between nodes, reflecting the computational flow in directed acyclic graphs. This network utilises Transformer blocks with an attention mask and incorporates positional embeddings to accurately represent the topological ordering of layers, ensuring that generated architectures adhere to specific search space rules.
DiffusionNAG integrates a parameterised predictor into the reverse diffusion process, forming a predictor-guided conditional generation scheme. This allows the model to generate task-optimal architectures by sampling from regions more likely to satisfy desired properties.

In Transferable NAS scenarios, DiffusionNAG employs a meta-learned dataset-aware predictor conditioned on image classification datasets. This predictor, trained over a distribution of tasks, enables accurate predictions for unseen datasets without additional training. By integrating this predictor into the conditional generative process, DiffusionNAG facilitates efficient architecture generation for new tasks.
By leveraging predictor-guided conditional architecture generation within the diffusion framework, DiffusionNAG reduces computational overhead and enhances search efficiency in NAS.

## 3 Method

This section outlines the techniques developed and implemented in POMONAG, a Transferable NAS model derived from the DiffusionNAG framework proposed by An et al. []. It presents the Many-Objective Reverse Diffusion Guidance process, the creation of new Meta-Datasets, and enhancements to the Performance Predictors. Additionally, it explains the Pareto Front Stretching and Pareto Front Filtering techniques used to optimise and refine the architecture generation process.

Figure: Figure 1: Overview of the POMONAG framework illustrating the Many-Objective Reverse Diffusion Guidance process.
Refer to caption: extracted/5890392/POMONAG.png

### 3.1 Many-Objective Reverse Diffusion Guidance

The diffusion model in DiffusionNAG employs the Reverse Diffusion Process for architecture generation, described as:

$$ $\displaystyle\begin{split}dA_{t}={}&\left[f_{t}(A_{t})-g_{t}^{2}\nabla_{A_{t}} \log p_{t}(A_{t})\right]d\bar{t}+g_{t}d\bar{w}.\end{split}$ $$

The score function $\nabla_{A_{t}}\log p_{t}(A_{t})$ is approximated by introducing the Score Network, which iteratively applies the transformation $s_{\theta}(A_{t},t)$ to the noisy architecture $A_{t}$. The process is guided using the dataset-aware Performance Predictor $f_{\phi}(y|\tilde{D},A_{t})$, resulting in:

$$ $dA_{t}={}\Bigr{\{}f_{t}(A_{t})-g_{t}^{2}\Bigl{[}\leavevmode\hbox to39.74pt{ \vbox to19.57pt{\pgfpicture\makeatletter\hbox{\hskip 19.86897pt\lower-9.783pt \hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{ {}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgffillcolor}{ rgb}{1,0.87890625,0.515625}\pgfsys@color@rgb@fill{1}{0.87890625}{0.515625} \pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgffillcolor}{rgb}{ 1,0.87890625,0.515625}\pgfsys@color@rgb@fill{1}{0.87890625}{0.515625} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{}\pgfsys@moveto{15.66898pt}{9.58301pt} \pgfsys@lineto{-15.66898pt}{9.58301pt}\pgfsys@curveto{-17.87814pt}{9.58301pt}{ -19.66898pt}{7.79218pt}{-19.66898pt}{5.58301pt}\pgfsys@lineto{-19.66898pt}{-5. 58301pt}\pgfsys@curveto{-19.66898pt}{-7.79218pt}{-17.87814pt}{-9.58301pt}{-15. 66898pt}{-9.58301pt}\pgfsys@lineto{15.66898pt}{-9.58301pt}\pgfsys@curveto{17.8 7814pt}{-9.58301pt}{19.66898pt}{-7.79218pt}{19.66898pt}{-5.58301pt} \pgfsys@lineto{19.66898pt}{5.58301pt}\pgfsys@curveto{19.66898pt}{7.79218pt}{17 .87814pt}{9.58301pt}{15.66898pt}{9.58301pt}\pgfsys@closepath\pgfsys@moveto{-19 .66898pt}{-9.58301pt}\pgfsys@fillstroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-16.33597pt}{-3.75pt} \pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$s_{\theta}(A_{t},t)$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}+k_{\phi}\nabla_{A_{t}} \log\leavevmode\hbox to53.94pt{\vbox to20.1pt{\pgfpicture\makeatletter\hbox{ \hskip 26.96826pt\lower-10.04967pt\hbox to0.0pt{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{{}}\hbox{\hbox{{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{ 0.80859375,0.8828125,0.953125}\pgfsys@color@rgb@fill{0.80859375}{0.8828125}{0. 953125}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{ 0.80859375,0.8828125,0.953125}\pgfsys@color@rgb@fill{0.80859375}{0.8828125}{0. 953125}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{ {}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{}\pgfsys@moveto{22.76826pt}{9.84967 pt}\pgfsys@lineto{-22.76826pt}{9.84967pt}\pgfsys@curveto{-24.97743pt}{9.84967 pt}{-26.76826pt}{8.05884pt}{-26.76826pt}{5.84967pt}\pgfsys@lineto{-26.76826pt} {-5.84967pt}\pgfsys@curveto{-26.76826pt}{-8.05884pt}{-24.97743pt}{-9.84967pt}{ -22.76826pt}{-9.84967pt}\pgfsys@lineto{22.76826pt}{-9.84967pt}\pgfsys@curveto{ 24.97743pt}{-9.84967pt}{26.76826pt}{-8.05884pt}{26.76826pt}{-5.84967pt} \pgfsys@lineto{26.76826pt}{5.84967pt}\pgfsys@curveto{26.76826pt}{8.05884pt}{24 .97743pt}{9.84967pt}{22.76826pt}{9.84967pt}\pgfsys@closepath\pgfsys@moveto{-26 .76826pt}{-9.84967pt}\pgfsys@fillstroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-23.43526pt}{-3.48334 pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$f_{\phi}(y|\tilde{D},A_{t})$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\Bigr{]}\Bigl{\}}d\bar{ t}+g_{t}d\bar{w}.$ $$

The Score Network is responsible for denoising the noisy architecture sampled from the generative distribution at each denoising step. The output of the Performance Predictor is used to compute the guidance term that steers the generation process towards regions of the search space with a higher density of accurate architectures for the given dataset.

In POMONAG’s Reverse Diffusion Guidance process, the generation gradient is directed in a many-objective fashion by simultaneously considering three additional terms:

- •
$k_{\pi}\nabla_{A_{t}}\log f_{\pi}(p|\tilde{D},A_{t})$, where $p$ represents the number of parameters of architecture $A_{t}$ and $k_{\pi}$ is a constant scaling factor;
- •
$k_{\mu}\nabla_{A_{t}}\log f_{\mu}(m|\tilde{D},A_{t})$, where $m$ represents the number of MACs of architecture $A_{t}$ and $k_{\mu}$ is a constant scaling factor;
- •
$k_{\lambda}\nabla_{A_{t}}\log f_{\lambda}(l|\tilde{D},A_{t})$, where $l$ represents the inference latency of architecture $A_{t}$ and $k_{\lambda}$ is a constant scaling factor.

Each term represents the log-likelihood of architecture $A_{t}$ satisfying the corresponding metric evaluated on dataset $D$ at timestamp $t$. These three terms are approximated by introducing three specialised Performance Predictors. The formulation for the new Many-Objective Reverse Diffusion Guidance is defined as:

$$ $\displaystyle\begin{split}dA_{t}={}&\Bigr{\{}f_{t}(A_{t})-g_{t}^{2}\Bigl{[} \leavevmode\hbox to39.74pt{\vbox to19.57pt{\pgfpicture\makeatletter\hbox{ \hskip 19.86897pt\lower-9.783pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0} \pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }{}{{}}\hbox{\hbox{{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor{pgffillcolor}{rgb}{1,0.87890625,0.515625} \pgfsys@color@rgb@fill{1}{0.87890625}{0.515625}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgffillcolor}{rgb}{ 1,0.87890625,0.515625}\pgfsys@color@rgb@fill{1}{0.87890625}{0.515625} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{}\pgfsys@moveto{15.66898pt}{9.58301pt} \pgfsys@lineto{-15.66898pt}{9.58301pt}\pgfsys@curveto{-17.87814pt}{9.58301pt}{ -19.66898pt}{7.79218pt}{-19.66898pt}{5.58301pt}\pgfsys@lineto{-19.66898pt}{-5. 58301pt}\pgfsys@curveto{-19.66898pt}{-7.79218pt}{-17.87814pt}{-9.58301pt}{-15. 66898pt}{-9.58301pt}\pgfsys@lineto{15.66898pt}{-9.58301pt}\pgfsys@curveto{17.8 7814pt}{-9.58301pt}{19.66898pt}{-7.79218pt}{19.66898pt}{-5.58301pt} \pgfsys@lineto{19.66898pt}{5.58301pt}\pgfsys@curveto{19.66898pt}{7.79218pt}{17 .87814pt}{9.58301pt}{15.66898pt}{9.58301pt}\pgfsys@closepath\pgfsys@moveto{-19 .66898pt}{-9.58301pt}\pgfsys@fillstroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-16.33597pt}{-3.75pt} \pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$s_{\theta}(A_{t},t)$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}+k_{\phi}\nabla_{A_{t}} \log\leavevmode\hbox to53.94pt{\vbox to20.1pt{\pgfpicture\makeatletter\hbox{ \hskip 26.96826pt\lower-10.04967pt\hbox to0.0pt{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{{}}\hbox{\hbox{{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{ 0.80859375,0.8828125,0.953125}\pgfsys@color@rgb@fill{0.80859375}{0.8828125}{0. 953125}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{ 0.80859375,0.8828125,0.953125}\pgfsys@color@rgb@fill{0.80859375}{0.8828125}{0. 953125}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{ {}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{}\pgfsys@moveto{22.76826pt}{9.84967 pt}\pgfsys@lineto{-22.76826pt}{9.84967pt}\pgfsys@curveto{-24.97743pt}{9.84967 pt}{-26.76826pt}{8.05884pt}{-26.76826pt}{5.84967pt}\pgfsys@lineto{-26.76826pt} {-5.84967pt}\pgfsys@curveto{-26.76826pt}{-8.05884pt}{-24.97743pt}{-9.84967pt}{ -22.76826pt}{-9.84967pt}\pgfsys@lineto{22.76826pt}{-9.84967pt}\pgfsys@curveto{ 24.97743pt}{-9.84967pt}{26.76826pt}{-8.05884pt}{26.76826pt}{-5.84967pt} \pgfsys@lineto{26.76826pt}{5.84967pt}\pgfsys@curveto{26.76826pt}{8.05884pt}{24 .97743pt}{9.84967pt}{22.76826pt}{9.84967pt}\pgfsys@closepath\pgfsys@moveto{-26 .76826pt}{-9.84967pt}\pgfsys@fillstroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-23.43526pt}{-3.48334 pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$f_{\phi}(y|\tilde{D},A_{t})$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\ &+k_{\pi}\nabla_{A_{t}}\log\leavevmode\hbox to53.56pt{\vbox to19.57pt{ \pgfpicture\makeatletter\hbox{\hskip 26.78082pt\lower-9.783pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{{}}\hbox{\hbox{{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{0.6015625,0.6015625,1} \pgfsys@color@rgb@fill{0.6015625}{0.6015625}{1}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{ 0.6015625,0.6015625,1}\pgfsys@color@rgb@fill{0.6015625}{0.6015625}{1} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{}\pgfsys@moveto{22.58083pt}{9.58301pt} \pgfsys@lineto{-22.58083pt}{9.58301pt}\pgfsys@curveto{-24.79pt}{9.58301pt}{-26 .58083pt}{7.79218pt}{-26.58083pt}{5.58301pt}\pgfsys@lineto{-26.58083pt}{-5.583 01pt}\pgfsys@curveto{-26.58083pt}{-7.79218pt}{-24.79pt}{-9.58301pt}{-22.58083 pt}{-9.58301pt}\pgfsys@lineto{22.58083pt}{-9.58301pt}\pgfsys@curveto{24.79pt}{ -9.58301pt}{26.58083pt}{-7.79218pt}{26.58083pt}{-5.58301pt}\pgfsys@lineto{26.5 8083pt}{5.58301pt}\pgfsys@curveto{26.58083pt}{7.79218pt}{24.79pt}{9.58301pt}{2 2.58083pt}{9.58301pt}\pgfsys@closepath\pgfsys@moveto{-26.58083pt}{-9.58301pt} \pgfsys@fillstroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-23.24782pt}{-3.75pt} \pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$f_{\pi}(p|\tilde{D},A_{t})$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}+k_{\mu}\nabla_{A_{t}} \log\leavevmode\hbox to57.49pt{\vbox to19.57pt{\pgfpicture\makeatletter\hbox{ \hskip 28.74632pt\lower-9.783pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0} \pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }{}{{}}\hbox{\hbox{{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{0.80078125,0.6015625,1 }\pgfsys@color@rgb@fill{0.80078125}{0.6015625}{1}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{ 0.80078125,0.6015625,1}\pgfsys@color@rgb@fill{0.80078125}{0.6015625}{1} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{}\pgfsys@moveto{24.54633pt}{9.58301pt} \pgfsys@lineto{-24.54633pt}{9.58301pt}\pgfsys@curveto{-26.7555pt}{9.58301pt}{- 28.54633pt}{7.79218pt}{-28.54633pt}{5.58301pt}\pgfsys@lineto{-28.54633pt}{-5.5 8301pt}\pgfsys@curveto{-28.54633pt}{-7.79218pt}{-26.7555pt}{-9.58301pt}{-24.54 633pt}{-9.58301pt}\pgfsys@lineto{24.54633pt}{-9.58301pt}\pgfsys@curveto{26.755 5pt}{-9.58301pt}{28.54633pt}{-7.79218pt}{28.54633pt}{-5.58301pt}\pgfsys@lineto {28.54633pt}{5.58301pt}\pgfsys@curveto{28.54633pt}{7.79218pt}{26.7555pt}{9.583 01pt}{24.54633pt}{9.58301pt}\pgfsys@closepath\pgfsys@moveto{-28.54633pt}{-9.58 301pt}\pgfsys@fillstroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-25.21332pt}{-3.75pt} \pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$f_{\mu}(m|\tilde{D},A_{t})$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}+k_{\lambda}\nabla_{A_{ t}}\log\leavevmode\hbox to51.79pt{\vbox to19.57pt{\pgfpicture\makeatletter \hbox{\hskip 25.89275pt\lower-9.783pt\hbox to0.0pt{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{{}}\hbox{\hbox{{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{1,0.6015625,1} \pgfsys@color@rgb@fill{1}{0.6015625}{1}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{ 1,0.6015625,1}\pgfsys@color@rgb@fill{1}{0.6015625}{1}\pgfsys@invoke{ }{{}{}{{} }}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{ }{{}}}{}{}{}\pgfsys@moveto{21.69275pt}{9.58301pt}\pgfsys@lineto{-21.69275pt}{9 .58301pt}\pgfsys@curveto{-23.90192pt}{9.58301pt}{-25.69275pt}{7.79218pt}{-25.6 9275pt}{5.58301pt}\pgfsys@lineto{-25.69275pt}{-5.58301pt}\pgfsys@curveto{-25.6 9275pt}{-7.79218pt}{-23.90192pt}{-9.58301pt}{-21.69275pt}{-9.58301pt} \pgfsys@lineto{21.69275pt}{-9.58301pt}\pgfsys@curveto{23.90192pt}{-9.58301pt}{ 25.69275pt}{-7.79218pt}{25.69275pt}{-5.58301pt}\pgfsys@lineto{25.69275pt}{5.58 301pt}\pgfsys@curveto{25.69275pt}{7.79218pt}{23.90192pt}{9.58301pt}{21.69275pt }{9.58301pt}\pgfsys@closepath\pgfsys@moveto{-25.69275pt}{-9.58301pt} \pgfsys@fillstroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-22.35974pt}{-3.75pt} \pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$f_{\lambda}(l|\tilde{D},A_{t})$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\Bigr{]}\Bigl{\}}d\bar{ t}+g_{t}d\bar{w}.\end{split}$ $$

Each Performance Predictor – the Accuracy Predictor $f_{\phi}$, Parameters Predictor $f_{\pi}$, MACs Predictor $f_{\mu}$, and Inference Latency Predictor $f_{\lambda}$ – is trained to predict the corresponding metric of architecture $A_{t}$ when presented with dataset $D$. Similar to the Accuracy Predictor $f_{\phi}$, the outputs of each additional Performance Predictor are used to adjust the unconditioned gradient direction and update the generative probability distribution at each denoising step during the diffusion phase. In doing so, the generative probability distribution is progressively guided towards regions of the search space that yield optimal architectures in terms of accuracy, number of parameters, number of operations (MACs), and inference latency. The entire many-objective generation process of POMONAG is depicted in Figure 1.

### 3.2 Meta-Dataset

The training of the Score Network, as well as the Performance Predictors, requires a Meta-Dataset containing the types of architecture structures to be generated, along with their characteristics and performance metrics. These architectures must be sampled from a sufficiently large search space and subsequently trained to solve specific tasks using a broad pool of datasets. This approach generalises the models and enables the Score Network to perform dataset-aware generation.

In line with practices used in other models within the Transferable NAS family [, , ], the datasets used for training the architectures – whose metadata will compose the Meta-Dataset – are extracted from the ImageNet32 dataset [].
For each iteration, extraction is performed randomly by selecting 20 classes from ImageNet32. All corresponding samples of these classes form the dataset $D$. In parallel, an architecture $A$ is selected from the designated search space to be associated with this classification task. The structure of this architecture is stored in encoded form. For more information regarding the encoding of architectures, refer to Appendix A.
Subsequently—and in contrast to other approaches—the number of parameters, MACs, and the inference latency on a single 32$\times$32 sample are calculated for the architecture. Since latency is highly susceptible to noise, each measurement is repeated 100 times; measurements outside the 90% confidence interval are discarded, and the mean of the remaining values is computed.

Finally, the architecture is trained on the extracted dataset, split into training, validation, and test sets (80/10/10), and its accuracy is calculated on the test split. This information is aggregated and stored as a new tuple in the Meta-Dataset, forming a triplet:

- •
Dataset: The dataset $D$ of 20 classes extracted from ImageNet32.
- •
Architecture: The structure of the architecture $A$ in encoded form, extracted from the designated search space.
- •
Objectives: The test accuracy, parameters, MACs, and latency values of architecture $A$ on dataset $D$.

The search spaces used in this work are NASBench201 [] and MobileNetV3 [], utilised independently to create two distinct Meta-Datasets. The choice of training and augmentation techniques is as crucial as the size of the Meta-Datasets to obtain accurate Performance Predictors and generate high-performing and efficient architectures.
To this end, the Meta-Datasets created for training the models composing POMONAG have undergone the following modifications compared to those used in the DiffusionNAG model:

- •
Training Pipeline Optimisation. The training pipelines for the architectures included in the Meta-Datasets have been optimised to maximise accuracy performance through hyperparameter tuning. For each search space, five architectures were selected and five associated datasets were extracted, as previously described. For each architecture, 50 optimisation steps were executed using Optuna [], incorporating a Tree-structured Parzen Estimator sampling method [] alongside a Hyperband pruning mechanism [], aiming to maximise validation accuracy. For more information on the search space and the obtained pipelines, refer to Appendix D.
- •
Dataset Expansion. The size of the Meta-Datasets was increased by including a larger number of architecture-dataset pairs. For the NASBench201 search space, the cardinality was expanded from 4,630 to 10,000 triplets. Since the cardinality of the MobileNetV3-based Meta-Dataset was not provided by An et al. [], and given the considerable size of the search space, the number of triplets was set to 20,000.

### 3.3 Score Network and Performance Predictors

Score Network. The POMONAG framework adapts the core architecture of the Score Network from DiffusionNAG [], incorporating modifications to accommodate the novel encodings detailed in Appendix A. The Score Network is responsible for iteratively denoising the encodings of the architectures to be generated. The fundamental structure remains consistent with DiffusionNAG. The input embeddings combine operation information (Embops), node positions (Embpos), and time step (Emb${}_{\text{time}}$):

$$ $\text{Emb}_{i}=\text{Emb}_{\text{ops}(v_{i})}+\text{Emb}_{\text{pos}(v_{i})}+ \text{Emb}_{\text{time}(t)}$ $$

where $v_{i}$ represents the $i$-th row of the operator type matrix $V$. The training strategy for the Score Network remains consistent with that defined by An et al. [].

Performance Predictors.
The Performance Predictors are models designed to predict the characteristics of architectures during and after the generation phase. In the first instance, the objective is to calculate the regression error used as Reverse Diffusion Guidance; in the second, the aim is to estimate the performance of the architecture once denoised, allowing for ranking without the need for any training and thereby identifying the most promising architectures.

In DiffusionNAG, there are two Performance Predictors: one for estimating the accuracy of noisy architectures during generation, and one for estimating the accuracy of denoised architectures.
In POMONAG, there are five Performance Predictors. Four are dedicated to the respective estimation of accuracy, parameters, MACs, and inference latency of noisy architectures during the diffusion phase. The fifth estimates the accuracy of denoised architectures, given that the other metrics can be extracted with negligible overhead.

The Performance Predictors in POMONAG retain a similar architecture to that in DiffusionNAG. The substantial modifications pertain to the training procedure – which has been revised to maximise the Spearman correlation between the predicted values and the actual metrics – the size and content of the Meta-Dataset used during training, and the model employed as the Dataset Encoder.
Specifically, while DiffusionNAG employs a ResNet18 architecture, POMONAG utilises a Vision Transformer (ViT-B-16) model for dataset embedding.
From the conducted ablation studies, where the two Performance Predictors of DiffusionNAG are gradually adapted to the version used in POMONAG, a gradual improvement in terms of Spearman correlation is observable. Concretely, this amounts to improvements of +0.168 and +0.117 for the versions applied to noisy and denoised architectures, respectively. More details on the study are presented in Appendix C.

### 3.4 Pareto Front Filtering and Stretching

In DiffusionNAG, architectures are generated in batches of size 256. The scaling factor $k_{\phi}$ used to weight the contribution of the Performance Predictor during the generation process is set to 10000 and remains constant across all experiments. The generated architectures are then validated by filtering out those with inadequate or incorrect structures. Subsequently, the Performance Predictor for denoised architectures estimates their accuracy, obviating the need to train them. The architectures are then ranked based on this estimate, and the top five are returned as the output of the generation.

Pareto Front Filtering. In POMONAG, an additional filtering step is introduced through the construction of a Pareto Front, aimed at leveraging the secondary metrics. Specifically, after the many-objective generation of architectures, the elimination of invalid configurations, and the estimation of accuracies via the Performance Predictor, a Pareto Front is constructed for each of the three secondary metrics, and only the dominant architectures are retained. This approach enables the selection of architectures that are Pareto-optimal, allowing for choices based on trade-offs between accuracy and secondary metrics.
To this end, three configurations extractable from each Pareto Front are identified. The configuration POMONAGAcc represents the architecture for which the highest accuracy is predicted; the configuration POMONAGBal represents the architecture for which the ratio between predicted accuracy and the considered secondary metric is highest; finally, the configuration POMONAG${}_{\text{Eff}}$ represents the architecture with the lowest value of the secondary metric and therefore the most efficient.

Pareto Front Stratching.
The Many-Objective Reverse Diffusion Guidance process introduced in POMONAG utilises four different guides, one for each Performance Predictor, to model the distribution of the architecture space. To optimise the corresponding scaling factors, POMONAG employs a dynamic approach rather than a fixed value as in DiffusionNAG. This optimisation process aims to maximise the mean of the estimated accuracies of the architectures in the Pareto Front by adjusting the scaling factors corresponding to each guide.

This optimisation leverages Optuna [], incorporating a Tree-structured Parzen Estimator sampling method [] alongside a Hyperband pruning mechanism []. The process involves 100 evaluations on four datasets (CIFAR10, CIFAR100, Aircraft, Oxford III Pets). This procedure is replicated for each search space.

To fully exploit the secondary metrics, POMONAG introduces a novel technique termed Pareto Front Stretching. This entails generating architectures in two phases, each with batches of 128 elements. The first batch favours efficient architectures with respect to the secondary metrics, assigning greater weight to the predictors of parameters, MACs, and inference latency. The second batch aims to obtain highly accurate architectures while still maintaining constraints on their complexity.

To implement these two generation processes, the optimisation of the scaling factors is executed twice with different constraints. For efficient architectures, the scaling factor for the accuracy guide is searched within the interval $[1000,5000]$, while for the other guides it is within $[100,500]$. For highly accurate architectures, the scaling factor for the accuracy guide is optimised within $[10,000,50,000]$, while for the other metrics within $[10,50]$.

The optimisation results yielded specific values for each search space. For NASBench201, the optimal scaling factors for efficient architectures were 4732, 482, 421, and 368, respectively for accuracy, parameters, MACs, and latency. For highly accurate architectures, the values are 24,943, 12, 26, and 13. In the case of MobileNetV3, for efficient architectures, the values obtained are 4987, 494, 478, and 481, while for highly accurate ones, 48,321, 21, 16, and 39.

This approach allows for a more thorough exploration of the solution space, generating an overall set of architectures more widely spread with respect to the secondary metrics. The effect is therefore a diversified generation of architectures and a Pareto Front with a more elongated shape. For more information, the reader is referred to Appendix B.

## 4 Experiments and Results

This section presents the experiments conducted and the results obtained to evaluate the performance of POMONAG compared to other NAS models, particularly DiffusionNAG. All experiments were conducted using an NVIDIA Quadro RTX 6000 graphics card.

**Table 1: Comparison with Transferable NAS methods on the MobileNetV3 search space. The POMONAG architectures in this comparison correspond to the POMONAG${}_{\text{Acc}}$ configuration. The best scores are highlighted in bold.**
| Dataset | Stats. | MetaD2A | TNAS | DiffusionNAG | POMONAG |
| --- | --- | --- | --- | --- | --- |
|  |  | [] | [] | [] | (ours) |
|  | Max | 97.45$\pm$0.07 | 97.48$\pm$0.14 | 97.52$\pm$0.07 | 97.85$\pm$0.14 |
| CIFAR10 | Mean | 97.28$\pm$0.01 | 97.22$\pm$0.00 | 97.39$\pm$0.01 | 97.85$\pm$0.14 |
|  | Min | 97.09$\pm$0.13 | 95.26$\pm$0.09 | 97.23$\pm$0.06 | 97.85$\pm$0.14 |
|  | Max | 86.00$\pm$0.19 | 85.95$\pm$0.29 | 86.07$\pm$0.16 | 86.18$\pm$0.11 |
| CIFAR100 | Mean | 85.56$\pm$0.02 | 85.30$\pm$0.04 | 85.74$\pm$0.04 | 86.18$\pm$0.11 |
|  | Min | 84.74$\pm$0.13 | 81.30$\pm$0.18 | 85.42$\pm$0.08 | 86.18$\pm$0.11 |
|  | Max | 82.18$\pm$0.70 | 82.31$\pm$1.31 | 82.28$\pm$0.29 | 87.62$\pm$0.12 |
| Aircraft | Mean | 81.19$\pm$0.11 | 80.86$\pm$0.15 | 81.47$\pm$0.05 | 87.62$\pm$0.12 |
|  | Min | 79.71$\pm$0.54 | 74.99$\pm$6.65 | 80.88$\pm$0.54 | 87.62$\pm$0.12 |
|  | Max | 95.28$\pm$0.50 | 95.04$\pm$0.44 | 95.34$\pm$0.29 | 95.28$\pm$0.09 |
| Oxford III Pets | Mean | 94.55$\pm$0.03 | 94.47$\pm$0.10 | 94.75$\pm$0.14 | 95.28$\pm$0.09 |
|  | Min | 93.68$\pm$0.16 | 92.39$\pm$0.47 | 94.28$\pm$0.17 | 95.28$\pm$0.09 |

Transferable NAS Evaluation on MobileNetV3.
In Table 1, the comparison between POMONAG and other Transferable NAS methods on the MobileNetV3 search space is presented. POMONAG achieves the best performance in terms of minimum, mean, and maximum accuracy on almost all the datasets considered. Notably, on the Aircraft dataset, POMONAG reaches an accuracy of 87.62%, significantly surpassing other methods. On CIFAR10, CIFAR100, and Oxford III Pets, POMONAG also demonstrates superior or comparable performance to the state of the art.
Unlike other approaches, where the generation of architectures yields multiple alternatives and diversifies the obtained performances, by exploiting the Pareto Front and selecting the POMONAG${}_{\text{Acc}}$ configuration, it is possible to reduce the number of architectures to be trained to just one. These results demonstrate POMONAG’s ability to generate architectures that, even when considering multiple conditionings based on competing metrics, manage to outperform previous state-of-the-art methods.

**Table 2: Comparison of NAS techniques on the NASBench201 search space across four datasets. ’Trained Archs’ indicates the number of neural architectures trained to achieve the reported accuracy. Results show mean accuracy $\pm$ 95% confidence intervals over three runs. The POMONAG architectures used in this comparison correspond to the POMONAG${}_{\text{Acc}}$ configuration. Best scores are highlighted in bold.**
|  |  | CIFAR10 | CIFAR100 | Aircraft | Oxford III Pets |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Type | Method | Accuracy $\uparrow$ | Trained | Accuracy $\uparrow$ | Trained | Accuracy $\uparrow$ | Trained | Accuracy $\uparrow$ | Trained |
|  |  |  | Archs $\downarrow$ |  | Archs $\downarrow$ |  | Archs $\downarrow$ |  | Archs $\downarrow$ |
|  | RSPS [] | 84.07$\pm$3.61 | N/A | 52.31$\pm$5.77 | N/A | 42.19$\pm$3.88 | N/A | 22.91$\pm$1.65 | N/A |
|  | SETN [] | 87.64$\pm$0.00 | N/A | 59.09$\pm$0.24 | N/A | 44.84$\pm$3.96 | N/A | 25.17$\pm$1.68 | N/A |
| One-shot | GDAS [] | 93.61$\pm$0.09 | N/A | 70.70$\pm$0.30 | N/A | 53.52$\pm$0.48 | N/A | 24.02$\pm$2.75 | N/A |
| NAS | PC-DARTS [] | 93.66$\pm$0.17 | N/A | 66.64$\pm$2.34 | N/A | 26.33$\pm$3.40 | N/A | 25.31$\pm$1.38 | N/A |
|  | DrNAS [] | 94.36$\pm$0.00 | N/A | 73.51$\pm$0.00 | N/A | 46.08$\pm$7.00 | N/A | 26.73$\pm$2.61 | N/A |
|  | BOHB [] | 93.61$\pm$0.52 | $>$500 | 70.85$\pm$1.28 | $>$500 | - | - | - | - |
|  | GP-UCB [] | 94.37$\pm$0.00 | 58 | 73.14$\pm$0.00 | 100 | 41.72$\pm$0.00 | 40 | 40.60$\pm$1.10 | 11 |
| BO-based | BANANAS [] | 94.37$\pm$0.00 | 46 | 73.51$\pm$0.00 | 88 | 41.72$\pm$0.00 | 40 | 40.15$\pm$1.59 | 17 |
| NAS | NASBOWL [] | 94.34$\pm$0.00 | 100 | 73.51$\pm$0.00 | 87 | 53.73$\pm$0.83 | 40 | 41.29$\pm$1.10 | 17 |
|  | HEBO [] | 94.34$\pm$0.00 | 100 | 72.62$\pm$0.20 | 100 | 49.32$\pm$6.10 | 40 | 40.55$\pm$1.15 | 18 |
|  | TNAS [] | 94.37$\pm$0.00 | 29 | 73.51$\pm$0.00 | 59 | 59.15$\pm$0.58 | 26 | 40.00$\pm$0.00 | 6 |
| Transferable | MetaD2A [] | 94.37$\pm$0.00 | 100 | 73.34$\pm$0.04 | 100 | 57.71$\pm$0.20 | 40 | 39.04$\pm$0.20 | 40 |
| NAS | DiffusionNAG [] | 94.37$\pm$0.00 | 5 | 73.51$\pm$0.00 | 5 | 59.63$\pm$0.92 | 2 | 41.32$\pm$0.84 | 2 |
|  | POMONAG (Ours) | 95.42$\pm$0.12 | 1 | 75.94$\pm$0.24 | 1 | 63.38$\pm$0.51 | 1 | 68.82$\pm$0.19 | 1 |

NAS Evaluation on NASBench201. Table 2 compares the performance of POMONAG with that of other NAS techniques on the NASBench201 search space across the datasets CIFAR10, CIFAR100, Aircraft, and Oxford III Pets. POMONAG achieves the best accuracies on all datasets, using only one trained architecture per dataset, mirroring the approach taken for the MobileNetV3 search space. Specifically, for the CIFAR10 and CIFAR100 datasets, where all other Transferable NAS methods obtain accuracy results with a 95% confidence interval of 0.00 due to lookup procedures, POMONAG presents a non-zero value as its performances were recalculated over three different runs, confirming superior performance. It is crucial to note that POMONAG minimises the number of architectures to be trained to only the final architecture, thereby reducing computational complexity to a minimum.

**Table 3: Comparison of Spearman’s correlation coefficients for Performance Predictors between DiffusionNAG and POMONAG on the NASBench201 and MobileNetV3 search spaces. Higher coefficients indicate better predictions. ’Accuracy’, ’Params’, ’MACs’, and ’Latency’ refer to predictions for noisy architectures during diffusion, while ’Accuracy*’ represents predictions for denoised architectures after diffusion. ’N/A’ indicates experiments that could not be replicated; ’-’ indicates metrics not applicable to DiffusionNAG. Best scores are highlighted in bold.**
|  | NASBench201 | MobileNetV3 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Accuracy | Accuracy* | Params | MACs | Latency | Accuracy | Accuracy* | Params | MACs | Latency |
| DiffusionNAG | 0.687 | 0.767 | - | - | - | N/A | N/A | - | - | - |
| POMONAG | 0.855 | 0.884 | 0.666 | 0.656 | 0.470 | 0.857 | 0.988 | 0.828 | 0.847 | 0.618 |

Performance Predictors. Optimising the Performance Predictors is fundamental for effectively guiding the many-objective conditional generation process. The comparative analysis summarised in Table 3 highlights a marked superiority of the predictors employed in POMONAG compared to those of DiffusionNAG. In the context of NASBench201, a substantial increase in Spearman’s correlation for accuracy is observed: from 0.687 to 0.855 for noisy architectures and from 0.767 to 0.884 for denoised ones. POMONAG also stands out for introducing efficient predictors for parameters, MACs, and latency, absent in DiffusionNAG. These new predictors show significant correlations—0.666 for parameters, 0.656 for MACs, and 0.470 for latency – thus providing reliable guidance for many-objective optimisation. In the MobileNetV3 search space, POMONAG achieves even higher correlations, exceeding 0.8 for accuracy, parameters, and MACs. Appendix C presents a detailed analysis of the ablation studies related to these predictors, corroborating the effectiveness of the implemented modifications.

**Table 4: Comparison of generation metrics—’Validity’, ’Uniqueness’, and ’Novelty’—between DiffusionNAG and POMONAG on the NASBench201 and MobileNetV3 search spaces. Results show mean values $\pm$ 95% confidence intervals over three runs. Best scores are highlighted in bold.**
|  | NASBench201 | MobileNetV3 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | Validity $\uparrow$ | Uniqueness $\uparrow$ | Novelty $\uparrow$ | Validity $\uparrow$ | Uniqueness $\uparrow$ | Novelty $\uparrow$ |
| DiffusionNAG | 98.97$\pm$0.29 | 5.10$\pm$0.33 | 19.87$\pm$1.35 | 99.09$\pm$0.28 | 100.00$\pm$0.00 | 100.00$\pm$0.00 |
| POMONAG (ours) | 99.97$\pm$0.10 | 34.14$\pm$17.33 | 37.41$\pm$7.73 | 72.58$\pm$5.84 | 91.79$\pm$7.32 | 100.00$\pm$0.00 |

Generation Performance. The comparative analysis in Table 4 shows generation metrics between DiffusionNAG and POMONAG. Validity measures the percentage of generated architectures that adhere to the constraints of the search space; uniqueness assesses the diversity among the generated architectures; and novelty indicates the proportion of architectures not present in the training dataset. In the NASBench201 search space, POMONAG demonstrates significant improvements across all metrics, particularly in uniqueness (34.14% vs. 5.10%) and novelty (37.41% vs. 19.87%) compared to DiffusionNAG. For the MobileNetV3 search space, a slight decrease in validity is observed for POMONAG (72.58% vs. 99.09%), likely due to a more ambitious exploration of the search space driven by many-objective conditioning. Nonetheless, POMONAG maintains high levels of uniqueness (91.79%) and novelty (100%), illustrating its capability to generate diverse and innovative architectures while simultaneously achieving superior classification performance.

**Table 5: Comparative analysis of DiffusionNAG and POMONAG variants on the NASBench201 and MobileNetV3 search spaces. Metrics include accuracy, number of parameters (millions), MACs (millions), and latency (milliseconds). Values are means over the 15 image classification datasets considered, with $\pm$ 95% confidence intervals. Bold indicates the best performance, and underlined indicates the second-best. Green and red shading denote POMONAG’s improvements or deteriorations, respectively, relative to DiffusionNAG.**
|  | NASBench201 | MobileNetV3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Accuracy $\uparrow$ | Params [M] $\downarrow$ | MACs [M] $\downarrow$ | Latency [ms] $\downarrow$ | Accuracy $\uparrow$ | Params [M] $\downarrow$ | MACs [M] $\downarrow$ | Latency [ms] $\downarrow$ |
| DiffusionNAG | 77.83$\pm$10.86 | 1.02$\pm$0.01 | 18.02$\pm$0.21 | 18.83$\pm$0.46 | 87.29$\pm$6.32 | 7.86$\pm$1.05 | 100.75$\pm$8.18 | 32.13$\pm$2.95 |
| POMONAG${}_{\text{Eff}}$ | 70.25$\pm$12.46 | 0.10$\pm$0.03 | 1.27$\pm$0.63 | 6.51$\pm$0.77 | 88.66$\pm$5.28 | 3.47$\pm$0.01 | 33.69$\pm$0.85 | 17.96$\pm$0.76 |
| POMONAG${}_{\text{Bal}}$ | 77.91 $\pm$ 10.69 | 0.43 $\pm$ 0.18 | 7.33 $\pm$ 3.21 | 11.06 $\pm$ 2.12 | 89.02 $\pm$ 5.22 | 4.24 $\pm$ 0.48 | 38.02 $\pm$ 6.92 | 21.62 $\pm$ 3.14 |
| POMONAG${}_{\text{Acc}}$ | 81.89 $\pm$ 9.21 | 0.65 $\pm$ 0.11 | 11.40 $\pm$ 2.06 | 13.68 $\pm$ 1.17 | 89.96 $\pm$ 4.86 | 6.01 $\pm$ 0.37 | 90.92 $\pm$ 4.62 | 34.24 $\pm$ 2.19 |

Many-Objective Evaluation. A comprehensive comparison between DiffusionNAG and the three variants of POMONAG (Efficient, Balanced, Accurate) was conducted across both NASBench201 and MobileNetV3 search spaces considering all the key metrics involved in the many-objective optimisation – accuracy, number of parameters, MACs, and inference latency. The results, summarised in Table 5, represent the means over 15 diverse image classification datasets, depicted in more details in Appendix B.

In the NASBench201 search space, POMONAGAcc achieves the highest average accuracy (81.89%), surpassing DiffusionNAG’s 77.83%, while also improving upon all secondary metrics. This indicates that accuracy gains do not come at the expense of efficiency. The POMONAGEff variant achieves remarkable reductions in parameters (-90%), MACs (-93%), and latency (-65%) compared to DiffusionNAG, albeit with a trade-off in accuracy. This highlights POMONAG’s capability to generate highly efficient architectures suitable for resource-constrained environments. The POMONAG${}_{\text{Bal}}$ configuration offers a balanced compromise, maintaining an accuracy comparable to DiffusionNAG while significantly enhancing efficiency metrics.
In the MobileNetV3 search space, all POMONAG variants outperform DiffusionNAG in terms of average accuracy. Notably, POMONAGEff achieves significant reductions in parameters (-56%), MACs (-67%), and latency (-44%), demonstrating its effectiveness in generating compact and fast architectures suitable for devices with limited computational capabilities. The POMONAGAcc variant attains the highest average accuracy (89.96%), with a 24% reduction in parameters and a 10% reduction in MACs compared to DiffusionNAG. This illustrates that it is possible to enhance performance while also improving efficiency.
These results underscore POMONAG’s effectiveness in generating Pareto-optimal architectures that provide a balanced trade-off between accuracy and efficiency, catering to various deployment scenarios and resource constraints.

Generation and Training Time. The time required for generating and training the architectures depends on the search space, the selected POMONAG configuration, and the size of the dataset used for training. For the generation phase, the diffusion process takes an average of approximately 5 minutes and 45 seconds ($\pm$32 seconds) on NASBench201, and about 18 minutes and 15 seconds ($\pm$1 minute and 12 seconds) on MobileNetV3.
Regarding training times, utilising the optimised pipelines detailed in Appendix D, the training durations for NASBench201 range from approximately 2 hours and 33 minutes ($\pm$54 minutes) to 3 hours and 32 minutes ($\pm$1 hour and 19 minutes), depending on the POMONAG configuration. For MobileNetV3, training times vary from about 2 hours and 8 minutes ($\pm$31 minutes) to 2 hours and 26 minutes ($\pm$39 minutes), again contingent on the configuration used.
Considering that only a single generation and a single training run are required, and in light of the superior performances achieved in terms of both accuracy and efficiency, POMONAG represents a substantial advancement in the field of Neural Architecture Search. This reduction in computational time and resources not only accelerates the development process but also makes the approach more accessible for practical applications where time and computational budget are limited.

## 5 Conclusion and Future Directions

This paper has presented POMONAG, an innovative framework for generating neural architectures optimised in a many-objective context. By integrating advanced techniques like Many-Objective Reverse Diffusion Guidance, the creation of extended Meta-Datasets, and the adoption of improved Performance Predictors, POMONAG addresses and overcomes limitations of existing approaches, effectively combining many-objective and Transferable NAS paradigms.
The conducted experiments demonstrate that POMONAG is capable of generating architectures offering an effective balance between accuracy and computational efficiency. Specifically, it achieves superior performance across various datasets and search spaces requiring training only the identified optimal architecture.
As a future direction, extending POMONAG’s approach to other computer vision tasks – such as segmentation and object detection – is envisaged. This trajectory aims to develop a foundational NAS model for computer vision tasks that is both task-aware and dataset-aware, capable of effectively adapting to a variety of applications, domains, and computational constraints. Such an advancement would further enhance the adaptability and applicability of NAS methodologies in diverse real-world scenarios.

## Acknowledgments

This paper is supported by the FAIR (Future Artificial Intelligence Research) project, funded by the NextGenerationEU program within the PNRR-PE-AI scheme (M4C2, investment 1.3, line on Artificial Intelligence

## Appendix A Meta-Datasets

This appendix describes the search spaces utilised in this work, the encoding and sampling strategies employed for the creation of the Meta-Datasets, and introduces the Meta-Datasets themselves.

### A.1 NASBench201 Search Space

NASBench201, introduced by Dong et al. [], is a standardised search space for Neural Architecture Search (NAS) tasks within the domain of image classification. This search space is confined to the composition of an optimal cell, which is used in series to construct the final architecture.
A cell comprises four fixed nodes, representing the summation operation of the feature maps received as input, and six possible connections between the nodes. The connections are associated with operations for transforming and adapting the feature maps. The available operations are Zeroise, Skip connection, 1x1 2D Convolution, 3x3 2D Convolution, and 3x3 Average Pooling. Each convolution operation is followed by a Batch Normalisation layer and a ReLU activation.
This leads to a total of $5^{6}=15,625$ unique architectures within the NASBench201 search space. The evaluation of models in NASBench201 is conducted through full training on three datasets: CIFAR10, CIFAR100, and ImageNet16. For each architecture, metrics such as accuracy, cross-entropy loss, training time, number of parameters, MACs, and inference latency are provided.

Architecture Encoding and Sampling. The encoding of NASBench201 architectures used in this work involves two complementary matrices: an operations matrix and an adjacency matrix, as proposed by Dong et al. [].

The operations matrix describes the transformations applied to the feature maps between the nodes of the graph. This 8$\times$7 matrix is structured such that each cell represents a possible edge of the directed acyclic graph. There are eight rows—one for the input, six for the possible intermediate connections, and one for the output—and seven columns, one for each possible operation, including placeholders for the input and output. The matrix is binary: a ’1’ indicates that the connection is assigned the corresponding operation, and a ’0’ otherwise. In this sense, each row has exactly one ’1’, since each connection must be assigned an operation.
Parallelly, the adjacency matrix defines the connection structure of the graph. Also of size 8$\times$8, this binary matrix is fixed for all architectures in NASBench201. The elements of the matrix are ’1’ to indicate the presence of a connection between two nodes summing the feature maps, and ’0’ to indicate the absence of a connection. Again, the first row and column represent the input, while the last row and column represent the output.
The sampling of architectures is optimised by exploiting the knowledge of the evaluations present in NASBench201. Specifically, the top-250 most performant architectures are identified by calculating the average relative to the three datasets used by the authors for their evaluation. Sampling is conducted such that there is a 95% probability of sampling from this subset of high-performing architectures, and a 5% probability from the rest of the search space.

### A.2 MobileNetV3 Search Space

The MobileNetV3 search space [], implemented via Once-for-All (OFA) introduced by Cai et al. [], is a flexible search space for NAS tasks in the domain of image classification on mobile devices. This search space is defined by a pre-trained super-network that supports various architectural configurations, allowing exploration of a wide range of sub-networks.
The super-network comprises a sequence of inverted bottleneck blocks, similar to those in MobileNetV2 []. The main dimensions of the search space are:

- •
Depth. Each stage of the network can have a variable number of blocks, typically chosen from 2, 3, 4.
- •
Expansion ratio. The number of channels in each layer can be adjusted using two width multipliers.
- •
Kernel size. For depthwise convolutions, the kernel size can be 3$\times$3, 5$\times$5, or 7$\times$7.
- •
Input resolution. The input image can have variable dimensions, typically from 128$\times$128 to 224$\times$224.

The input resolution is fixed at 224$\times$224. Each of the five stages in the super-network, as provided by the original implementation, can include a Squeeze-and-Excitation module and use different activation functions (ReLU or h-swish) based on its position within the architecture.
This approach leads to a search space comprising approximately $10^{19}$ unique architectures. The evaluation of models in this space is conducted through the selection of subgraphs of the pre-trained super-network, eliminating the need to train each architecture from scratch.

Architecture Encoding and Sampling. As with NASBench201, the encoding of MobileNetV3 architectures used in this work involves representations related to the operators of the architecture and the internal connections within the network.
The operators are represented by a 21$\times$9 matrix, with the first row dedicated to the placeholder of the width multiplier, followed by one row for each of the 20 blocks within the architecture.
If the width multiplier is set to 1.0, then the first row is populated with zeros; if it is set to 1.2, the first row is populated exclusively with ones.
From the second row onwards, the columns are divided into three groups of three, each representing a combination of expansion ratio and kernel size. The first group corresponds to an expansion ratio of 3, the second to 4, and the third to 6. Within each group, the three columns represent kernel sizes of 3$\times$3, 5$\times$5, and 7$\times$7, respectively. Each row of the matrix contains a ’1’ in the position corresponding to the chosen combination of expansion ratio and kernel size for that block, and zeros elsewhere. If a block is inactive (as indicated by the depth mask), its corresponding row is filled with zeros.
The adjacency matrix, on the other hand, represents the connections between the blocks of the architecture. It is a square matrix of size 20$\times$20, where each element (i, j) is ’1’ if there is a direct connection from block $i$ to block $j$, and ’0’ otherwise. The structure of this matrix reflects the topology of the network, taking into account the variable depth of each stage.
For this search space, the sampling of architectures is conducted uniformly with respect to the values of the width multiplier, depth, expansion ratio, and kernel size, drawing from a possible number of architectures equal to $2\times 10^{19}$ (the factor of 2 arises from the addition of the width multiplier).

### A.3 Meta-Datasets Distributions

Figure 2 illustrates the distributions of key metrics for the Meta-Datasets created in this study. For the NASBench201 search space, 10,000 architectures were sampled, while 20,000 were selected for the MobileNetV3 space. These architectures were sampled from their respective search spaces and then trained using the proposed pipelines on a dataset of 20 classes with 1,000 samples per class, randomly extracted from ImageNet32. The reported accuracy performance is based on a test set of 10 samples per class for the same 20 classes.
The distributions shown are consistent with the characteristics of their respective search spaces. Notably, the graphs clearly demonstrate that MobileNetV3 architectures are, as expected, more complex yet relatively lightweight, and also more performant. This is due to their incorporation of more advanced architectural designs compared to NASBench201 architectures, as well as the benefit of fine-tuning during training.

Figure: Figure 2: Distributions of key metrics in the proposed Meta-Datasets. Histograms display accuracy, number of parameters, MACs, and latency for architectures sampled from the NASBench201 (top row, 10,000 architectures) and MobileNetV3 (bottom row, 20,000 architectures) search spaces. The architectures were randomly selected and trained on subsets of ImageNet32, demonstrating the diversity within each search space.
Refer to caption: extracted/5890392/metadatasetsdistributions.png

## Appendix B Extensive Results

Figure: Figure 3: Performance comparison of architectures generated by DiffusionNAG and POMONAG on 15 datasets using the NASBench201 search space. Each plot illustrates accuracy versus a secondary metric (number of parameters, MACs, or latency). The blue lines represent POMONAG’s Pareto Fronts, with circles highlighting the efficient (yellow), balanced (purple), and accurate (green) variants. Orange points depict DiffusionNAG results. Error bars show 95% confidence intervals averaged over three runs. Dashed lines indicate DiffusionNAG results as reported by An et al. [].
Refer to caption: extracted/5890392/pomonagbenchmarknb201.png

Figure: Figure 4: Performance comparison of architectures generated by DiffusionNAG and POMONAG on 15 datasets using the MobileNetV3 search space. Each plot illustrates accuracy versus a secondary metric (number of parameters, MACs, or latency). The blue lines represent POMONAG’s Pareto Fronts, with circles highlighting the efficient (yellow), balanced (purple), and accurate (green) variants. Orange points depict DiffusionNAG results. Error bars show 95% confidence intervals averaged over three runs. Dashed lines indicate DiffusionNAG results as reported by An et al. [].
Refer to caption: extracted/5890392/pomonagbenchmarkmbnv3.png

Figures 3 and 4 present a comprehensive comparison of neural architectures generated by DiffusionNAG and POMONAG across 15 diverse datasets: CIFAR10, CIFAR100, Aircraft, Oxford III Pets, BloodMNIST, DermaMNIST, EuroSAT, FashionMNIST, OCTMNIST, OrganAMNIST, OrganCMNIST, PathMNIST, STL10, TinyImageNet, and TissueMNIST. The comparisons are made using the NASBench201 and MobileNetV3 search spaces, respectively. The plots illustrate the trade-offs between accuracy and secondary metrics (parameters, MACs, and latency) for each dataset.
POMONAG’s results are represented by Pareto Fronts, showcasing the range of optimal solutions. Three key variants are highlighted: Efficient, emphasising minimal secondary metric; Accurate, prioritising highest predicted accuracy; and Balanced, optimising the ratio of predicted accuracy to secondary metric.
DiffusionNAG’s results, both reproduced (orange points) and as reported by An et al. [] (dashed lines), provide a benchmark for comparison. The 95% confidence intervals, averaged over three runs, underscore the robustness of the results.
These visualisations demonstrate POMONAG’s capability to generate a diverse set of architectures, effectively balancing performance and efficiency across various datasets and search spaces. The consistent outperformance of POMONAG, particularly in terms of Pareto-optimal solutions, highlights its effectiveness in navigating complex Neural Architecture Search spaces.
It is important to note that the performances of POMONAG’s architectures, as well as those replicated from DiffusionNAG and represented in both figures, were explicitly calculated and are not estimates from the Performance Predictors.

## Appendix C Performance Predictors Ablation Study

An ablation study was conducted to evaluate the improvements introduced in the Performance Predictors on the NASBench201 search space. The study involved a series of modifications to the predictors and their training strategies, aiming to maximise the Spearman correlation between the predicted values and the actual metrics.

Initially, a new predictor was implemented using the same training pipeline as An et al. [], which yielded slight improvements in correlation scores over the baseline DiffusionNAG predictor. Specifically, the Spearman correlation for the accuracy of noisy architectures increased from 0.687 to 0.705, and for denoised architectures from 0.767 to 0.772.
Subsequently, the introduction of a novel training strategy significantly enhanced performance for both noisy and denoised architecture accuracy estimates. This new training strategy involved switching from the Adam optimiser to AdamW, incorporating a cosine annealing learning rate scheduler, and introducing a weight decay of $5\times 10^{-3}$. These modifications improved convergence and generalisation of the Performance Predictors, leading to Spearman correlations of 0.822 for noisy architectures and 0.854 for denoised architectures.

Expanding the Meta-Dataset size from 4,630 to 10,000 architectures further improved the results, with correlations reaching 0.842 for noisy architectures and 0.884 for denoised ones. This demonstrates the benefit of a larger and more diverse training set, providing the Performance Predictors with a richer array of examples to learn from.
Finally, replacing the ResNet18 Dataset Encoder with a Vision Transformer (ViT-B-16) feature extractor led to the best performance, achieving Spearman correlations of 0.855 and 0.884 for noisy and denoised architectures, respectively. These results indicate that both the architectural choices and the training strategies for the Performance Predictors have a substantial impact on their effectiveness, directly influencing the quality of guidance provided during the diffusion process.

**Table 6: Ablation study results for Performance Predictors on the NASBench201 search space. ’Performance Predictor’ specifies the training and the encoding strategy used for the predictors. ’Archs Training’ indicates the training pipeline for the architectures in the Meta-Dataset. ’Size’ denotes the Meta-Dataset’s cardinality. ’Accuracy’ and ’Accuracy*’ are Spearman correlations for noisy and denoised architectures, respectively. Best scores are highlighted in bold**
| Performance Predictor | Meta-Dataset | Spearman Correlation |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Training Strategy | Dataset Encoder | Archs Training | Size | Accuracy | Accuracy* |
| An et al.[] | ResNet18 | An et al.[] | 4630 | 0.687 | 0.767 |
| Ours | ResNet18 | An et al.[] | 4630 | 0.705 | 0.772 |
| Ours | ResNet18 | Ours | 4630 | 0.822 | 0.854 |
| Ours | ResNet18 | Ours | 10000 | 0.842 | 0.884 |
| Ours | ViT-B-16 | Ours | 10000 | 0.855 | 0.884 |

## Appendix D Hyperparameter Tuning and Optimisation

The training of the architectures included in the Meta-Datasets required meticulous hyperparameter tuning to maximise performance. For each search space – NASBench201 and MobileNetV3 – a comprehensive hyperparameter search was conducted to identify optimal training configurations.
The hyperparameter search involved exploring a range of values for key training parameters, such as the number of epochs, warm-up epochs, optimisers, learning rates, learning rate schedulers, weight decay, label smoothing, and various data augmentation techniques. The search space for these hyperparameters is summarised in Table 7, providing an overview of the parameters considered during the optimisation process.
The optimal training pipelines identified for each search space are detailed in Table 8.

**Table 7: Hyperparameter search space used for optimising the training pipelines for architectures in the Meta-Datasets.**
| Hyperparameter | Search Space |
| --- | --- |
| Epochs | {50, 100, 200} |
| Warm-up epochs | {-, 10, 20} |
| Optimiser | {SGD, Adam, AdamW} |
| Learning rate | {1$\times$10^-1, 1$\times$10^-2, 1$\times$10^-3, 1$\times$10^-4} |
| Learning rate scheduler | {-, Cosine annealing, Cosine annealing with restarts} |
| Weight decay | {-, 5$\times$10^-3, 5$\times$10^-4, 5$\times$10^-5} |
| Label smoothing | {-, 0.1} |
| Heavy augmentation | {-, AutoAugment, TrivialAugmentWide, RandAugment, AugMix} |
| Random horizontal flip | {-, 0.5} |
| Random erasing (Gaussian) | {-, 0.2} |
| MixUp | {-, 0.2} |
| CutMix | {-, 0.2} |

**Table 8: Optimal training hyperparameters identified for architectures in the NASBench201 and MobileNetV3 search spaces.**
| Search Space | NASBench201 | MobileNetV3 |
| --- | --- | --- |
| Weights initialisation | HeNormal initialisation | ImageNet-1k pre-training |
| Mixed precision | True | True |
| Training technique | Ordinary | Fine-tuning |
| Epochs | 200 | 50 |
| Warm-up epochs | 20 | - |
| Early stopping patience | 120 | 30 |
| Optimiser | SGD | AdamW |
| Learning rate | 1$\times$10^-1 | 1$\times$10^-3 |
| Learning rate scheduler | Cosine annealing | - |
| Weight decay | 5$\times$10^-4 | 5$\times$10^-5 |
| Label smoothing | - | 0.1 |
| Resizing | 32$\times$32 | 224$\times$224 |
| Heavy augmentation | TrivialAugmentWide | AugMix |
| Random horizontal flip | 0.5 | 0.5 |
| Padding (constant) | 4 | 21 |
| Random crop | 32$\times$32 | 224$\times$224 |
| Random erasing (Gaussian) | 0.2 | - |
| MixUp | - | 0.2 |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Moreover, I have the following questions:

- Can the authors provide more theoretical or empirical justification for the scaling factors in the Pareto Front Stretching process? How sensitive is the model to these values?

- Can the authors provide more detail on the architecture sampling process, dataset splits, and hyperparameter tuning methods used in the experiments? This is particularly important for the performance predictors.
soundness: 2
strengths: The motivation to extend DiffusionNAG to a many-objective setting is valid and POMONAG does so by incorporating both accuracy and efficiency metrics like latency and MACs, which are critical for resource-constrained environments. The paper provides extensive experimental comparisons with DiffusionNAG, including evaluations across multiple datasets and search spaces, which helps demonstrate the general applicability of POMONAG.
Balancing the different objectives being optimized is also very important in my opinion. The authors do so by proposing a pareto front filtering and stretching subroutine.
summary: This paper introduces POMONAG, an extension to DiffusionNAG that applies a many-objective diffusion model to optimize neural architecture generation for many-objective optimization. By incorporating additional performance predictors for hardware efficiency metrics such as number of parameters, multiply-accumulate operations (MACs), and inference latency, POMONAG aims to provide a more balanced approach to architecture optimization across accuracy and computational efficiency. Experiments validate POMONAG’s efficacy on two major CNN search spaces (NASBench201 and MobileNetV3).
weaknesses: I have the following main concerns related to this submission, which I believe were crucial in the final decision:

- **Incremental Contributions**: Although POMONAG claims to extend DiffusionNAG’s capabilities by addressing more objectives, the modifications appear incremental and lack substantial theoretical advancement. More specifically, I see the adaptation of diffusion models to accommodate multiple objectives, as described in section 3.1, more as a technical modification rather than a novel conceptual framework. I would recommend the authors to reiterate over their methodology and pinpoint the main contributions of their approach.

- **Experimental Evaluation**: The benchmarks that POMONAG was evaluated contain only CNN spaces. It would be beneficial for the paper if the authors would demonstrate the efficacy of POMONAG in Transformer search spaces, such as the one from HW-GPT-Bench [1]. Most importantly, in the multi-(many-)objective experiments, the proposed method is not compared to any baseline. I would recommend the authors to add baselines in their experimental evaluation and report hypervolume indicator together with the individual objective values, as well as the search time. Ultimately, I would also be interested in visualizing the pareto front plots in the main paper. As for baselines, you can find a non-exhaustive list of simple ones in SyneTune (https://syne-tune.readthedocs.io/en/latest/getting_started.html#supported-multi-objective-optimization-methods). Finally, the experiments lack a thorough ablation study that demonstrates the impact of POMONAG’s unique contributions independently of DiffusionNAG’s foundational structure. 

- **Clarity and Presentation**: The paper seems to have a somehow fragmented structure, making it challenging for readers to follow the main contributions and crucial take-away points. Equations are not thoroughly explained, and there is a heavy reliance on citations from DiffusionNAG rather than a detailed elaboration of POMONAG itself, making the paper not self-contained. One major point here, which I have also pointed out to the AC, is that the authors have used a smaller font size starting from page 4. The guidelines clearly state that the maximum page limit is 10 and that means 10 pages with the default font size, not a smaller one. I suggest the authors that in future submissions they adhere to the submission guidelines.


-- References --

[1] Sukthanker et al. HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models. In NeurIPS 2024 DBT

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: See above as the questions are mostly addressing the weakness of this paper.
soundness: 2
strengths: The motivation of this study, introducing multi-objective evaluation in NAS, is commendable as a task in reality is often not just about accuracy.  Other metrics should be considered simultaneously as well.

The writing is easy to follow. 

It is nice to see equations with highlights of different colours.
summary: This study improved DiffusionNAG by introducing a multi-objective approach which modifies DiffusionNAG's reverse diffusion process as a reverse diffusion guidance process.  Other than accuracy, #params, MACs and inference latency are also considered in the multi-objective metrics.  The proposed method POMONAG has been tested on NASBench201 and MobileNetV3 with 15 image classification tasks, showing better performance than DiffusionNAG and a series of other methods.
weaknesses: **First of all**, the work claims to be on Pareto multiobjective search for architectures.  However, that point is not obvious from the paper.  
* What are the benefits of using the proposed POMONAG?  
* How can a Pareto front be generated and utilized? Need to explicitly demonstrate how POMONAG generates and utilizes Pareto fronts.
* How can users select architectures from the Pareto front according to their needs or under different circumstances?  Show examples of such selection based on different priorities, for example prioritizing small-size architectures for portable devices or focusing on latency reduction etc.  
* It seems non-dominated sorting is absent. Explain how non-dominated sorting is incorporated or can be incorporated in POMONAG.
* In its current form, the paper reads like a combination or integration of single-objective evaluations rather than a multi-objective evaluation.  The equation of POMONAG at Line 209/210 is a linear combination of four objectives.  Please clarify if the linear combination of objectives is intended as a scalarization approach.  If so, discuss its limitations.
--- 
**Secondly**, the performance of POMONAG appears better than DiffusionNAG and other methods shown in the paper.  However many SOTA methods, especially zero proxy methods are missing.  Their reported performance is similar or even better, for example, SWAP-NAS by Peng et al, ICLR'24, ZiCo by Li et al, ICLR'23, MeCo by Jiang et al, NeurIPS'23.
* Include a comparison with these SOTA methods. If a direct comparison is not possible, explain why and discuss the limitations of the current evaluation.
* Discuss how POMONAG's approach differs from or improves upon zero-proxy methods. 

--- 
**Thirdly**, the computational cost aspect of POMONAG is weak.  The section "Generation and Training Time" should be better presented.  The method requires a diffusion generation phase which takes extra time.  That itself is a disadvantage.  Also timewise, POMONAG cannot claim superiority as recent methods mentioned earlier are faster.
* Present a detailed table comparing computational costs (including generation and training time) of POMONAG with other methods, including these zero proxy methods mentioned above.  Seemingly these methods are faster. If POMONAG is indeed slower, discuss potential optimization strategies.
* Discuss the trade-offs between the additional diffusion generation phase and the method's performance gains.  Justify why the additional computational cost might be worthwhile.
---   
**Other points:**
 
The link at Line 091 is showing.  Also, including the code and dataset would be helpful for the assessment.

--- 

Fig 1 is not quite readable.  The figure further makes POMONAG look like three single-objective tasks combined rather than a four-objective task.
* Improve readability, especially on the right-hand side.
* Better illustrate the integration of all four objectives in a unified multi-objective framework if these objectives are not just simply added together (*see the first part of my comments*).
 * Provide a clearer visual representation of how POMONAG handles the trade-offs between objectives (*see the first part of my comments*).
--- 
Line 186, the term noisy architecture is not explained.  
* Provide a brief explanation of what "noisy architecture" means in this context and how it relates to the diffusion process in DiffusionNAG.
--- 
Equations and their connection to the processes/algorithms are not numbered and not clearly explained.   
* Number all equations for easy reference
* Clearly label the equation at Line 183. Is this equation for the Reverse Diffusion Process? Clarify that connection.
* Provide a brief explanation of the symbols used in this equation and other key equations.
* Explain the purpose of transformation s_θ(A_t,t).
* Explain the exact differences between the Reverse Diffusion Process and the Reverse Diffusion Guidance Process.
--- 
Line 280, "Four are dedicated to the respective estimation of accuracy, parameters, MACs, and inference latency of noisy architectures during the diffusion phase. " 
* Explain why not use these four metrics for denoised architectures as well.
* Justify the point that the denoised architecture uses accuracy as its only metric.
--- 

Explain the reason why POMONAG utilises Vision Transformer ViT-B-16 instead of other models (Line 286).

--- 
It is good to see the Spearman correlation experiment.  That is very important in NAS studies.  However, for a thorough comparison of correlation, it should be done on a set of tasks like NAS-Bench-Suite-Zero (Krishnakumar et al. NeurIPS'22).
* Perform a similar thorough comparison comparing correlations on different tasks using different search spaces.
--- 
In lines 400-402, the same latex problem appeared several times, ` not ' for the left quotation marks, Accuracy, Params, MACS ... \
* Fix these formatting issues.
--- 
Validity, uniqueness and novelty are nice metrics for a population of solutions but not so critical for tasks that focus on accuracy and speed.  What is the point of being excellent on these points but without good accuracy and speed?
* Explain the significance of these three additional metrics: validity, uniqueness and novelty.
* Show example how these measures can help improve the quality of generated architectures in POMONAG.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: I have several questions about this work.

1, How to decide the scaling factors? Since the intervals are [1000,5000], [100,500], [100,500], [100,500], and the values seesm to be integer, then the whole factor space equals 4000 * 400 * 400 * 400, which is quite huge. And the authors present one setting for NASBench201 and other experiments, respectively, so I am wondering whether there is some method or strategy to choose such factors? 

2, This work extends the basic motivation of DiffusionNAG, which is rather good and natural. Such extension include three more factors, including number of parameters, number of MACs, and the inference latency. But I am curious that, how about the performance of POMONAG if just considering adding one factor? 

3, From one factor, say, accuracy, to three more factors seems strenghening the proposed POMONAG, but my question is, the working mechanism of DiffusionNAG and POMONAG the same different? Although the two diffusion processes consider different factors, which is the obvious difference, but the analysis or discussion is important to interpret this issue.
soundness: 3
strengths: This paper introduces the ParetoOptimal Many-Objective Neural Architecture Generator (POMONAG), extending DiffusionNAG through a many-objective diffusion process. POMONAG simultaneously considers accuracy, the number of parameters, multiply-accumulate operations (MACs), and inference latency. The experiments validate the performance of the proposed model.
summary: This paper is a direct extension based on DiffusionNAG, which can deal with multi-objective optimization in NAS. These objectives include accuracy, the number of parameters, multiply-accumulate operations (MACs), and inference latency. This motivation is good and natural, and the authors expressed their work clearly, from the motivation to the experiments results. Some details need to be clarified.
weaknesses: 1, The multi-objective optimization problem formulation in this work can be given first, which then can be solved by the proposed weighted factors in the reverse diffusion process. But maybe the authors can consider other ways to sovle this. For example, using four single reverse diffusion process each targeting one factor, as DiffusionNAG did, then using multi-objective optimization for further trade-off may also work well.

2, The theoretical analysis should be strehghen. One objective to many objective is a breakthrough, but such process needs more analysis or discussion. Current work lacks such in-depth thinking. 

3, Several predictors are needed in this work, but the detailed information these predictors are missing.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the weaknesses. If the concerns raised are well addressed, I am glad to increase my rating.
soundness: 3
strengths: 1) The idea the overall framework of the proposed POMONAG method is simple and easy to understand. 
2) The details of the method and experiments are clearly stated. 
3) Generating neural architectures in the multi-objective manner is an important research topic.
summary: This paper presents the POMONAG method to generate neural architectures in the multi-objective manner. Specifically, the overall framework of POMONAG is designed based on that of DiffusionNAG, in order to achieve better performance in terms of number of parameters, MACs, and inference latency beyond the accuracy. There are four key parts designed to achieve this goal, i.e., the many-objective reverse diffusion guidance, the meta-dataset, the score network and performance predictors, and the pareto front filtering and stretching. The experimental results in NAS-Bench-201 and MobileNetV3 search spaces demonstrates the effectiveness of the proposed method.
weaknesses: 1) My major concern is about the motivation of this work. Specifically, there are four objectives considered, i.e., the accuracy, the number of parameters, MACs, and the inference latency. However, the last three objectives do not demonstrate conflict relationship. For instance, the smaller number of parameters seems certain to lead to lower inference latency. In this case, the necessity for adopting multi-objective optimization is limited. 
2) The novelty of the proposed method needs further discussion. Specifically, the proposed method seems to build on DiffusionNAG with the cooperation of the multi-objective optimization. It seems that the POMONAG is just a simple combination of these methods. More discussions in terms of the seminal contribution of POMONAG is needed. 
3) How the hyperparameters $k_{\phi}$, $k_{\pi}$, $k_{\mu}$, and $k_{\lambda}$ determined? It is suggested to provided more details in terms of the hyper-parameter study for these hyperparameters. 
4) The search cost of POMONAG is not well presented. In the pipeline of POMONAG, I think the pre-training process, the training of the score network, and the training for the performance predictors will introduce much additional search cost beyond the architecture generation. However, I cannot find any details about the overall search cost and the search cost for the above components. 
5) I am curious about why only one trained architecture is enough for POMONAG? Maybe more discussions or analysis are helpful to give more insights for this point. 
6) Lack of experimental results on more challenging tasks (i.e., the classification accuracy on ImageNet-1K). More results on such datasets are helpful to enhance the experiments.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: How did you choose the search spaces to apply POMONAG? 
The algorithmic contribution seems like a limited extension of DiffusionNAG. What complication arose from integrating multi-objective NAS into DIffusionNAG?
soundness: 3
strengths: Strong writing, ideas are explained well and thorough
The experiments are presented well and results are thorough
Novelty is presented in 2 algorithmic improvements and the contribution of a multi-objective meta dataset
summary: The work presents an extension to DiffusionNAG and incorporates multi-objective search.  Model complexity, computational efficiency, and inference latency are key measures captured through number of parameters, MACs, and latency estimation. These measures are recorded in a meta dataset for NASBench201 and MobileNetV3 with 10k and 20k architectures respectively. During search, pareto front filtering segments three regions corresponding to high accuracy, high efficiency, and best balance of the two using the auxiliary metrics from earlier. The experimental results are promising across a sufficiently diverse set of benchmarks.
weaknesses: For transferable NAS, the choice of benchmarks are interesting, TransNASBench provides a NAS dataset specifically for transferability in NAS. Exploring performance on this dataset would have been nice
MobileNetV3 and NB201 are also fairly dated search spaces, performance in more recent search space or architecture styles (vit) should be explored
The specific details of the algorithmic contribution are a bit vague. How is pareto front filtering done? 
ImageNet results are sparse and comparison to modern NAS methods on this benchmark are sparse

Meta-review: This study proposes POMONAG, extending DiffusionNAG for multi-objective optimization. While the method is clearly presented with experiments across 15 datasets, reviewers noted limited novelty, as the approach largely builds on DiffusionNAG with minor extensions. The lack of comparisons with state-of-the-art multi-objective and zero-shot NAS methods, along with insufficient detail on ablations and hyperparameter selection, weakens the contribution. The AC agrees with these concerns and recommends rejection.

Decision: Reject