Multi-objective Differentiable Neural Architecture Search
Authors: Rhea Sanjay Sukthanker
Arber Zela
Benedikt Staffler
Samuel Dooley
Josif Grabocka
Frank Hutter
Venue: ICLR 2025 Poster (2025)
Keywords: hardware efficiency
neural architecture search
network compression
Primary area: optimization
TL;DR: We introduce a novel hardware aware differentiable multi-objective architecture search method that efficiently profiles the whole Pareto-front.

Abstract: Pareto front profiling in multi-objective optimization (MOO), i.e., finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives that require training a neural network. Typically, in MOO for neural architecture search (NAS), we aim to balance performance and hardware metrics across devices. Prior NAS approaches simplify this task by incorporating hardware constraints into the objective function, but profiling the Pareto front necessitates a computationally expensive search for each constraint. In this work, we propose a novel NAS algorithm that encodes user preferences to trade-off performance and hardware metrics, yielding representative and diverse architectures across multiple devices in just a single search run. To this end, we parameterize the joint architectural distribution across devices and multiple objectives via a hypernetwork that can be conditioned on hardware features and preference vectors, enabling zero-shot transferability to new devices. Extensive experiments involving up to 19 hardware devices and 3 different objectives demonstrate the effectiveness and scalability of our method. Finally, we show that, without any additional costs, our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets, including MobileNetV3 on ImageNet-1k, an encoder-decoder transformer space for machine translation and a decoder-only space for language modelling.

Published as a conference paper at ICLR 2025
MULTI-OBJECTIVE DIFFERENTIABLE NEURAL ARCHITECTURE SEARCH
Rhea Sanjay Sukthanker'; Arber Zela't Benedikt Staffler?, Samuel Dooley?, Josif Grabocka', Frank Hutter
University of Freiburg, Bosch Center for AI, Meta, 4 University of Technology Nuremberg, 5 ELLIS Institute Tübingen
ABSTRACT
Pareto front profiling in multi-objective optimization (MOO), i... finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives that require training neural network. Typically, in MOO for neural architecture search (NAS), aim to balance performance and hardware metrics across devices. Prior NAS approaches simplify this task by incorporating hardware constraints into the objective function, but profiling the Pareto front necessitates computationally expensive search for each constraint. In this work, we propose a novel NAS algorithm that encodes user preferences to trade-off performance and hardware metrics, yielding representative and diverse architectures across multiple devices in just a single search run. To this end, we parameterize the joint architectural distribution across devices and multiple objectives via hypernetwork that can conditioned on hardware features and preference vectors, enabling zero-shot transferabilityy to new devices. Extensive experiments involving up to 19 hardware devices and different objectives demonstrate the effectiveness and scalability of our method. Finally. show that, without any additional costs, our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets, including MobileNetV3 on ImageNet- 1k, an encoder-decoder transformer space for machine translation and a decoder-only space for language modelling.
INTRODUCTION
The ability to make good tradeoffs between predictive accuracy and efficiency (in terms of latency and/or energy consumption) has become crucial in age of ever increasing neural networks complexity and size (Alabdulmohsin et al. 2023; Hoffmann et al,, 2022 Kaplan et al. 2020 Zhhii et al. 2022) and a plethora of embedded devices. However, finding the right trade-off remains challenging task that typically requires human intervention and a lot of trial-and-error across devices. With multiple conflicting objectives, it becomes infeasible to optimize all of them simultaneously and return a single solution. Ideally, NAS should empower users to choose from a set of diverse Pareto optimal solutions that represent their preferences regarding the trade-off between objectives.
Neural Architecture Search (NAS) (White et al.) 2023) provides a principled framework to search for network architectures in an automated fashion. Several works (Cai et al.. 2020 Chen et al., 2021a Elsken et al. 2019b Wang et al. (020b) have extended NAS for multi-objective optimization (MOO), considering performance and hardware efficiency metrics like latency and energy consumption. However, the best of our knowledge, no existing gradient-based method returns the full Pareto front for the MOO problem at hand without repeating their search routine multiple times with different hardware constraints.
In this work, we propose a scalable and hardware-aware Multi- Objei-Obeeetive Differentiable Neural Architecture Search (MODNAS) algorithm that efficiently trains a single supernet which can be used read off Pareto-optimal solutions for different user preferences and different target devices, without any additional search steps. To search across devices, we frame the problem as multi-task, multi-objective optimization problem, where each task (device) has multiple (conflicting) objectives,
"Equal contribution. Email to: (sukthank, zelaa]@cs. wnw uni-freibuurgde

---

Published as a conference paper at ICLR 2025
Supernet f( (10 (0b). .0
Volrict [i nef +...+ (
Dir(S)
(n
Ho(r,d)
Architect A (
T
:
s6 vo(ni)) L tnal) +... !
Mixup
dual)
Updata
Σ
using NGD:
4em
cetaProd.ctor (0b1.2)
Device
Device
MetaProdictor [ObJ
Lig
Fa
47
:salarizations
Jau embráá.
fl compermet
P2(-) predictor
Pe (a.
15
lass
He
hypenee
Device
arch param
Figure I: MODNAS overview. Given set of T devices, MODNAS seeks to optimize M (potentially conflicting) objectives across these devices. To this end, it employs a Metatlypernetwork (r, de). that takes input a scalarization r, representing the user preferences, and a device embedding d,, to yield an un nn-noomaaized architectural distribution á. The Architect uses á to sample differentiable discrete architectures, used in the Supernetword rk to estimate accuracy and in the MetaPredictor to estimate the other M - 1 loss functions (e.g. latency, energy consumption) for every device. By iterating over devices and sampling scalarizations uniformly from the M -dimennsional simplex, at each iteration we update the MettaHypernetworr using multiple gradient descent (MGD).
e.g., classification accuracy and latency. The user's preferences are modelled by preference vector that defines a scalarization (weighted sum), of the different objectives. This preference vector, along with features of the hardware of interest, is fed to hypemetwork (Ha et al. (017) that outputs continuous architectural parameters a. To search in the space of architectures, we employ a one-shot model and bi-level optimization scheme, is typically done in gradient- based NAS. In case, however, the upper-level parameters are the hypernetwork weights, optimized in expectation over different preference vectors and hardware devices multiple gradient descent (Désidéri) (0012).
To evaluate method, conduct experiments multiple NAS search spaces, including CNN and Transformer architectures, and up to 3 objectives across 19 hardware devices. While other NAS methods that utilize hardware constraints in their search objectives require substantial search costs both for each new constraint and each new hardware, MODNAS addresses both in a zero-shot manner, without extra search cost, while yielding higher quality solutions.
Our contributions can be summarized as follows:
We present a principled approach for Multi-objective Differentiable NAS, that leverages hyper- networks and multiple gradient descent for fast Pareto-Front approximation across devices.
2. This work is the first to provide global view of the Pareto solutions with just a single search run, without the need repeat search or fine-tune on new target devices.
Extensive evaluation of our method across 4 different search spaces (NAS- TAAS--Benc--20 Mo- bileNetV3, an encoder-decode and a decoder-only Transformer space), 3 tasks (image classi- fication, machine translation and language modeling), and up to hardware devices and objectives, show both improved efficiency and performance in comparison to previous approaches that use a constrained objective in their search.
To facilitate reproducibility, we provide our code htttt::/ggithbb.cmm http:///ttuubbcccmmuuuummm///mm/
BACKGROUND AND RELATED WORK
In this section, before describing our algorithm, we introduce some basic concepts, definitions and related work. Refer Appendix D for an extended related work.
Multi-objective optimization (MOO) for Multi-Tass Taski Learning. Consider multi-task dataset 'D consisting of N instances, where the feature vector of the i th instance is denoted as x; E X', and the M-many associated target variables as y y1 ,..., MM E yMM Moreover, consider there exists family of parametric models fæ; w) X + {3' .... x M) parameterized by w, that maps the input æ to the joint space of the multiple tasks. To simplify the notation, we denote the prediction of the m-th task as fm w): X + ymm and the respective loss C.'(w 2 N EN gm (y.", f fr w)). The vector of the values of all loss functions is denoted L(w) A (C) (w) ..., LM (w)). MOO
2

---

Published as a conference paper at ICLR 2025
then seeks to find set of Pareto-optimal solutions w" that jointly minimize L(w]
e E argmin L(w)
(I)
Definition 2.1. (Pareto Optimality); A solution w2 dominates w1 iff C (w2) S C (w1). W' E (1...., M), and L(w;) A L(wz). In other words, a dominating solution has a lower loss value on at least one task and no higher loss value on any task. A solution w² is called Pareto optimal iff there exists no other solution dominating w².
Definition 2.2. (Pareto front): The sets of Pareto optimal points and their function values are called Pareto set (Pw) and Pareto fromt (PI = (L(W)wePe. }), respectively.
Linear Scalarization. In MOO, a standard technique to solve the M-dimensional problem is
using preference vector r E S _ [R-M ΣM
m= I'm
1. 2 O. Vm E
M}} in
the M-dimensional probability simplex (Lin al.
2019 Mahapatra & Rajan] 2020; Ruchte
Grabocka 2021). Every E 8 yields a convex combination of the loss functions in Equation| L(w). Given preference vector v, one can apply standard, single-objective opti- mization algorithms to find minimizer w argmin L (ww). By sampling multiple vectors. one can compute Pareto-optimal solutions w that profile the Pareto front. Several methods (Hoang cet al.
as C, w
2023 Lin et al. 2020; Navon et al. 202 Phan al. 2022) employ hypernetwork (Ha et al.
2017 to generate Pareto-optimal solutions given different preference vectors as input. In this work, we utilize a hypernetwork conditioned on scalarizations generate Pareto-optimal architectures. Furthermore, we also extend the hypernetwork by conditioning it on different task vectors.
Multiple Gradient Descent (MGD). MOO can be solved to local optimality via MGD (Désidéri 2012), natural extension of single-objective gradient descent, which iteratively updates towards a direction that ensures that all tasks improve simultaneously (called Pareto improvement): w' <- w Eg where Iie is a vector field that needs be determined. If denote by gm V n V. (ww) the gradient of the m-th scalar loss function, via Taylor approximation. the decreasing direction of . when we update w towards Iw is given by (ow Iw) R (C""("" LTh (w))/F. In MGD g is chosen to maximize the slowest update rate among all objectives:
{
Fw α
arginax ERd flo ||
min (ow Fw m MM(M)
(2)
The early work of Désidéri (2012) has been extended in various settings, particularly multi-task
learning, with great promise (Lin et al. 2019 Liu & Vicente, [02T]] Mahapatra & Rajan 2020; Sener & Koltun 2018), but these approaches are applied to mainly a fixed architecture and extending them to supernet subsuming a search space of multiple architectures is non-trivial.
One-shot NAS and Bi-Level optimization. With the architecture space being intrinsically discrete, large (often consisting of upto 10 architectures) and hence expensive to search on, most existing differentiable NAS approaches leverage the weight sharing paradigm and continuous relaxation to enable gradient descent (Bender et al., [2018] Chen et al. 20216 Dong & Yang 2019 Liu al.
20196 2023; Movahedi et al.) 2022; Pham et al.) 2018 818.1 Xie et al. 2019: Xu et al. 2020a] Zhang et al. 2021). lypieally, in these approaches, architectures are stacks of cells, where the cell structure represented as directed acyclic graph (DAG) with N nodes and E edges. Every transition from node to j. i.e. edge (i.j). is associated with an operation of (++)) E O, where O is a predefined candidate operation set. [Liu et al.] (2019b) proposed a continuous relaxation of the search space by parameterizing the discrete operation choices in the DAG edges via a learnable vector a. This enables framing the NAS problem as a bi-level optimization one, with differentiable objectives w.r.t. all variables:
argmin L (un" ()) s.t. w² un( = argmin gtrain (ur ox);
where Ctrain and Loal are the empirical losses on the training and validation data, respectively, w the supernetwork parameters, a E A are the continuous architectural parameters, and wn (a)
A + ddd is a best response function that maps architectures to their optimal weights.
Comparison to single-objective constrained NAS. Early NAS methods predominantly targeted high accuracy, whereas contemporary hardware-aware differentiable NAS approaches (Cai al. 2018 Fu
can he replaced with any other parameter here, also architectural ones (see Section
3

---

Published as a conference paper at ICLR 2025
et al. 2020 Jiang al. 2021 Wan et al. 2020 Wang al. 202 Wu et al. 2019 2021 Xu et al. 2020bb are designed to identify architectures optimized for target hardware efficiency. Typically, these methods integrate hardware constraints within their objectives, yielding single optimal solution and necessitating multiple search iterations to construct the Pareto front. Our proposed algorithm addresses this by profiling the entire Pareto front in single search iteration. While single-ohjective constrained optimization is advantageous in scenarios demanding optimization of one objective under specific constraint, practical applications often require a suite of models adaptable varying user preferences even on single device. Our efficient Pareto-fron approximation algorithm provides such a suite of optimal models to choose from.
3 HARDWARE-AWARE MULTI-OBJECTIVE DIFFERENTIABLE NEURAL ARCHITECTURE SEARCH
We first formalize the multi-objective bi-level optimization NAS problem across multiple hardware devices, and then introduce scalable and differentiable method that combines MGD with linear scalarizations to efficiently solve this problem.
3.1 PROBLEM DEFINITION & SKETCH OF SOLUTION APPROACH
In multi-objective NAS, the bi-level problem described in Equation I becomes more difficult, since not only concerned with finding w" given fixed architecture, but want optimize in the space of architectures A as well. Assuming we have hardware devices (target functions) and objectives (e.g. accuracy, latency, etc.), similar to (3), for every / E (1 (1...7), the Pareto can be obtained by solving the following bi-level optimization problem:
palid argmin L (wo" (w0 (x).
s.l. w w"(a) argmin L, frain (w, (wa a),
(4)
w
where the M-dimensioonll loss vector L, (a), ax) A evaluated Vt E (1...., T). L, and L Lpali are the vectors with all M loss functions evaluated on the train and validation splits of D, used in the lower- and upper-level problems of 4 respectively.
L.M (w (a), a)) is
Our goal is to find Pareto-optimal architectures for each target device, covering diverse and representative preferences for different objectives. However, naively solving (4) for each device requires T independent bi-level searches, making this very inefficient for large models. To overcome this, we incorporate a single hypernetwork within the one-shot model (supernetwork) commonly used in conventional NAS (Bender et al. 2018 Liu et al. 2209bb Pham et al. 2018). This allows us generate architectures based on device embeddings and preference vectors in just one search run, reducing the search cost from (T) O(1).
hpni
hpng
dx
3.2 ALGORITHM DESIGN AND COMPONENTS
Figure 2: Architecture overview of
the MetaHypernetwo rk, which gets as input device embedding d, (in- put to an embedding layer E) and scalarization r (input to K hyper- networks) and yields an architecture encoding ā.
Our search procedure is composed of four core modular com- ponents (see Figure M (1) MetaHypernetwork that gen- erates the architectural distribution; (2) an Architect that samples discrete architectures from this distribution; (3) Superne repernetwort rk that exploits the weight-sharing paradigm for search efficiency and serves as a proxy for the network accu- racy; and (4) MetaPredictor that predicts hardware metries and enables gradient propagation back to the MetaaHypernetwork We now discuss each of these in detail.
MetaHlypernetwork. In order to generate architectures across multiple devices, inspired by Wang et al. (2022) and [in et al. (2020), we propose MetaHlyperettwort rk that can meta-learn across different hardware devices (see Figure 2). Hypernetworks are class of neural networks that generate the parameters of another model. They were initially proposed for model compression (Ha et al. 2017) and were later adopted for NAS (Brock et al. 2018) and MOO (Lin et al. 2020; Navon et al. 2021) Here, given preference vector (ri,..., . FMs) and hardware device feature vector d, for
4

---

Published as a conference paper at ICLR 2025
device & {1,.... T) the mettatlye (r, de), parameterized by Φ to generate an un mommooallieed architecture distribution Oep that is later used to compute the upper-level updates in (4). Similar to Lee et al. (2021b), de is fixed-size feature vector that is obtained by evaluating a fixed set of reference architectures on device f. The Metallypernetwork is composed of 2 main components (see F)
1. A bank K independent hypernetworks: hpn k. that parse the preference vector r and generate the architectural parameters .1,.... åg, respectively.
linear layer E, that learns a similarity map from device feature vectors to the bank of hpns. takes as imput the device feature vector d, and outputs an attention vector of size K.
The final output, Ökg of the MetaHype cecyppernetwet tnow is computed a weighted sum of the outputs of the K hypernetworks, where the vector of weights is the output of the linear layer E. For more detailed description of the MetaHype roe we refer the reader Appendix D.2
In all experiments, we initialize the Metaatypernetwork all architectural parameters for all scalarizations and device embeddings. By using the preference vector r to create linear scalarization of L, and the GaaHypernetwo rk to model the architectural distribution across T devices, the bi-level problem in reduces to:
yield a uniform probability mass
argmin Er-.s ['T Ly (wo" (a.). s.l. w (as) = arrmmin Ltrain (w, aw)] (5)
LL
where Cup the normalized architectural parameters obtained from the Architect A(ā.)
and rTL,(;,a)) Em= (,, is the scalarized loss for device t. Conditioning the MetaHypernetwork on the hardware embeddings allows us to generate architectures on new test devices without extra finetuning or meta-learning steps. Iniutively, the MetaHype wetttyyppneewworr learns to map the new test device, to the most similar device, in its learnt bank of embeddings (see also Figure [2]in the appendix). We use the Dirichlet distribution Dir(3). = (B1, (B1...., 3MM) to sample the preference vectors and approximate the expectation over the scalarizations using Monte Carlo sampling. In our experiments, we set 81 = = Ba = 1, for a uniform sampling over the (M - 1)-imple,, however, one parameter (Chen et al. 20211b)
these differently based user priors make it learnable
MetaPredictor. For the cheap-to- evaluate hardware objectives, such as latency, energy consumption, we employ regression model pl (ax, drr) that predicts the target labels yi" for objective and device f. given an architecture and device embedding d:", We the same predictors as Lee et al. (2021b) and optimize the MSE loss: ming E Aa-A,, y
Algorithm 1: MODNAS
Data:
D Deraini è D oatiai Supernetwork: device features (de
retaaHppereetwrrk Hę; of objectives M; Architect tect A; learning rates &t, E2.
while not converged do for E (I.....77 T) do
Sample scalarization r Dir(3) Set arch params ue +- Hr dz) Sample ag A(āxp) from Architect
9bb >M Im (De W. agg) .......)))))) Alg
pf (a, dr))² as done in Lee et al. (2021a) for meta-learning pertor- mance metrics across datasets. In our experiments, we pretrain separate MetaPredictor for every hardware objective m (e.g- latency, energy, etc.) on subset of (a,y") pairs, and use its predicted value directly in as (.. = p (a d."). Since the MetaPredictor is in principle a small neural network this pretraining step inexpensive. During search, we freeze and do not further update the MetaPredictor parameters 0.
7 FrankMolfeSol
g < Σt
7t S4
+ 1
// update Metatlypernetwork
for (I.....
.) do
Sample scalarization r ~ Dir(3) Set arch params āe +- Ho de) Sample a& A(āā)) from Architect
9'u <- ΣM I'm Vw th (Deraini w. a..)
14 is
Pe + T 1 Li
IT
un 4- w E2 Fi
// update Supernetwork
return Ha
Supernetwork. For expensive objectives like neural network classification accuracy, we use a Supernetwork rk that encodes the architecture space and shares parameters between architectures,
5

---

Published as a conference paper at ICLR 2025
providing best response function w" could estimate this function, such as performance predictors (Lee et al. expensive prior step of creating the training dataset for the predictor, by training architectures from scratch. To reduce memory costs of Supernetwor differentiable architecture sampling (Cai et al. 2018 [Dong & Yang 20T9 Xie et al. (019), activating only one architecture per step, and (2) entangle operation choice parameters in the Superne further increasing memory efficiency of the supernetwork beyond weight sharing (Sukthanker et 2023).
(a.p) w" for the scalarized loss in (5) While any parametric model
2021a). this requires an
training, we: (1) use a one-hot encoding of oxe for
Architect. The Architect A(ā) samples discrete architectural configurations from the un- normalized distribution 0i. H& (r, de) and enables gradient estimation through discrete variables for V a..). Methods such as GDAS (Dong & Yang 2019) utilize the Straight-Through Gumbel- Softmax (STGS) estimator (Jang al. 2017), that integrates the Gumbel reparameterization trick to approximate the gradient. Here we employ the recently proposed ReinMax estimator (Liu et al. 2023). that yields second-order accuracy without the need to compute second-order derivatives. See Appendix A. for more details on these discrete samplers. Similar to the findings in [iu et al. (2023). in our initial experiments, ReinMax outperforms the GDAS' STGS estimator (see Figure the Appendix), therefore, use ReinMax in all experiments that follow.
3.3 OPTIMIZING THE MetaHypee rnetwork VIA MGD
We denote the gradient of the scalarized loss in (5) with respect to the MetaHlpperettookk parameters Φ shared across all devices t E I,...,T, as: L r VT as)
Σm= I' (,,T",, where ogg. is the discrete architectural sample from the Architect A(ā4). Multiple Gradient Descent (MGD) (Désidéri 2012, Sener& Koltun 2018) provides a plausible approach to estimate the update directions for ev- ery task simultaneously by maximizing (2). Via the Lagrangian duality, the optimal solution to equation is 96 x Et=1 "h sh. where {gg Fz=1 is the solution of the following minimization problem:
Prosentta phall somtung Aitirer
41L4
32
E nis
T Σ T 2m1
min Th....
-x=1,m
1, 2 0.
The solution to this problem is either or, given small step size ç, descent direction that mono- tonically decreases all objectives at the same time and terminates when it finds Pareto sta- tionary point, i.c. dh 0, vt E {l....,?y, T". When T 2, the problem above simplifies to min..e(o.1) !l + (1 -1--0 7)9 112- which a quadratic function of yy with closed form solu- tion:
for Sae D0 Ohn
5224
MODNAS
CMA ELMMOO
M5-ASHA
RHPN
LS
1560
Goeal opt.
Figure 3: Hypervolume (HV) of MODNAS and baselines across 19 devices on NAS-Bench-200 For every device, we optimize for 2 objectives, namely latency (ms) and test accuracy on CIFAR- 10. For each method, metric and device we report the mean of 3 independent search runs. Higher area in the radar plot indicates better HV. Test devices are colored red around the plot.
(
9 T 22
max
min
11.b
38 112
When T 2, we utilize the Frank- Wolfe
solver (Jaggi 2013) as in /eener & Koltun (2018).
where the analytical solution in for 7 - 2 is used inside the line search. We provide the full algorithm to compute 7 in Algorithm 4 in Appendix A.3
In Algorithm and Figure we provide the pseudocode and an illustration the overall search phase of MODNAS. For every mini-batch sample from Dratid, we iterate over the device features de (line 2). sample one scalarization r and condition the MetaHypernen twork on both r and d, to generate the mn-rrmaalized architectural distribution 2œ (lines 3-4). We then compute the device-specifie gradient in line 6 which is used to estimate the y coefficients (line 7) used from MGD to update (lines 8-9). Similarly to Liu et al. (2099)), we use the first-order approximation to obtain the best

---

Published as a conference paper at ICLR 2025
response function in the lower level (lines 10-14) and repeat the same procedure for the upper-level (lines 2-6), except now the Supe rnewwork weights w are updated with the mean gradient (line 15), over devices.
EXPERIMENTS
In this section, we firstly demonstrate the scalability and generalizability of our MODNAS approach on a NAS tabular benchmark (Section 4.1]. Then, we validate MODNAS on larger search spaces for
Machine Translation (Section 4.2 Image Classification and Language Modeling (Section 4.3
Search Spaces and Datasets. We evaluate MODNAS on 4 search
trated NosttEN upprodisates Purts by generizag actixenure gindz It prefernce soclos
2) devics tepe
spaces: (1) NAS-Bench-200 (Dong
Yangj 2020 Li al. [021 with 19 devices and CIFAR-10 dataset;
MetaHPN
(2) MobileNetV3 from Once-for-All
Test Device:
(OFA) (Cai et al.]
2020) with 12
TPGA
Lacenny
devices and ImageNet-Ik dataset;
Figure 4: Illustration of MODNAS inference.
(3) Hardware-Aware-TTe waw--arar---rteee
(HAT) (Wang et al. 2020b) on the machine translation benchmark WMT' 14 En-De across 3 different hardware devices; (4) HW-GPT-Bench (Sukthanker et al.] 2024 GPT-2 based search space used
for language modeling on the OpenWebText Text (Gokaslan & Cohen 2019) across 8 devices. We refer to
Appendices E and for more details on these search spaces.
Evaluation. At test time, in order to profile the Pareto front with MODNAS unseen devices, sample 24 equidistant preference vectors r from the M --dimensional probability simplex and pass them through the pretrained MetaHype rnetwork (r, de) to get 24 architectures. Here the test device feature de is obtained similarly as for the train devices. See Figure or an illustration.
Baselines. We compare MODNAS against several base- lines such as Random Scarch (RS), Local Search (LS) and various Evolutionary Strategy and Bayesian Optimization MOO methods. Please refer to Appendix B for more comprehensive description of each of them. Furthermore, we also evaluate the Me teHHpernetwortt rnetwork with randomly initialized weights (RHPN).
titanrrtx
aroo
8.0-
ND coms
Metrics. To assess the quality of the Pareto set solu- tions, use the hypervolume (HV) indicator, which a standard metric in MOO. Given a reference point p (pt,.... "0",,---,""" p²*] R& that is an upper bound for all ob- jectives (f"'(: w. M
20
evaluations
i.e. supo, (= w. a) S pT A' E [M], and Pareto set Pa C A. HV(Pa) measures the region of non nom-nnminteed points bounded above from pt
Figure 5: HV number of evalu- ated architectures on NAS-Bench-200 of MODNAS and the blackbox MOO baselines on test device. For MOD- NAS we only do 24 full evaluations.
M Xlge R 3a Pa:ge
m=1
where X(-) is the Euclidean volume. HV can be interpreted as the total volume of the union of the boxes created by the Pareto front.
4.1 SIMULTANEOUS PARETO SET LEARNING ACROSS 19 DEVICES AND ABLATIONS
We firstly validate the scalability and learning capability of MODNAS by evaluating on the NAS- Bench-200 (Dong& & Yang 2020) cell-based convolutional space. Here want to optimize both latency and classification accuracy on all devices. We utilize the same set of 19 heterogeneous devices as Lee et al. (2021b), from which we use the one from HELP, namely a graph convolutional network (GCN), which we pretrain for GPU hours on the ground truth latencies on the 13 search devices described in Section We run
13 for search and at test time. For the lateney predictor,
For baselines, we SyneTTune Salinas al. 2022 https://ggthuub.com//assllaa
7

---

Published as a conference paper at ICLR 2025
FIGA
gold 6226
67
IO BE
MODIASIEAA HE
MODNAS F2
SAE
8.C
0.2
Figure 8: HV (left) and Pareto front (right) of MODNAS and baselines on FPGA with 3 nor- malized objectives: error, latency and energy usage. HV is computed using the (1, 1, 1) refer- ence point on the right 3D plot. See Fig. 16 results on Eyeriss.
Figure Pareto front on Eyeriss (left) and HV across devices (right) of MODNAS ran with various latency constraints on TAS-Bench-201. See Fig. in Ap- pendix H for all results.
the MODNAS search (see Appendix D for more details on the search hyperparameters), as described in Algorithm [ for 100 epochs (22 GPU hours on a single NVidia RTX2080Ti) and show the HV in Figure 3 of the evaluated Pareto front in comparison to the baselines, for which we allocate the same search time budget across all devices equivalent to the MODNAS search + evaluation.
Most notably, MODNAS consistently outperforms other baselines across every device. For the baselines, we con- duct 19 separate search runs (one for each device), whereas MODNAS leverages meta-learning generate the Pareto set on each device using the same MetaHypernetwer single search run. Interestingly, the trained MODNAS attention-basse MetaHyyernettor forms the RHPN baseline in profiling the Pareto front, demonstrating its effectiveness in optimizing across multi- ple devices and conflicting objectives simultaneously. In Figure 20a in the Appendix, we compare MODNAS with additional baselines, running them at double the budget used for the experiments in Figure 3 Figure 5 (see Fig-
fpge
0.95
0.90
0.85
Hycirvolume
in
0.75
significantly outper-
0.70
MC Sampling
0.65
Sequential
Mean
0.60
60
80
100
Search Epochs
Figure 6: HV over search epochs of dif- ferent gradient schemes in MODNAS.
ure 2 in the appendix for all devices) shows that most baselines require more than twice the number of architecture evaluations to reach the same HV as MODNAS. Results show that MODNAS remains the top performer across hardware devices on average. Furthermore, in the appendix, Figure 20 presents radar plots for four additional metries, and Figure 18 and 17 results on NB201 when optimizing CIFAR- 100 accuracy and device latency.
Reliably learnt embeddings for hardware devices. To demonstrate the effectiveness of our Metaalyprreetorkk in learning hardware device similarities, Figure 12 in the appendix shows K-means clustering of original and MetaHypernetwe
embeddings, reduced via t-SNE. The Me Metattypernetwork successfully clusters similar devices, confirming its efficacy.
Me MetaHlyerrettork update schemes: robustness of MGD, We compare the MGD update scheme for the MetaaHypernetwork + (line 9 in Alg. D against (1) the mean gradient over tasks: <- Φ $ Σ A (2) sequential updates with all single tasks' gradients: Tb < T - Egh. Vt; (3) single updates using gradients of MC samples over tasks: & + 1 - Egg. t t~ {1.... T). Figure (see Figure 24 in Appendix(H]oor more results) shows the HV over search epochs for these schemes. MGD, by accounting for inter-task dependencies, achieves higher final HV, better anytime performance, and faster convergence than the other schemes.
Scalability to three objectives. We show the scalability of MODNAS to 3 objectives, namely, accuracy, latency and energy consumption. For this experiment we use the FPGA and Eyeriss tabular energy usage values from TW-NAS-Bende (Li et al. 2021). In addition the MetaPredic sttr for latency, we pretrain a second predictor on the energy usage objective. We then run MODNAS and the MOO baselines with the same exact settings as for 2 objectives. Results shown in Figure indicate that MODNAS can scale to M > 2 without additional search costs or hyperparameter tuning and yet achieves HV close the global optimum front of the WAS-Bench-201 space.
MODNAS vs. constrained single-objective optimization. To compare against single-objective NAS with hardware constraints in the objective, we run MetaD22+HELL (Lee et al. (0216). Since MetaD2A HELP P is not able to profile the Pareto front directly, the NAS search 24 times with
8

---

Published as a conference paper at ICLR 2025
different constraints, which compute by denormalizing the same 24 equidistant preference vectors we use to evaluate MODNAS. We also extend MODNAS to incorporate user prior constraints over the multiple objectives being optimized during search. Namely, we add normalized constraint cmL. such that if the predicted value from the MetaPredi coor during search satisfies this constraint, i.e. p8 (ass, d!") I cmT, remove the gradient w.r.t. to that objective in lines 6 and 14 of Algorithm [ In Figure 7 (other devices in Figure 21) we can see that when increasing the latency constraint to (only cross-entropy optimized), though the HV decreases, MODNAS returns Pareto sets with more performant architectures. MetaD2A+HELL despite multiple search runs, prioritizes performance over diversity, resulting in less varied solutions.
4.2 PARETO FRONT PROFILING ON TRANSFORMER SPACE
To demonstrate its effectiveness beyond im- age classification and CNN spaces, we ap- ply MODNAS to the hawdware-aware Trans-
RuspbaryPi-CC PI-CUU
RaspberryPi-cPU
5UMA
6.064
5554-1
former (HAT) search space from Wang
$2400
4045
ct al.
2020bb the WMT 14 En- De (Jean
et al.
2015: Macháček & Bojar 2014) ma-
chine translation task. We pretrain the
MetaPredictor (details in Appendix D.I> for 5 GPU hours on 2000 architecture sam- ples from the search space and then con- duct the search for 110 epochs (6 days on NVIDIA RTX A6000 GPUs) using 2 search devices, adhering to the same hyperparameters as Wang et al. (2020b) to optimize for latency and validation cross entropy loss. We allocate to each baseline 2.5x more runtime budget than MODNAS, resulting in 1300 (RS-BO) to 6000 (MO-ASHA) total architecture evaluations, whereas MODNAS evaluates only 24 generated architectures. Details the HAT search space and search hyperpa- rameters are in Appendix E We evaluate MODNAS on all devices (2 search and 1 test) using the BLEU score, and results in Figure/yshow that MODNAS outperforms all baselines, achieving a higher hypervolume (left plot) of the generated Pareto fronts (right plot). For HAT, we evaluate the architectures provided in their paper. Additional results other training devices and evaluation metrics are presented in Figures 28 29 and 30 in the Appendix.
6.008
2000 4300
5200 5006 7003 6000 L 1912
HV and Pareto fronts of MODNAS and baselines across devices on the HAT space.
Figure 9:
4.3 EFFICIENT DIFFERENTIABLE MOO STARTING FROM PRETRAINED SUPERNETWORKS
Image Classification on ImageNet- 1k. We now evaluate MODNAS on ImageNet-I Ik using the Mev3 search space from Once-for-
Average HV (12 devices) for OFA
All (OFA) (Cai et al., 2020). For this experiment, we run MODNAS using 11 search (and test) devices starting with the pretrained OFA
supernetwork and run the search further for I day on 8 RTX2080TI
GPUs. During the search, we only update the MetaHype network weights and keep the pretrained Supernetwor tails on the search space and hyperparameters and D33 We use the simple MLP from Lee et al. (2021b) as our MetaPredictor, pretraining it for 6 hours 5000 sampled architecture-latency pairs. To evaluate the 24 points generated by our Metaa twork and baselines, we use the OFA pretrained Superne tooorneework two rk. Results in Figure show that MODNAS achieves a higher average HV across all devices compared to baselines, which we run for 192 hours using the OFA pretrained accuracy predictor (see Figure 34 for all results and Figure 33 for the Pareto fronts).
weights frozen. De-
in Appendices E
0.8
Figure 10: Average HV of MODNAS and baselines across 12 devices on OFA space. For every device we op- timize for 2 objectives, namely latency (ms) and test accuracy on ImageNet-Ik.
Comparison to Zero-Cost Proxies. We also compare the HV of the Pareto front obtained by MODNAS to that produced by NSGA- II (Deb et al. 22022), which uses a zero-cost proxy (ZCP) (Abdeliat) tah et al.] 2021) (we chose Zico (Li et al., 2023)) for performance estimation instead of the actual accuracy. Table in the appendix presents the results of this experiment on two devices. As shown, despite its improved runtime efficiency, the ZCP- ACP-guided search underperforms compared to both the existing baselines and MODNAS, which optimize for accuracy directly.
9

---

Published as a conference paper at ICLR 2025
Language Modeling with GPT-2. With the rapid growth of language model sizes, it is crucial to identify transformer variants that efficient during inference (latency) while maintaining competitive performance. We apply MODNAS to the GPT-S space from HW-GPT-Bench (Sukthanker et al. 2024), which features a non-convex Pareto front between perplexity and hard- ware metric objectives. Using pretrained Superne two rk weights from SW-GGPT-Bench we conduct a single 6-hour search on 4 Nvidia A100 GPUs, optimizing for energy consumption (Wh)
MOREA
L5
N5GA2
Pempesky
RS80
MOASHA
and perplexity across 8 different GPU devices, See Appendix [ for details on the MetaHlyerree rnetwort architecture and search hyperparameters. The Supernetwork weights are kept frozen while updating the MetaHlyperetter with the same time budget, MODNAS matches surpasses other MOO baselines, demonstrating its effectiveness in optimizing beyond convex Pareto fronts.
0.
0.3
0.4 0.5
Energy lWhl
Figure 11: Pareto front of MOD-
Figure [T shows that,
NAS and baselines on the HW- GPT-Bench, A100 GPU.
4.4 COMPUTATIONAL COMPLEXITY
Ignoring the cost to train final architectures in the Pareto set, methods like MetaD2A + HELP (Lee et al.) 2021a by have worst- OCTT) to build the Pareto set, where T is the number of devices and C is the number of constraints. MODNAS reduces this O(1) by con- ditioning a single metatHypereetworr rk on both device types and constraints. Methods like LEMONADE (Elsken et al.) 2019a) and ProxylessNAS (Cai et al., 2018) apply constraints during the search phase, requiring an independent search per device. Black-box methods such as LEMONADE, NSGA-II (Deb et al. 2002). or qEHVI (Daulton et al.) (0200) train O(NT) architectures or a surrogate based on O(N) architectures in the case of MetaD2A HELP. In contrast, MODNAS and OFA have a cost of O(1) as they train single supernetwork. Although MODNAS iterates over T shows that MODNAS generalizes well on 17 test devices with only 2 search devices due to its meta-learning capabilities. See Tables and in the Appendix for more details.
Table Cost of MODNAS and other methods. N is the number of trained architectures during search, T the number of devices and C the number of constraints.
case time complexity
Mathod
Build
LEMONADE [Eisken et al 20150)
O(T)
Hizckbox sutoromoorr
2022
O(t)
ProsslesNAAS Cupse
Mend2ZA + Hib
ONN
(XCT)
ORA Gonvral 20dy
(NCT)
MOLDAS TO225)
O11)
devices to compute 9 and Iw. Figure 25 in Appendix H.1
5 CONCLUSIONS BROADER IMPACT AND LIMITATIONS
In this paper, we propose a novel hardware-award differentiable NAS algorithm for profiling the Pareto front in multi-objective problems. In contrast to constraint-based NAS methods, ours can generate Pareto optimal architectures across multiple devices with single hypernetwork that conditioned on preference vectors encoding the trade-off between objectives. Experiments across various hardware devices (up to 19), objectives (accuracy, latency and energy usage), search spaces (CNNs and Transformers), and applications (classification, machine translation, language modeling) demonstrate the effectiveness and efficiency of our method.
Broader Impact. In an era of large-scale models (e.g. foundation models), speeding up the search and training cost for inference-optimal neural architectures is important aspect of responsible research (Cai et al. 2024; Muralidharan et al.) 2024. Zhang et al.) 2024). The main goal of this work is to improve the search costs, as well as the efficiency of the found architectures in terms of various hardware metrics, therefore reducing the energy consumption and CO2 footprint. The energy savings of these architectures will be amplified they might be deployed on large number of devices.
Limitations. While our differentiable multi-objective search method shows promising results, there are potential limitations. MODNAS inherits challenges common to gradient- the risk of failure without proper tuning regularization (Zela et al. 2020). For example, gradients may favor one objective, leading to local optima that hinder exploration of the full Pareto front. Additionally, the method relies differentiable proxies for objectives, which may not always align with ground truth values.
search, such as
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was partially supported by the following sources: TAILOR, project funded by EU Horizon 2020 research and innovation programme under GA No 952215; the Deutsche Forschungs- gemeinschaft (DFG, German Research Foundation) under grant number 417962828; the European Research Council (ERC) Consolidator Grant "Deep Learning 2.0" (grant no. 101045765). Robert Bosch GmbH is acknowledged for financial support. The authors acknowledge support from EL- LIS and ELIZA. The authors gratefully acknowledge the Gauss Center for Supercomputing eV www.ggass-coer cauus--entte.eu for funding this project by providing computing time on the GCS supercom- puter JU JUWLSS at Jülich Supercomputing Center (JSC). Funded by the European Union. Views and opinions expressed however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC. Neither the European Union nor the ERC can be held responsible for them.
Funded by the European Union
REFERENCES
M. Abdelfattah, A. Mehrotra, L.. Dudziak, and N. Lane. Zero-cost proxies for lightweight NAS. In Proceedings of the International Conference on Learning Representations (ICLR'21), 2021. Published online: iccr.cc g
Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optine model design. Thirty-sevetth Conference on Neural Information Processing Systems, 2023.
Stephan Patrick Baller, Anshul Jindal, Mohak Chadha, and Michael Gerndt. Deepedgebench: Benchmarking deep neural networks on edge devices. 2021 IEEE International Conference on Cloud Engineering (IC2E), pp. 20-30, 2021.
G. Bender, P-J. Kindermans, B. Zoph, V. Vasudevan, and Q. Le. Understanding and simplifying one-shot architecture search. In Proceedings of the 35th International Conference on Machine Learning (ICML'18) volume 80. Proceedings of Machine Learning Research, 2018
Hadjer Benmeziane, Kaoutar El Maghraoui, Hamza Ouarnoughi, Smail Niar, Martin Wistuba, and Naigang Wang. Hardware-aware neural architecture search: Survey and taxonomy. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 4329,, 2021. Survey Track. 444
J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13:281-305, 2012. 21
Andrew Brock, Theo Lim, J.M. Ritchie, and Nick Weston. SMASH:
One-shot model architecture search through hypernetworks. In International Conference on Learning Representations, 2018.
H. Cai, C. Gan, T. Wang, 7. Zhang, and S. Han. Once-for-All: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations (ICIR), 2020.
10 21
27
28
43
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In International Conference on Learning Representattons, 2018.
Ruisi Cai, Saurav Muralidharan, Greg Heinrich, Hongxu Yin, Zhangyang Wang, Jan Kautz, and Pavlo Molchanov. Flextron: Many-in-one flexible large language model. arXiv preprint
arXiv:2406.1022 2024. 10
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEFE/CVF international conference on computer vision, pp. 12270-12280. 2021a. 1
Xiangning Chen and Cho-Jui Hsieh. Stabilizing differentiable architecture search via perturbation- based regularization. In International conference on machine learning, pp. 1555-1165.. PMLR, 2020. 43
11

---

Published as a conference paper at ICLR 2025
Xiangning Chen, Ruochen Wang. Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. DrNAS: Dirichlet neural architecture search. In International Conference on Learning Representations. 2021b. 3
Richeek Das and Samuel Dooley. Fairer and more accurate tabular models through nas. Algorithmic Fairness through the Lens of Time Workshop at NeurIPS, 2023.
S. Daulton, M. Balandat, and E. Bakshy, Differentiable expected hypervolume improvement for parallel Multi-Objective Bayesian optimization. In Advances in Neural Information Processing Systems, volume 33, pp. 9851-9864 Curran Associates, Inc., 2020. TO 21
Samuel Daulton, David Eriksson, Maximilian Balandat, and Eytan Bakshy. Multi-objective bayesian
optimization over high-dimensional search spaces. In Uncertainty in Artificial Intelligence, pp. 507-517. PMLR, 2022. 43
Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and Tanaka Meyarivan, A fast elitist non-dominated sorting genetic algorithm for multi objeii-oeeective optimization: Nsga-ii. Parallel Problem Solving from Nature PPSN VI: 6th International Conference Paris, France, September 18-20. 2000 Proceedingg 6, pp. 849-858. Springer, 2000. 43
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist multiob- jective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computation, 6(2):182-197, 2002. 10
Antoieet Désidéri. Muultipeegrraiies descent algorithm (mgda) for multiobjective optimization.
Comptes Rendus Mathematique, 350:313-318, 2012.
43
X. Dong and Y. Yang. Searching for a robust neural architecture in four gpu hours. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 3
28
30 43
X. Dong and Y. Yang. NAS-Bemch-201: Extending the scope of reproducible neural architecture search. In Proceedings of the International Conference Learning Representations (ICLR 20), 2020. Published online: ccclr.cc 7 27
Samuel Dooley, Rhea Sanjay Sukthanker, John Dickerson, Colin White, Frank Hutter, and Micah Goldblum. Rethinking bias mitigation: Fairer architectures make for fairer face recognition. In Thirty-seventh Conference on Neural Information Processing Systems, 2023, [44
Lukasz Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas Lane. Brp-nas: Prediction-bee based nas using gens. Advances in Neural Inforation Processing Systems,
13:10480-00490, 2020. 22 44
T. Elsken, J. Metzen, and F. Hutter. Efficient multi-objective neural architecture search via lamarekian evolution. In International Conference on Learning Representations, 2019a. [0
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. In International Conference Learning Representationss 2019b.
Yonggan Fu, Wuyang Chen, Haotao Wang, Haoran Li, Yingyan Lin, and Zhangyang Wang. AutoGAN-distilles Searching to compress generative adversarial networks. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine
Learning Research, pp. 3292-3303. PMLR, 13-18 Jul 2020. 44
Aaron Gokaslan and Vanya Cohen. OpenMebTexxtCrpus 2019. 27
Openwebtext corpus.
http://SkyLion6077ggtthbb io/
Nyoman Gunantara. review of multi-objective optimization: Methods and its applications. Cogent
Engineering, 5(1): 2018. 43
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Computer Ession-ECCy 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XVI 16, pp. Springer, 2020.
44
12

---

Published as a conference paper at ICLR 2025
D. Ha, A. Dai, and Q. Le. Hypemetworks. In Proceedings of the International Conference Learning Representations (ICLR' 17), 2017.
Chaoyang He, Haishan Ye, Li Shen, and Tong Zhang. Milenas: Efficient neural architecture search via mixed-level reformulation. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11990-11999, 2020. 30 44
Long P Hoang, Dung D Le, Tran Anh Tuan, and Tran Ngoc Thang. Improving pareto front learning via multmmulimaaaa sample hypernetworks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 7875-7883, 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems, volume 35, pp. 30016-30030. Curran Associates, Inc., 2022.
Chi-Hung Hsu, Shu-Huan Chang, Jhao-Hong Liang. Hsin-Ping Chou, Chun-Hao Liu, Shih-Chieh Chang, Jia-Yo Yu Pan, Yu-Ting Chen, Wei Wei, and Da-Cheng Juan. Monas: Multi-objective neural architecture search using reinforcement learning, arXiv preprint arXiv:1806. 10332, 2018. 44
C. Igel, Nikolaus Hansen, and Stefan Roth. Covariance matrix adaptation for multi-objectivv optimization. Evolutionary Computation, 15:1-28, 2007. 21
Rafael C Ito and Fernando Von Zuben. Ofa 2: A multi-objective perspective for the once-for-all
neural architecture search. arXiv preprint arXiv:2303./3683, 2023. 44
Martin Jaggi. Revisiting frafk-wolfe: Projection-fiee sparse convex optimization. In International
Conference Machine Learning, 2013. 6 20
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax.
In Proceedings of the International Conference Learning Representations (ICLR'17), 2017 Published online: jicr. cc 19
Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. Montreal neural machine translation systems for wmt' 15. In Proceedings of the tenth workshop on statistical machine translation, pp. 134-144 140, 2015.
g
Qian Jiang, Xiaofan Zhang, Deming Chen, Minh N Do, and Raymond A Yeh. Eh-dnas: End-to-end hardware-aware differentiable neural architecture search. arXiv preprint arXiv:21 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint orXiv:2001.08361, 2020.
Sunghoon Kim, Hyunjeong Kwon, Eunji Kwon, Youngchang Choi, Tae-Hyun Oh, and Seokhyeong Kang. Mdarts: Multi-objective differentiable neural architecture search. In 2021 Design, Automa- tion & Test in Europe Conference & Exhibition (DATE), pp. 1344-1349. IEEE, 2021. 44
Ivan Lazarevich, Matteo Grimaldi, Ravi Kumar, Saptarshi Mitra, Shahrukh Khan, and Sudhakar Sah. Yolobench: Benchmarking efficient object detectors on embedded systems. 2023 IEEE/CVF International Conference Computer Vision Workshops (ICCVW), pp. 1161-1170 2023. 44
Hayeon Lee, Eunyoung Hyung, and Sung Ju Hwang. Rapid neural architecture search by learning to generate graphs from datasets. In International Conference on Learning Representations, 202 la.
10
Hayeon Lee, Sewoong Lee, Song Chong, and Sung Ju Hwang. addatie efficient latency prediction for nas via meta-learning. In Advances in Neural Information Processing Systems, volume 34, pp. 77016-27022 Curran Associates, Inc., 2021b.
22
28
41
13

---

Published as a conference paper at ICLR 2025
Jaeseong Lee, Duseok Kang, and Soonhoi Ha. S3nas: Fast npu-aware neural architecture search
methodology. arXiv preprint arXiv:2009.02009 2020. 44
Jayoung Lee, Pengcheng Wang. Ran Xu, Sarthak Jain, Venkat Dasari, Noah Weston, Yin Li, Saurabh Bagchi, and Somali Chaterji. Virtuoso: Energy- and latency-aware streamlining of streaming videos on wystems-on-chipe ACM Trans. Des. Autom. Electron. Syst., 28(3), April 2023. ISSN 1084-4309. 44
Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, and Yingyan (Celine) Lin. Hw-nas-bench Hardware-aware neural architecture search benchmark. In International Conference on Learning Representations, 2021.
7,828
Guihong Li, Yuedong Yang, Kartikeya Bhardwaj. and Radu Marculescu. Zico: Zero-shot NAS via inverse coefficient of variation on gradients. In The Eleventh International Conference on Learning Representations, 2023. 9
L. Li and A. Talwalkar. Random search and reproducibility for neural architecture search. In J. Peters and D. Sontag (eds.). Proceedings of The 36th Uncertainty in Artificial Intelligence Conference (UAI'20), pp. 367-377. PMLR, 2020.
21 44
Liam Li, Kevin G. Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. Massively parallel hyperparameter tuning. ArXiv, abs/1810.05944, 2018. 21
X. Lin, H. Zhen, Z. Li, Q. Zhang, and S. Kwong. Pareto multi-task learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Xi Lin, Zhiyuan Yang. Qingfu Zhang. and Sam Tak Wu Kwong. Controllable pareto multi-task learning. ArXiv, abs/2010.06313, 2020.
Chenxi Liu, Liang-Chios Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Loddon Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. 2019 IEEE/VVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 82 92, 2019a. 44
H. Liu, K. Simonyan, and Y. Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019b.
Liyuan Liu, Chengyu Dong, Xiaodong Liu, Bin Yu, and Jianfeng Gao. Bridging discrete and backpropagation: straight-through and beyond. Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
19 43
Suyun Liu and Luis Nunes Vicente. The stochastic multi-gradient algorithm for multi-objectivv optimization and its application supervised machine learning. Annals of Operations Research, pp. 1572-9338, 2021.
43
Zhichao Lu, Kalyanmoy Deb. Erik Goodman, Wolfgang Banzhaf, and Vishnu Naresh Boddeti. Nsganetv2: Evolutionary multi-objective surrogate-assisted neural architecture search. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pp. 35-51. Springer, 2020. 43
Matouš Machášek and Ondřej Bojar. Results the WMT14 metrics shared task. In Proceedings of the Ninth Workshop Statistical Machine Translation, pp. 293-301, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. doi: 10.31 111///////1--333/ 27
D. Mahapatra and V. Rajan. Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization. In Proceedings of the 36th International Conference on Machine Learning (ICML") 20). pp. 6597-6607. Proceedings of Machine Learning Research, 2020.
Natalia Martinez, Martin Bertran, and Guillermo Sapiro. Minimax pareto fairness: multi objective perspective. In International Conference on Machine Learning, pp. 6755-6764. PMLR, 2020. 44
14

---

Published as a conference paper at ICLR 2025
Rohit Mohan, Thomas Elsken, Arber Zela, Jan Hendrik Metzen, Benedikt Staffler, Thomas Brox, Abhinav Valada, and Frank Hutter. Neural architecture search for dense prediction tasks in computer vision. Int. J. Comput. Viston, (1(():1784-1807)) April 2023. ISSN 0920-5691. 44
Michinari Momma, Chaosheng Dong, and Jia Liu. A multi-objeetive/mmllii- task learning framework induced by pareto stationarity. In International Conference Machine Learning, pp. 15895-15907.
PMLR, 2022. 43
Sajad Movahedi, Melika Adabinejad, Ayyoob Imani, Arezou Keshavarz, Mostafa Dehghani, Azadeh Shakery, and Babak N Araabi. X-darts: Mitigating performance collapse by harmonizing operation selection among cells. The Eleventh International Conference on Learning Representations, 2022.
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. arXiv preprint arXiv:2407. 14679, 2024.
10
Aviv Navon, Aviv Shamsian, Gal Chechik, and Ethan Fetaya. Learning the pareto front with hypemneworrss International Conference on Learning Representations, 2021.
43
Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. flexible framework for multi- objective bayesian optimization using random scalarizations. In Uncertainty in Artificial Intelli- gence, pp. 766-776. PMLR, 2020. 21
H. Pham, M. Guan, B. Zoph. Q. Le, and J. Dean. Efficient neural architecture search via parameter sharing. In International Conference Machine Learning. 2018.
Hoang Phan, Ngoc Tran, Trung Le, Toan Tran, Nhat Ho, and Dinh Phung. Stochastic multiple target sampling gradient descent. Advances in neural information processing systems, 35:22643-22663 2022.
43
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. In Proceedings of the Thirty-Thirl AAAI Conference on Artificial Intelligence, AAAI'19, 19. AAAI Press, 2019. ISBN 978-1-57735-809-11 21
Binxin Ru, Clare Lyle. Lisa Schut, Miroslav Fil, Mark van der Wilk, and Yarin Gal. Speedy perfor- mance estimation for neural architecture search. In Advances in Neural Information Processing Systems, 2021. 22
Michael Ruchte and Josif Grabocka. Scalable pareto front approximation for deep multi-objective learning. In 2021 IEEE international conference on data mining (ICDM), pp. 1306-13630 1311. IEEE, 2021.
25
David Salinas, Valerio Perrone, Olivier Cruchant, and C. Archambeau. A multi-objective perspective on jointly tuning hardware and hyperparameters. ArXiv, abs/2106.0568/ 2021. 21
David Salinas, Matthias Seeger, Aaron Klein, Valerio Perrone, Martin Wistuba, and Cedric Archam- beau. Syne tune: A library for large scale hyperparameter tuning and reproducible research. In International Conference on Automated Machine Learning, AutoML 2022, 2022. 7
Shreyas Saxena and Jakob Verbeek. Convolutional neural fabrics. Advances in neural information processing systems, 29, 2016. 44
Robin Schmucker, Michele Donini, Muhammad Bilal Zafar, David Salinas, and C. Archambeau.
Multi-objective asynchronous successive halving. ArXiv, abs/2106. 2021. 21
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objectivee optimization. In Neural 3 6
Information Processing Systems, 2018.
Albert Shaw, Daniel Hunter, Forrest Landola, and Sammy Sidhu. Squeezenas: Fast neural architecture
search for faster semantic segmentation. In Proceedings the the IEEE/CVF International Conference
Computer Vision Workshops, pp. 0-0, 2019. 44
15

---

Published as a conference paper at ICLR 2025
Samuel L Smith, Benoit Dherin, David Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. In International Conference on Learning Representations, 2021. 43
Rhea Sanjay Sukthanker, Arjun Krishnakumar, Mahmoud Safari, and Frank Hutter. Weight- entanglement meets gradient-base based neural architecture search. arXiv preprint arXiv:2312.10440,
2023. 43
Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler, Joerg K.H. Franke, and Frank Hutter. Hw-gpt-bench: Hardware-aware architecture benchmark for language models. arXiv preprint arXiv:240S. 10299, 2024.
41
M. Tan, B. Chen, R. Pang. V. Vasudevan, M. Sandler, A. Howard, and Q. Le. Mnasnet: Platform- aware neural architecture search for mobile. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Alvin Wan, Xiaoliang Dai, Peizhao Zhang. Zijian Hc, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12965-12974 2020. 4
Dilin Wang, Meng Li, Chengyue Gong, and Vikas Chandra. Attentivenas: Improving neural architecture search via attentive sampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6418-642 2021.
H. Wang, Z. Wu, Z. Liu, H. Cai, L. Zhu, C. Gan, and S. Han. Hat: Hardware-aware transformers for efficient natural language processing. arXiv:2005. 14187/cs.CL. 2020a. 27
Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. HAT: Hardware-awar transformers for efficient natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7675-7688, Online, July
2020b. Association for Computational Linguistics.
28
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang. Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 139 149, 2022. 23
Colin White, Mahmoud Safari, Rhea Sanjay Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, Debadeepta Dey, and Frank Hutter. Neural architecture search: Insights from 1000 papers. ArXiv, Abs/2300.087221 2023. [
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-awart efficient convnet design via differentiable neural architecture search. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Yan Wu, Zhiwu Huang, Suryansh Kumar, Rhea Sanjay Sukthanker, Radu Timofte, and Luc Van Gool. Trilevel neural architecture search for efficient single image super-resolution. arXiv preprint 2rXiv:2/01.0660 2021.
S. Xie, H. Zheng, C. Liu, and L. Lin. SNAS: stochastic neural architecture search. In International Conference on Learning Representations, 2019.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi. Qi Tian, and Hongkai Xiong. Pe-darts: Partial channel connections for memory-efficient architecture search. In International Conference on Learning Representations, 2020a. 3
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Bowen Shi, Qi Tian, and Hongkai Xiong. Latency-aware differentiable neural architecture search. arXiv preprint arXiv:2001.06392 2020b.
Xinmin Yang, Wei Yao, Haian Yin, Shangzhi Zeng. and Jin Zhang. Gradient- based algorithms for multi-objective bi-level optimization. Science China Mathematics, 2024.
43
16

---

Published as a conference paper at ICLR 2025
Feiyang Ye, Baijiong Lin, Xiaofeng Cao, Yu Zhang, and Ivor W. Tsang. A first-order multi maati-gradient algorithm for multi-objective bi-level optimization. In ECAL, volume 392 of Frontiers in Artificial Intelligence and Applications, pp. 2621-2628 IOS Press, 2024. 43
Mao Ye and Qiang Liu. Pareto navigation gradient descent: a first-order algorithm for optimization in pareto set. In Uncertainty in Artificial Intelligence, pp. 2246-2255. PMLR, 2022.
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. In International Conference on Learning Representations, 2020.
10 43
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
2022. [
Dingkun Zhang, Sijia Li, Chen Chen, Qingsong Xie, and Haonan Lu, Laptop-diff: Layer pruning and normalized distillation for compressing diffusion models. arXiv preprint arXiv:2404.11098,
2024. 10
Li I Lyna Zhang, Yuqing Yang, Yuhang Jiang, Wenwu Zhu, and Yunxin Liu, Fast hardware-aware neural architecture search. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 692-693, 2020.
Miao Zhang, Steven Su, Shirui Pan, Xiaojun Chang, Ehsan Abbasnejad, and Reza Haffari. idarts: Differentiable architecture search with stochastic implicit gradients. In International Conference on Machine Learning, 2021. 3
Qi Zhang, Peiyao Xiao, Shaofeng Zou, and Kaiyi Ji. MGDA converges under generalized smoothness, provably. In The Thirteenth International Conference on Learning Representations, 2025.
43
Yiyang Zhao, Linnan Wang, Kevin Yang, Tianjun Zhang, Tian Guo, and Yuandong Tian. Multi- objective optimization by learning space partition. In International Conference Learning
Representations, 2022.
10.2 21
B. Zoph and Q V. Le. Neural architecture search with reinforcement learning. In Proceedings of the International Conference Learning Representations (ICLR'I7) 17), 2017. 43
17

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
Algorithmic components
A.1 Discrete Samplers
A.2 Details the Architect Gradient Computation
19
A.3 Frank-Wolff Solver
20
B
Multi-objective NAS algorithms
21
Evaluation Details
21
Other Metries
21
MODNAS-So1L
22
D
Experimental Details
22
D
MetaPredictor Architectures
22
D.2
Metallypernetwork Architecture
................................
23
D.3 MODNAS Hyperparameter Contigurations
25
Normalization of objectives
25
E Details Search Spaces
27
F Datascts and Devices
27
G
Runtime Comparison
28
Additional Experiments
28
H.
Additional Results NAS NAS-Bench-2011
28
H.2
Additional Results Hardware-aware Transtormers (En-De)
.........
39
H.3
Additional Results the HW-GPT space
39
114
Additional Results on MobileNet33
41
Further Discussion on the Robustness of MODNAS
43
Extended Related Work
18

---

Published as a conference paper at ICLR 2025
A ALGORITHMIC COMPONENTS
In this section, provide the pseudocodes for some of the algorithmic components in MODNAS.
A.1 DISCRETE SAMPLERS
Given the architecture parameters ag from the Metaatypernetwork, obtain a differen- tiable discrete architecture sample from the Architect as a4 f π stop g(T) + oee, where O4 Cat Cat(softmax, (d+)) and
f- 2 softmax stop g [nn 0g softmax, (ā+)) k.) + (+.)
coftmax((au)
Here, Cat is the categorical distribution, T the temperature the tempered softmax softmax-(a): Σ ezp(as/T)* and stop__(() duplicates its input and detaches it from backprop-
exp(o4/7)
agation. Refer to the ReinMax paper (Liu al. 2023) for more details. The algorithm pseudocode on how a one-hot encoded (discrete) architecture is sampled given an unnormalized architectural
distribution à is given in Algorithm 2 and Algorithm Bl for the trrighh-Thhough (Jang al. 2017)
and ReinMax (Liu et al. 2023) gradient estimators, respectively.
Algorithm 2: Straight Through (Jang
Algorithm 3: ReinMax (Liu al. 2023
al. (017)
Data: â: softmax input, 7 temperature
Data: à: softmax input, 7: temperature
Result: a: one-hot samples
Result: a: one-hot samples
f softmax (ā)
To < softmax; (ā)
2 a Cat(TT)
(mo(
a+softnax, (a)
< softmax, (ā)
softmax; ((to)_g(
< 1 - stop (()p_ +
5 m 2T To
return
6 a' 72
stto__((m)) +
7 return a
A.2
DETAILS ON THE Architect GRADIENT COMPUTATION
In this section, provide additional details on how the Architect utilizes the Straight-Through Estimator (STE) to backpropagate through the sampling of discrete architectural parameters. Forward pass:
The detaHypernetwon parameterizes the unnormalized architectural distribution: ā H4. where & are the MetaaHypernetworf parameters.
2. is passed to Architect and it does the following steps:
(a) Normalizes ā and samples a one-hot (discrete) ac a ~ Cat(softmax(ū)).
(b) Sets the Supernetwor rk architectural parameters to the one-hot a, i.e. resulting in a single subnetwork by masking the rnetwork
(c) Passes & as input to MetaPredictor.
The Supernetwork and MetaPredicton do a forward pass using the training data (e.g., images) and hardware embedding, respectively.
4. Compute the scalarized loss function.
The main problem now is that we cannot directly backpropagate the gradient computation through the Architect to update the Metaltypernetwork parameters कb This is due to the sampling from the Categorical distribution in step 2/(a) above being non-differentiable. The STE approximates the gradient for the discrete architectural parameters by ignoring this actual non-differentiable sampling operation.
19

---

Published as a conference paper at ICLR 2025
Backward pass:
1. Calculate the gradient of the scalarized loss with respect the discrete architectural parameters 0C/0o.
Propagate this gradient back to & (Metatyperrnetwork parameters) via the probability distribu-
tion:
ac
dc
VoL da Osoftmax
V @ssoftaax H.)).
STE backpropagates "through" a proxy that treats the non-differentiable function (sampling of a) as identity function (as result de MetaHypernetwort parameters:
1) and computes the gradient w.r.t. to the
Oc V V Vesoftmax(Ho)) da
To recap, during the forward pass the Architect samples discrete architecture from architecture distribution parameterized by the MetaHypernetworrk and during backpropagation the STE is utilized propagate back through the sampling operation and update the MetaHyperne teseereewwrrr rk parameters, hence the distribution from which the discrete architectures in the next iteration will be sampled.
A.3 FRANK-WOLFE SOLVER
In this section, we provide the pseudocode of the Frank-Wolfs solver (Jaggi 2013) used to compute
the gradient coefficients used for the MGD updates. To solve the constrained optimization problem. the Frank-Waan solver uses analytical solution for the line search with T = 2 (Algorithm
Algorithm FrankWolfeSolver (Jaggi 2013)
Data: g :90 - Result: (7..... ?T)
Initialize y (7.....
Precompute M st. Mij (g6) (196)
repeat
it argmin, Et= MrM
e; Mi,:
// 8-th row of M
8 + argming ((1- d dei)) (11-117 8ez)
// using Algorithm
+ (1- 8)y + de;
until 0 Number of Iterations Limit; return
Algorithm 5: Solver minse(o,1) ||00 + (1 0)011
i 2 T then
8
else if AT
0T0 then
8 8 0
else
8
110-011
return
8
20

---

inner routine in Published as a conference paper at ICLR 2025
B MULTI-OBJECTIVV NAS ALGORITHMS
This section elaborates the multi-objective NAS methods utilize baselines in Section 4
Random Search (RS) is a robust baseline for both single-objective (Bergstra Bengio, 2012
& Talwalkar 2020) and multi-objective (Cai et al. [2000] Chen et al.) 2021a) architecture searches. This baseline involves randomly sampling architectures from the search space and computing the Pareto front from these samples. While RS is computationally efficient and often effective, it may not always find the optimal architectures, especially in larger search spaces.
Local Search (LS) is adapted to refine solutions near Pareto-optimal points in multi-objecttvv optimization, iteratively improving solutions within defined neighborhoods.
Multi-objective Asynchronous Successive Halving (MO-ASHA) (Schmucker et al. 2021 is multi-fidelity method that utilizes an asynchronous successive halving scheduler (L.i al. 2018 and non-dominating sorting for budget allocation. MO-ASHA uses the NSGA-II selection mechanism and the e-net (Salinas et al. 2021) exploration strategy that ranks candidates the same Pareto set by iteratively selecting the one with the largest Euclidian distance from the previous set of candidates.
Multi-Objectinn Regularized Evolution (MO-RE) builds on Regularized Evolution (RE) (Real et al. 2019). which evolves population of candidates through mutation and periodically removes the oldest individuals, thus regularizing the population. MO-RE adapts this by using multi-objective non-dominated sorting score candidates, with parents sampled based on these scores.
Non-dominated Sorting Genetic Algorithm II (NSGA-II) (Deb et al. (2002) is multi objectvvbeetivee evolutionary algorithm designed to find a Pareto set of architectures. It ranks architectures using non- dominated sorting and maintains diversity with crowding distance. Through selection, crossover, and mutation, NSGA-II evolves populations towards the Pareto front, although it is known for being sample inefficient.
Covariance Matrix Adaptation Evolution Strategy (CMA-ES) (Igel et al. 2007 is an evolu- tionary algorithm particularly effective in continuous optimization problems. context, it adapts its covariance matrix to the shape of the search space, iteratively updating its sam- pling distribution to favor promising regions. This method efficiently handles complex, non-linear optimization landscapes and can be adapted to multi-objective scenarios by using techniques such as Pareto-basse selection to maintain a diverse set of solutions.
multi-objective
Latent Action MOO (LaMOO) (Zhao et al., (0022) uses parametric model and Monte Carlo Tree Search (MCTS) to learn to partition the objective space based on the dominance number, which indicates the vicinity of a point to the Pareto front relative to the other samples. qEHVI+LaMOO and CMA-ES+LaMO the learned subspaces.
the original qEHVI and CMA-ES, respectively,
Bayesian Optimization with Random Scalarizations (RS-BO) (Paria et al. 2020) uses an acquisition function based on random linear scalarizations of objectives across multiple points to find the Pareto-optimal set that minimizes Bayesian regret.
Bayesian Optimization with Linear Scalarizations (LS-BO) is similar to RS-BO but optimizes a single objective derived from a fixed linear combination of two objectives instead of using randomized linear scalarizations.
Expected Hypervolume Improvement (qEHVI) (Daulton al. 2020) is Bayesian optimization acquisition function that explores the Pareto front by quantifying potential hypervolume improve- ment. This approach measures the volume dominated by Pareto-optimal solutions and guides the search towards regions likely to offer better trade-offs, aiding in the discovery of diverse Pareto-optimal solutions.
C EVALUATION DETAILS
C.1 OTHER METRICS
For MASS-Benc-224 in addition, evaluate the generational distance (GD) and inverse generational distance (IGD) (see Appendix(Cy (). See Figure for the results complementary to the hypervolume radar plot in Figure of the main paper.
21

---

Published as a conference paper at ICLR 2025
Generational Distance (GD) and Inverse Generational Distance (IGD). Given reference set C A and a Pareto set Pa C A with dim( = K, the GD indicator is defined as the distance between every point a Pa and the closest point in E S, averaged over the size of P.:
1/2
GD(Pa, = [Pal
Σ
min d(a, d(a, s)² #ES
aePa
where d(a,
E (ax - -k)) is the Euclidean distance from a to its nearest reference point
in S.
The inverted generational distance (IGD) is computed as TGD(Pa, = (
Generational Distance Plus (GD+) and Inverse Generational Distance Plus (IGD+).
GD (Pa. IGD+GG* (S, Pa) replaces the euclidean distance d(a, s) in GD with:
dt (a, (maxfan 8k, 0})
C.2 MODNAS-SOTL
On the NAS-Bench-201 search space, since the architectures evaluated with the supernetwork weights are not highly correlated to the ones trained independently from scratch, we employ the Sum of Training Losses (SoTL) proxy from Ru et al. (202T). To profile the Pareto front with SoTL, we firstly evaluate the 24 architectures using the exponential moving average of the sum of training losses for the initial 12 epochs of training as Σ 0.9 Ctrain (w, a)). and then train from scratch only the subset of architectures the Pareto set built using the SoTL evaluations. We present the results of MODNAS- in Figure 20 where compare to the other baselines well. As we see, further decrease the evaluation cost via MODÑAS-SOn -STTL, by trading off the number of solutions in the Pareto set with HV.
D EXPERIMENTAL DETAILS
D.1 MetaPredictor ARCHITECTURES
For all search spaces we the dimensionality of the hardware embedding to 10. This corresponds to latency evaluations on set of 10 reference architectures, which are the same used by Lee et al. (2021b).
Graph Convolutional Network (GCN) as proposed in Dudziak et al. (2020). Furthermore. in addition to the one-hot operation encoding and adjacency matrix corresponding also input the hardware embedding to this predictor, as done by Lee et al. (2021b). The number of nodes in the GCN is 8 and the dimensionality of the layers is set 100 tollowing HELP (Lee) al. 20211). In order to show the effecttiveness of our metaayypernetwrk to learn the hardware device similarities, in Figure 12 we cluster the original device embedding vectors and the learned Me eetaHyperneet tewpporrnewwrrrr rk embeddings using K-means clustering after reducing their dimensionality using t-SNE. As we can see, the detaHypernettwon learns to cluster similar devices together in latent space, demonstrating the efficacy of our algorithm.
-25- For the NAS-Bench-201 (Dong & Yang 2020) search space
the architecture cells, we
MobileNet (OFA). Following HELP (Lee et al.] (021b), we employ simple feedforward neural network in the MobileNetV3 search space. The input dimension of the MetaPredis ctor is set to 160, matching the concatenated architecture encoding dimension. We set the size of the hidden layers to 100. Specifically, the MetaP redictor comprises linear layers with ReLU activation for processing the 160-dimmnsional one-hot architecture encoding and 2 linear layers for processing the hardware embedding. The outputs from these two paths are concatenated and passed through a final linear layer to predict the latency.
22

---

Published as a conference paper at ICLR 2025
TSNE Projection for Hardware Embeddings
TSNE Projection for Device Pool Similaiitie
fanx
300
Cunterd
200
0000
@amsung
guspl
Cunter
Custer 2
Custer 3
100
gel 5240
do d
103
32
@ 235
TSkV Diners
go0o 254
graent al, ph
-200
hrst 4210r
255
gold 6260
-100
esmital [
@ 258
CusterO
500
do 62.222
g0000 32
602
502
400
200
200
400
500
230
200
153
100
50
50
100
TSNE Dimorsion
Oemrnssion
(a) Original hardware embeddings
(b) MetaHlyerrettook embeddings
Figure 12: K-means clustering on the t-SNE projections of the original hardware device embeddings and learned embeddings from the MetaHypernetun on NB201.
Seq-Seq Transformer (HAT). HELP does not release the architecture the meta-learned pretrained predictor for HAT et al. 2020b). However, HAT f releases code and pretrained models for each of the devices and tasks trained independently. Hence, we build our single per-task MetaPredictor based on the architecture of the HAT predictor, i.e. a simple feedforward neural network. The input dimension corresponds to the one-hot architecture encoding of the candidate Transformer architecture. Additionally, to condition on the hardware embedding, we include extra linear layers for processing the hardware embedding, which is then concatenated with the processed architecture encoding to produce the final latency prediction. The hidden dimension of the MetaHypernee twow rk is set 400, with hidden layers. The predictor's input feature dimension is 130.
HWW-GPT-enchh. We utilize the raw energy observations released in (Sukthanker et al. 2024) to train single hardware aware meta-predictor across energy observations from eight GPU types. Our meta-predictor is simple MLP, similar to the one in HAT, with 4 hidden layers, 2 layers for processing the hardware embedding (which the network conditioned on). The MLP's hidden dimension is 256, and the input feature dimension matches the one-hot encoded architecture feature map for this space, i.e., 80.
D.2 Metaatypernetwork ARCHITECTURE
R + A. parameterized by & E I2", to generate an un-normalized architecture distribution ā that is later used to compute the upper-level updates in ( In our experiments, he is composed of M embeddding layers
Given a preference vector E M we use the hypernetwork hT(r)
e" m E (,...., M) with n.m possible learnable vectors of size dímm(AA) The output of he is the concatenation of all M- - outputs of em,, such that its size matches (A)). See Figure 13 for details.
M-1-
In order to enable the hypernetwork to generate architectures across multiple devices, inspired by Wang et al. (022) and Lin et al./ (2020). we propose a MetaHypernetwork Ha(r, d:) : :R. R
x A that can meta-learn across T different hardware devices (see Figure I. The input to Ho is a concatenation of device feature vectors across all metrics, i.e. d, = 4 M d Similar to
HM
Lee et al. 20211b, dmhh H is fixed-size feature vector representative of device E (1 (1..... 7) and objective m E {2,... ..
.MM }. that is obtained by evaluating a fixed set of reference architectures for given metric. The MetaHype remprrreewwrrr
with Φ- o parameters, contains a bank of K T hypernetworks (h (r) and an additional linear layer ego (de): HM-1 R at the beginning. that learns a similarity map for every device feature to the hypernetworks' bank. If we denote by
https:://iituub..m//ivveeenn com/mittommmttmmmomemm han- lab/ha hww.aaawwwwweeeeeeeee aware tranee--rrsfffoomrr (CE loss) does have hardware embedding.
23

---

Published as a conference paper at ICLR 2025
h (hdr ... hee) )J the vector of all hypernetworks in the bank, then, given a preference vector to obtain for device t, we compute weighted mixture of predictions all he in the hypernetwork bank as follows:
K
öΦ = Ho de) = E0o (de) hor (r)
E ( (dz) Iepaaa (r).
We keep the MetaHlpprnetworrk architecture similar across search spaces. The only thing we adapt is the output dimensionality of the hypernetwork (in the hypernetwork bank of MetaHypernetwon
dint M -1
2
which corresponds the dimensionality of the architecture parameters of the respective search space. We set the size of the initial hardware embedding layer and the hypernetwork bank to 50 for all search spaces. Furthermore, each hypernetwork has 100 pos- sible learnable embeddings e"h, . for every ob- jective E ....... MM, to map the scalar- ization vector to an architecture. We quantize the continuous sampled r' [0, 1] the dis- crete [0, 1,... 100] interval before indexing the respective embedding layers. See Figure 2 for an illustration of the MetaHyperne two rk archi- tecture.
Tm
22
hpp
hp:
d
Cb.
hΦz
For the NAS-Bench-201 search space, a single embedding layer of dimensionality 30, corresponding the dimensionality of the ar- chitecture space: x 5 (6 edges and 5 operation choices on each edge). For the 3-objective ex- periment, we include an additional embedding for the energy usage objective, concatenated with the latency embedding before passing it The individual hy-
Figure 13:
MetaHypernetwon architecture overview in the case of M objectives. Note that m
reserved for the accuracy objective, which we model through the cross-entropy loss the Supernetwork, The initial linear layer 6 gets the de hardware embedding and outputs weight that scales each of the K hypermetworks' (orange boxes) outputs from the hypernetwork bank. The scaled architectural parameters are then summed up element-wise. All individual hyper- network hog, get as input the same scalarization r. Each of them has M - embedding layers with dimensions nm x dim(AA) that gets input the scalarizations for objectives 2, ......"h M, and yields a vector of size (A) The output from the M 1 embedding layers concatenated to give the architecture coding à.
to the Met MetaHypereettee pernetworks in the MetaHyperne twork bank have embedding layers with dimensionality 15, whose outputs concatenated to match the architecture space dimensions.
the MobileNetV3 space, we use embedding layers- - for depth, expansion ratio, kernel size, and resolution. The space comprises blocks, each with 3 depth choices, making the depth em- bedding layer dimensionality x 3. The kernel and expansion embedding layers have dimen- sions x 4 x 3, corresponding to 5 blocks with maximum depth of and 3 possible kernel size or expansion ratio choices. The resolution embedding layer has dimension of 25, repre- senting 25 possible resolution choices.
M Ym E {2..... .......) M)
M-1
In the Seq-Seq Transformer (HAT) space, the individual hypernetworks of the MetaaHypernetwork utilize 9 embedding layers (the encoder layer count is fixed; Table
2 embedding layers size for the encoder and decoder blocks to map the scalarization to embedding dimension architecture parameter, held constant throughout the encoder or decoder block.
2 embedding layers with dimensions x 3 (6 encoder/decoder layers, choices) for the linear layer size in every attention block for both encoder and decoder.
24

---

Published as a conference paper at ICLR 2025
2 embedding layers with dimensions 6 X for the number of heads in each attention block. embedding layer of size 6 to encode the possible choices for the number of layers in the decoder.
embedding layer of size 6 x 3 (6 encoder layers, 3 choices) for the arbitrary encoder layer choice for attention.
embedding layer of size 6 x 2 (6 encoder layers, choices) for the number of heads in the encoder-decodes attention.
For the LW-GGPT-Bench space, the individual hypernetworks the MetaaHypernetwork contain embedding layers:
embedding layer of dimension 1 x 3 for mapping the scalarization to the embedding dimension architecture parameter of the language model, with choices.
embedding layer of dimension x for mapping the scalarization the layer number dimension architecture parameter of the language model, with choices.
embedding layer of dimension 12 3 for mapping the scalarization to the mlp_ratee ratio dimension architecture parameter of the language model, with 12 layers and mip_rane aip_ratio choices per layer.
embedding layer of dimension 12 x for mapping the scalarization to the num heads dimension architecture parameter of the language model, with layers and choices per layer.
embedding layer of dimension for toggling the bias in linear layers on or off.
D.3 MODNAS HYPERPARAMETHR CONFIUURATIOSS
In Table 2 we show the search hyperparameters and their corresponding values we use to conduct our experiments with MODNAS. For the convolutional spaces we subtract a cosine similarity penalty from the scalarized loss following (Ruchte & Grabocka 2021):
r T (D w. (..)
d - TV L. L w, (,,a)) AV
(6)
L, (D natidd w, (0,a)))l
where || is the le norm. We set À to 0.001. Empirically we did not observe significant differences on disabling the cosine penalty term.
D.4 NORMALIZATION OF OBJECTIVES
Since method relies scalarization of different objectives, it is important that the objectives being optimized are on the same scale. For simplicity, lets consider the scenario where the two objectives of interest the cross-entropy loss and latency. Since pretrain and freeze MetaPredicte the latency-scale remains constant throughout the search, while the eross-entropy loss of the Supernetwor (likely) decreases over time. To this end, we the following max-min normalization to normalize the objectives:
CT (.)))))) min(L)
C-
max(L.) min(L.
where L U stop_e stop_g((o))) (, a;)))) is the set of losses evaluated on N architectures and potentially N previous steps. For the latency objective, we precompute these sample-statistics using N samples (ground-truth for MAS-Bench-200 and predicted for OFA and HAT spaces) from the search space, whilst for the cross-entropy loss we compute them throughout the search. Furthermore, to take into account the decreasing cross-entren cross-entropy, we reset the cross-entropy loss statistics after every epoch.
25

---

Published as a conference paper at ICLR 2025
Table 2: Hyperparameters used on different search spaces
Search Space
Hyperparameter Type
Value
learning
3e.4
weight docay
le-3
embedding layer
Metaltypernetwokk
hypernetwork bank size
50
optimizer
Adam
ReinMax tomperature
karning
0.025
momenium
0.9
NAS-Bench-201
weight decay
0.0027
learning rate scheduler
cusine
opochs
Supernetwork
batch Size
256
gradient Clipping
culout
true
length
Initial channels
16
optimizer
SGD
train portion
0.5
learning rite
le-5
Weight decay
emhedding layer size
100
Met Metaitypernetwork
hypernework bank
optimizer
Adam
ReinMax temperature
karning
1o-3
momeniru
MobileNetV3 (OFA)
weight decay
30-5
Isarning rate scheduler
cusine
epochs
50
Supernetwork
bulch size
32
bi. on_momentum
0,1
bn, ops
dropout
0.1
width
optimizer
SGD
train portion
learning mite
3e.4
weight decay
1e3
Me Metaltype tntwwork
emheddling layer size
100
hypernetwork bank
optimizer
Adam
ReinMax temperature
learning rale
lo-7
momentum
Seq-Seq Transformer (HAT)
weight decay
0.0
Ssarning scheduler
cosine
epochs
110
batch sizc/mxxttoken
4096
Supernetwark
criterion
smoothed cross cntropy
affention-dropout
dropout
0.3
precision
float32
optimizer
Adam
portion
1.0
learning
weight decay
1e33
Me Metaltypernetwork
embbediin layer
hypernetwork bunk size
50
optimizer
ReinMax temperature
learning rale
0.000316
momenfum
HW-GPT-Bench
weight decay
learning rate scheduler
cosine
steps
800k
batch size/max-tokens
32768
Supernetwark
criterion
cross, coos_cyttrppy
attention-dropour
0.0
drogout
precision
biloat 16
optimizer
W
train portion
1.0
26

---

Published as a conference paper at ICLR 2025
E DETAILS ON SEARCH SPACES
NAS-Bench-201 DDngg& & Yang 2020) is con- volutional, cell-based search space. The search space consists of 3 stages, each with number of channels 16, 32 and 64, respectively. Each stage contains a convolutional cell repeated 5 times. Here, every cell is represented as a di- rected acyclic graph (DAG) which has nodes, densely connected with 6 edges. Each edge has 5 possible operation choices: a skip connection, zero operation, a 3x3 convolution, a 5 con- volution or an average pooling operation. NAS- Bench-201 is a tabular benchmark exhaustively constructed, where the objective is finding the optimal cell for the given macro skeleton.
Table 3: Encoder-Decoder Search Space for HAT.
Module Encoder
Searchable Dim
Choices
No. of Layers Embedding dim
[6] (fixed)
[640, 512]
No. of heads
[8.
FFN dim
[3072. 2048. 1024]
Decoder
No. of layers
[6. 5, 4,3,
Embedding dim
[640, 512]
No. of heads
[8.
FFN dim
[3072. 2048, 1024]
Eibitrary-Encodae Layer
[-1,
Enc-Dec attention num heads [8,
MobileNetV3 proposed in OFA (Cai et al., 2020) is a macro convolutional search space. The different searchable dimensions in the search space are the depth (per block), the kernel size (for every layer in every block) and the channel expansion ratio (for every layer every block). There are a total of 5 blocks, each with 3 possible depth choices and every layer in this block has 3 possible kernel sizes and channel expansion ratio choices. This amounts to a total search space size of ((3 3) + (3 3) + (3 x 3) = 2 x 10 Additionally, every architecture has 25 possible choices for the size of the input resolution. The 3 possible choices for depth, kernel size and expansion ratio {2, 3, 4), {3, 5, 7) and (3.4, 4, 6). respectively. The input resolution choices {128, 132, 136, 140, 144, 148, 152, 156, 160, 164, 168, 172, 176, 180, 184, 188, 192, 196, 200, 204, 208, 212, 216, 220, 224). We use a width factor of 1.2 similar to OFA (Cai et al. 2020).
Seq-Seq Encoder-Decoder Transformer (HAT) (Wang et al. 2020a) for the En-De machine translation task has a searchable number of layers, embedding dimension, feedforward expansion layer dim per-layer, number of heads per-layer for both the encoder and the decoder sub-modules. In addition to this, the number of encoder layers the decoder attends to, and the number of attention heads in the encoder-decoder attention is also searchable. We present the details of the search space in Table 3
HW-GPT-Bench (Sukthanker et al., 2024) is a decoder-only transformer space designed for autore- gressive language modeling. The search space includes choices for embedding dimensions [768, 384, 192), the number of layers from (10, 11, 12), the MLP expansion ratio per layer from (2,3,4), the number of heads per layer from (12,8,4, and the option toggle the bias parameter on off in the layers.
Table 4: Search-test split for hardware devices and datasets for different search spaces.
Search Space
Train-devices
Test devices
Datasel
1080ri_ J. 1080ti 32, 1080ti 256, silver 4114,
ritan_ stx_ 256, 6226 fpga, pixel2, raspi4, eyeriss
NAS-Bench-201
silver 4210r, samsung a50, pixel3, essential ph
CIFAR
samsung_s7, titanx_ titanx__ 32, titanx__566 gold_6240
2080u_1 20800i_322 2080t 64, titm_xp_1. titan xp_32, _4___ v100_I, v100_32,
iita_rtx__ 64
MobileNetV3 (OFA)
ImageNet-1k
v00 64, titan_rix__ tilan_ rtx_2
Seq.Seq Transformer (HAT)
titanxp gpa, cpu xeon
epu raspherrypi
WMT14.en-de
HW-GPT-Bench
a40, v100, ax20800 r0X3880
a100, h100, P100,
OpenWebiexxt
F DATASETS AND DEVICES
This section describes the hardware devices and tasks used to evaluate MODNAS and the MOO baselines throughout the paper. We assess our methods across small- and large-scale image clas- sification datasets, including 10 and I For the machine translation task, we evaluate our method on the WMT' 14 En-De dataset (Macháček & Bojar 2014), and we use the Seeeee (Gokaslan & Cohen 2019) dataset for language modeling. Furthermore, we evaluate MODNAS across 19 devices on NAS-Bench-A 12 devices on MobileNetV3, three devices on Seq- Seq Transformer, and eight devices from HW-GPT-Bende (Sukthanker et al. 2024), with zero-shot
27

---

Published as a conference paper at ICLR 2025
generalization to test devices. Table 4 lists the devices used. For more details on the devices, refer readers to Lee et al. (2021b), Cai et al. (2020), Wang et al. (2020b). Li et al. (2021). and Sukthanker al. 2024).
G RUNTIME COMPARISON
In Table 5 we provide the number of GPU hours we ran MODNAS and baselines on every search space. We ran the search on WAS-Bench-200 OFA, together with the evaluations on Nvidia RTX2080Ti, while for HAT we used NVidia A6000. For both OFA and HAT, we used GPUs in parallel. Similar as Sukthanker et al. (2024), on the HW-GPT-Bench space we ran the MODNAS search and evaluations on Nvidia A T00 GPUs.
Table 5: Total amount GPU hours required to run MODNAS' and baselines' search on every search space.
Search Spaces
Method
Lat/En/Mee Pred.
Supernet
Acc//ppl Pred.
Search
Total Time
MetaD2A-HELPP
8629
0.3
8654.3
NASBench201
MOO Basclines
370.5
370.5
MODNAS
3
22
0.05
25.25
OFA+HELP
1200
356
10
1572
Once-For-All
MOO Baselines
1200
356
192
1754
MODNAS
1392
0.05
1398.25
HAT
15
346.7
210.9
572.6
HAT
MOO Baselines
346.7
576
937.7
MODNAS
0.05
581.25
MOO Baselines
HW-GPT-Bench
MODNAS
216
0,05
217.25
H ADDITIONAL EXPERIMENTS
H.1 ADDITIONAL RESULTS ON
In Figure 19 we present the Pareto fronts obtained by our method in comparison to different baselines on the NAS-Bench-201 search space. In Figure 20, present different additional metrics, such as GD and IGD (see Section C). to evaluate the quality of the Pareto fronts obtained on NAS-Bench-201. Figure 21 presents the Pareto front MODNAS yields when applying different latency constraints during the search phase. Figure T5a compares our method using the ReinMax gradient estimator to the GDAS estimator (Dong Yang 2019). As we can see, ReinMax obtains qualitatively better hypervolume coverage compared to GDAS. Figure 16] presents the 3D Pareto front and hypervolume obtained by MODNAS compared other baselines when optimizing for accuracy, latency and energy usage on NAS-Bench-209 Figure 24 presents the comparison of MODNAS with MGD to other gradient aggregation schemes, such as mean, sequential and MC sampling (see Section (.I). across multiple hardware devices. Finally, in Figure25 we present the robustness of MODNAS to the fraction of devices used for the predictor training and the search phase. addition, in Figure [8 and [7] we compare MODNAS against different MO baselines the CIFAR-100 dataset two different devices.
H.1.1 PREDICTED v/s GROUND-TRUTH LATENCIES
In Figure 14 we present the seatter plots of the predictions of hardware-aware MetaPredicton ctor vs. the ground-truth latencies of different architectures. In the figure title we also report the kendall- correlation coefficient for every device. As observed, our predictor achieves high kendall-r correlation coefficient across all devices.
28

---

Published as a conference paper at ICLR 2025
boric
Figure 14: Scatter plots of predicted latencies from our pretrained MetaPredictor vs. ground-truth latencies (test devices in red).
29

---

Published as a conference paper at ICLR 2025
AGD
AUD
10010 19060
MLAKE
ROCIM SariEAY
RROMI
(a) Hypervolume
(b) GD+
(c) IGD+
Figure 15: Hypervolume, GD+ and IGD+ of MODNAS with Reinmax gradient estimator in the Architect vs. the one from GDAS (Dong & Yang 2019) and MiLeNAS (He et al. 2020) across 19 devices NAS- A.S-Bench-201.. Higher area in the radar indicates better performance for every metric. Test devices are colored in red around the radar plot.
Glabal opt.
RS
Eyeriss
RHPN MODNAS
MD BE MO-ASIIA NSGA-1 RUS 3O
0.91 63
Hyporvoluao
Global ort.
Ho 4004
15.00
10.0
2.0
8.23
Figure HV (left) and Pareto front (right) of MODNAS and baselines on Eyeriss with normalized objectives: error, latency and energy usage. HV was computed using the (1, 1, 1) reference point on the right 3D plot.
edgegpu latency
pixe13 latency
0.35
0.90
0.9
0.90
0.92
0.29
0.85
Hypervolume
Ngpervolurg
0.50
0.75
6.73
0.75
0.70
0.80
68.80
80
s
ND Apile
Ascae
abyii
HD at
060
15.00
MOZENN
No-s5
Figure 17: Hypervolume on CIFAR- 100 and edgegpu device.
Figure 18: Hypervolume on CIFAR- 100 and Pixe13 device.
30

---

Published as a conference paper at ICLR 2025
2310
1S7afoad
polz
101012
00
0.2
0.4
0.5
0.3
03
344
0.5
03
02
344
3.5
3.
03
0.2
344
3.6
10007: 254
swtrto po
0utl 0240
9363
ONPOO 3to
3.3
60
3.5
012
123
Bonc
thars 32
Bhov: 256
Merraliee
Merraltee
Figure 19: Pareto fronts of MODNAS and baselines on NAS-Bench. 2011 MODNAS-SoTE is not shown for better visibility.
31

---

Published as a conference paper at ICLR 2025
Phots Sumung www ARLEY tomm 10099.35 PrR
PALL US SEAR 255 gol
MIDONAS
CMA-ESALAMDO
R5-80
NSGA-I
opt.
GEHVI+LANOO
MO ASHA
484V
(a) HV
OWAES
1540
22
L540
H50M
RAPS
M0
RIPH
POTTAO 5/T.
(b) GD
(c) IGD
MODIAN
K5.00
4000
CMACS-LUNOO
15-80
(DIM
RIFH
POTROO SVT.
(d) GD+
IGD+
Figure 20: HV, GD, GD+ IGD and IGD+ of MODNAS and baselines across 19 devices on NAS- Bench-201 For every device we optimize for objectives, namely latency (ms) and test accuracy on 10. For method, metric and device report the mean of 3 independent search runs. Higher area in the radar indicates better performance for every metric. Test devices are colored in red around the radar plot. Here allocate double the budget to baselines, i.e. run all baselines for 50 function evaluations.
32

---

Published as a conference paper at ICLR 2025
LEAEE CLESERE
36
LECOL CECCEM
LECIN AU
NODIAC 31
NOCRACHE 21
MOCING 1631
MOCING 1C.2
MECHAAN 241
MOROONID EI
-1-1
60
C8-
10
16-
0.
02
344
53
0.0
020
825
933
1950 JTS 256
10900)
10800) 12
NOCAAG
MOCIIC
MECHAAN 2-41
METAAAA 15
PISTOAA 11-5
3.4
NETTRAN
NETRGAAT FI
11
CUP
3.4
Santed aollara
Liten corerset
Liserc CEraERit
LICENC conmTare
MESHAA 2:21
MOOMAGEAA
PL
1017
MHNTTS H017
MinurSh 1017
forraleed
semsung 450
silver, 4114
atbvar 42201
6.0
3.1
21
NODAKS TOE
30
02
24
Blans
Blane
18ans 258
MODING
POTHOA
NetitaA HOGL)
Figure 21: Pareto fronts MODNAS ran with different latency constraints during search.
33

---

Published as a conference paper at ICLR 2025
ofdera 2730
2000
TOPOLE
MODDAS
MCONAS
MOCNAS NOONAS
(LO)
MODNASID
MCCNAS
MODMAS
MODNAS (D
(8) GD
(b) IGD
10 GOER SAMAL SD
A200
0.29
MODNAS MODNAS MODNAS
MODKAS
MODNAS
MODKAS
MOCNAS
MODNAS
MccuDAA-HE HELP
MODNAS
MODNAS
MODNAS
(c) GD+
(d) IGD+
Figure 22: GD. GD+ IGD and IGD+ of MODNAS with different latency constraints during search across 19 devices on metric. Test devices are colored red around the radar plot.
Higher area in the radar indicates better performance for every
34

---

Published as a conference paper at ICLR 2025
380
70444 10
372
1100 238
100
Figure 23: HV over number of evaluated architectures NAS-Bench-205 of MODNAS and the blackbox MOO baselines. Note that for MODNAS we only have 24 evaluations in the end.
35

---

Published as a conference paper at ICLR 2025
120
0.00
0.11
Exeere
Seart Exers
u00
oiver 4223
clana__
can
Figure 24: HV over time NAS-Bench-201 of MODNAS with different gradient update schemes.
36

---

Published as a conference paper at ICLR 2025
0:30
4033
E25
E20
Figure 25: HV over time on NAS-Bench-201 of MODNAS with different number devices during search. For number of devices less than 13 (default one) randomly select subset from these 13 devices.
37

---

Published as a conference paper at ICLR 2025
H.1.2 ALIGNMENT PREFERENCE VECTORS WITH PARETO FRONT
In this section, we provide empirical ev- idence that the solutions generated us- ing the MetaHypernette retwwrkk align well with the preference vectors. To this end, we utilize one of our runs on the NAS- Bench-201 test devices, namely Eyeriss. In Figure 26 we show the Pareto front of the normalized test error and latency on Eyeriss. Note that of the 24 sam- pled preference vectors, 17 generate lutions that are in the Pareto set. Each point in the Pareto front with certain color corresponds to the preference vee- tor with the same color. In the figure, there are actually 17 points in the Pareto front; however, some of them are really close to each other or are the same, since the function mapping preference vectors to architectures is many-to-one function. Nevertheless, vectors starting from the origin align very well with the generated solutions. The missing vectors are mainly in the center, where there not many solutions available for this particular device.
Pareto Front with Preference Vectors
Foretc Eyerts
Labency
0.1
0.
0.
0.8
Emar
Figure 26: Pareto front and preference vectors on the nor- malized Eyeriss latency and test error of WAS-Bennh-2201. visually notice that the preference
1.3.3 TRAINING AND VALIDATION Loss CURVES
In addition the hypervolume indicator, in this section, provide the training and validation loss curves in Figure 27 (left). At each mini-batch iteration we plot the average cross entropy loss across all devices. As expected both training and validation cross-entropy go down and we do not notice any overfitting. The high noise is common for sample-based NAS optimizers, since a sampled different architecture activate each mini-batch iteration. In the plot, for visualization purposes, we have used a running average with a window size of 100 to smooth out the noise.
Aisi samnung A210r ohear TorSt 258 EAR COna
Traiiing Loss Volidation
2.15
2.10
Awa
2.00
1.95
10
Stopc
10a
gold
MODNAS M0-A5HA
MC-RE
NSCM-
R5
QEHVI
opt
1580
Figure 27: (left) Average training and validation cross-entropy loss across devices during the MOD- NAS search on NAS-Bench-201. (right) HV of MODNAS and baselines across 19 devices on NAS-Bench-201. For every device we optimize for 2 objectives, namely latency (ms) and test accuracy on CIFAR-10. 10. For method, metric and device we report the mean of 3 independent search runs. Higher area in the radar indicates better performance for every metric. Test devices are colored red around the radar plot. Here we allocate times the budget baselines, i.e. we run baselines for 100 function evaluations.
38

---

Published as a conference paper at ICLR 2025
H.1.4 MULTI-OBIECTIVE OPTIMIZATION BASELINES WITH MORE BUDGET
Black-box multi-objective optimizers can potentially reach the global Pareto front the compute resources not concern and given enough time. However, is practical train evaluate these architectures, especially for larger model sizes (e.g. Transformer spaces from HW- GPT-Bench). Sometimes in practice, the user wants get quick estimation of the Pareto front instead the global optimum, and this is the use-case where MODNAS shines, Given enough budget, even a random search (RS) will find near-optimal solution. For example, in NAS- NAS-Bench-22 Bench-201, the size of the search space is K = 15625 architectures, The optimal theoretical number of RS steps achieve a success probability
approximately: 2 Kln ( a), therefore. for random search to have success probability higher than 0.5 requires 2 10781 iterations in theory. For other guided search methods, this number even smaller, though similar to MODNAS, they have the same limitation that they can converge to local minimum. We conducted the same experiment as the one Figure 3 but this time with baselines given times more budget than MODNAS, We see, of the methods such LS-BO can reach
show the result in Figure 27 (righr). As results closer to the global Pareto front compared to MODNAS.
H.2 ADDITIONAL RESULTS ON HARDWARE- TRANSFORMERS (EN-DE)
We show the Pareto fronts of MODNAS compared to baselines for the Transformer space in Figure 28 as well as their comparison with respect to hypervolume for the SacreBLEU metric in Figure 30 These results demonstrate the superior performance of our method compared to the other baselines on this benchmark. All evaluations are done by inheriting the weights of a pretrained supernet.
RaspberryPi-CPU
0.055
VO-BE
MO-R:
MSCK-1
RICAL
0.002
15:00
0.045
RS
ASPA
0.2
MO-A5-4A
2BP11
6DIV
1.5
2.5
0.232
0.00
033
0.037
3000
3000
5020
6500
1004
5000
120
159
200
250
102
150
230
250
300
150
JaCanc
L2009//
RaspberryPi.CPU
GPU-TItanXP
CPU-Xoon
0.065
4D-R2
0.245
0.244
L5:50
8583
0.003
VO-AG-4
MD ASHA
MD 43PA
0.90
2EP1
0.04
0.940
114"
0,00
0.003
0.008
0200
0000
102
230
Figure 28: Pareto fronts of MODNAS and baselines the HAT space for the WMT En-De task.
All performance metrics are obtained from the inherited supernet weights.
H.3 ADDITIONAL RESULTS ON THE HW-GPT SPACE
we present the Pareto fronts on all the 8 GPU types for MODNAS and different baselines. The Pareto fronts are obtained using the perplexity and energy predictors trained on data collected in the HW-GPT-Bend' (Sukthanker et al. 2024).
In figure 31
39

---

Published as a conference paper at ICLR 2025
RaspberryPi-CPU
TitanXP-GPU
Xeon-CPU
0.72
6.70
Hyperoluuee
9.76
Hypervtume
0:00
0.00
0.70
0.20
0.54
6.62
6.72
Roto
Figure 29: Hypervolume (HV) of MODNAS and baselines across devices the HAT space. The objectives used to compute the HV are latency and BLEU score. Leftmost plot is for the test device.
RaspberryPi-CPU
TitanXP-GPU
Xeon-CPU
3.2
3.73
3:73
Myypsrelume
Hyparelum
0.53 8:33
642
3722
affer
Figure 30: Hypervolume (HV) of MODNAS and baselines across devices on the HAT space. The objectives used to compute the HV are latency and SacreBLEU score. Leftmost plot is for the test device. MODNAS is the best or on par to the baselines across all three devices.
40.0
MORGA
15
40.5
15
41.5
26.2
SEGAZ
31.5
MSGA2
36.2
ICGA3
35.3
USDC
RCGA2
2..0
MOASHA
21.3
E/M
MOXEHA
MORCHA
22.9
Ei
MOXGHA
27.2
27.5
212
22.9
22.
Erergy
Exceyy M1
Tereyy M1
Pee (011
40.0
R5
40.3
40.3
USD
RSGA2
36.3
ME002
16GA3
26-2
KGGA2
LSC
MOAGHA
UM
41.4
E-M1
MOIGHA
2.4
MONGHA
MONGHA
LINT
214
20.8
224
Exergy
Exeay Mh1
Exer7y M1
Deergy 2011
Figure 31: Pareto fronts of MODNAS and baselines optimizing for GPU energy consumption (Wh) and perplexity on the HW-GPT-PB GeT--Bechhhhh space.
40

---

Published as a conference paper at ICLR 2025
H.3.1 EXPERIMENTS ON PERPLEXITY AND MEMORY USAGE OBJECTIVES
In this section, we showcase the application of MODNAS for optimizing memory usage (using 6 precision and context size of 1024) and perplexity on Open Web- text within the HW-GPT-Bench (Sukthanker et 2024)
30
GEAVI
MO-ASHA
OpenWeb-
28
MO-RE
Perplexity
26
GPT-L search space, featuring Transformer models up 774M parameters. Since memory usage does not de- pend on the device type, our approach does not utilize the
RS BO
24
MGD updates Algorithm[[]] for computing the common
22
gradient descent direction, instead leveraging only pref- erence vectors to calculate the scalarized objective. This highlights once again the flexibility of MODNAS across diverse settings, even the ones it was not designed for. Despite this adjustment, MODNAS remains competitive, delivering a Pareto front comparable to leading black-box MOO baselines. We show the results in Figure 32
20
1 2 3
Bfloat16 Memory (MB)
Figure 32: MODNAS vs. baselines on optimizing memory usage and perplexity on GPT-L (774M) of HW-GPT-Bench.
H.4 ADDITIONAL RESULTS ON MOBILENETV3
In Figure 33 we present the Pareto fronts of our method compared to different baselines for 12 different hardware devices the MobileNetV3 space. We show well the Pareto front of OFA+HELP (Lee et al.) 2021b), ran with the original setting.
Table 6: HV of MODNAS and baselines on the OFA search space. For every device we optimize for 2 objectives: latency (ms) and validation accuracy on ImageNet-Ik,
Device
RHPX
HKLP
KHVI
1.8
I.S.BO
MO-ASHA
MO-REA
NSGA-11
Ziea-NSGA-11
MODNAS
0.638
0.748
0.697
0.749
tinn Ix._
0.722
0.698
0.663
0.751
0.734
0.753
0.736
0.753
0.752
0.744
0.690
0,763
23991
11.00 22
viDe
3.79
3.5
MODNAG
6061
VOHA
330
1540
15.0
315
1580
11.00
18
200011
20605 64
Stan vp.
3.79
3.3
3.75
3.7
3.74
3.72
MOONS
3.79
244
150
3.6
1380
1300
3.9
1540
3.0
K540
1540
2500
1.2
Mann nt.
30
1.30
3.24
HJ
322
1017
gi81
1880
1940
2.0
LS50
1530
1540
36
1540
180
1580
350
22
60
130
22
Figure 33: Pareto fronts of MODNAS and baselines the MobileNetV3 space.
41

---

Published as a conference paper at ICLR 2025
titan rtx 1
titan rtx, 32
titan rtx 64
0.02
0.1
9.73
3.7
Hyporvo
0.2
3.65
0.60
3.64
3.64
355
139
2080ti 1
2080ti 32
2080ti 64
075
333
0.73
3.73
Hypovooum
Hyponoume
Hypenoomm
3.65
8.65
355
3.5
355
2
2
titan xp
titan 32
titan 64
1
075
373
3.72
3.73
typoredumm
Hyporouung
355
355-
355
v100_1 1
v100_33 32
v100 64
035
072
232
2.22
3.23
Hypored unnc
3.09
311
Figure 34: Hypervolume across devices on the MobileNetV3 search space of MODNAS and baselines. Here the Nvidia Titan RTX is the test device.
42

---

Published as a conference paper at ICLR 2025
I FURTHER DISCUSSION ON THE ROBUSTNESS OF MODNAS
Initially observed by (Zela et al. 2020), differentiable NAS methods be very sensitive their hyperparameter choices, especially the regularization ones responsible for the loss landscape in the upper level problem. In our experiments, there were three crucial components that made MODNAS robust and to work reliably across benchmarks:
1. Choice of MetalHypernetwork update scheme: this played pivotal role in the performance of MODNAS, Although other gradient update strategies underperformed or started diverging (Figure 6). MGD converged relatively quickly to a hypervolume close to that of the global Pareto front. The convergence of MGD to a pareto stationary point is discussed in Désidééi (2012) and more recently in (haaang et al. (202). The convergence of MGD in bilevel optimization an open research topic (see recent results from Ye et al. (2024) and Yang et al.] (2024)) One potential scenario when MGD could fail is when the gradient directions of the objectives it is optimizing point in different opposing directions; however, this becomes practically unlikely, especially as the number of objectives grows (in our case we use it to find the common gradient across devices, which is for instance 13 on JAS-Bench-201),
2. Choice of gradient estimation method in the Architect: In Section B we discuss our choice for the method that enables gradient estimation through discrete variables (since architectures are discrete variables). We noticed that the ReinMax (Liu et al.) 2023) estimator always outperformed previous estimators such as the one in GDAS (Dong & Yang 2019) (Figure [5a), so we believe
this choice is crucial.
3. Weight entanglement vs. weight sharing in the Supernetwork: In early experiments on NB201 we noticed that weight sharing in the Supernetwork, was not only more expensive, but much more unstable well when compared to weight entanglement (Cai et al. [2200] [Sukthanker et al., 2023). even yielding diverging solutions quite often (common pattern seen in differentiable NAS with shared weights as you mention; see Zela et al. (2020) for instance).
We hypothesize that all design choices mentioned above play an implicit regularization effect on the upper level optimization in the bi-level problem, leading to a faster convergence and robustness (Chen) Hsieh 2020 (mmih al. 2021 I Zela et al. 2020).
J EXTENDED RELATED WORK
Multi-objective optimization. Multi-objective optimization (MOO) (Gunantara) 2018) is a crucial field in optimization theory, tackling decision-making scenarios with multiple conflicting objectives. MOO techniques can be categorized into gradient-basee based and gradient- free approaches. Gradient- MOO approaches, such as evolutionary algorithms and dominance- based methods like NSGA-II (Deb et al.] 2000), often suffer from sample inefficiency and are typically unsuitable for deep learning applications. On the other hand, gradient- -bsed MOO methods leverage gradients. The foundational work by [Désidéri (2012) has been significantly extended in multi-task learning contexts, demonstrating considerable potential (Lin al. 2019 Liu & Vicente, 2021 Mahapatra & Rajan 2020 Sennrr& & Koltun| 2018). However, these methods are primarily applied to fixed architectures, and adapting them to architecture search spaces is complex. This adaptation would require retraining each architecture with multiple objectives, which is impractically expensive for large search spaces. Another major challenge in MOO balancing the different objectives. To address this, preference vectors have been proposed to guide the prioritization of objectives on the Pareto Front (Mommal al. 2022 Ye & Liu, (002). An emerging approach to mitigate the retraining issue involves hypernetworks, which determine the weights of the main network in MOO scenarios (Lin et al. 2020), often incorporating preference vector (Hoang et al. 2023 Navon et al. 2021 Phan et al. 2022).
Neural Architecture Search. A major challenge in the automated design of neural network architec- tures is the efficient exploration of vast search spaces. Early NAS methods relied on Reinforcement
Learning (Zoph & Le, 2017), evolutionary algorithms (Deb et al.) 2002 Elsken et al. 2019b Lu
et al. 2020), and other black-box optimization techniques (Daulton et al. 2022) to train and evalu- ate numerous architectures from scratch. The advent of one-shot NAS introduced weight sharing among architectures by training an over para-aaaaeeeeeee network, known as a supernet. to expedite the evaluation of individual networks within the search space (Bender et al. 2018 Liu et al. 2019b
43

---

Published as a conference paper at ICLR 2025
Pham al. 2018: Saxena & Verbeek 2016). Differentiable one-shot NAS methods (Cai et al. 2018 Fu et al. 2020: He et al. 2020: Wu et al. 2019 2021) further improved efficiency by applying a continuous relaxation to the search space, enabling the of gradient descent to identify optimal sub-models within the supernet. In contrast, two-stage NAS methods initially train a supernet, often through random sampling of subnetworks, and subsequently employ black-box optimization to
identify optimal subnetworks (Bender et al. 2018 Guo et al. 2020 Li & Talwalkar| 2020).
Hardware-aware and Multi-objective Neural Architecture Search. Early NAS methods primarily focused maximizing accuracy for given task. In contrast, hardware-aware NAS aims optimize architectures for efficient performance on specific hardware devices (Benmeziane et al. 2021 Lee
al. 2020; Shaw et al. 2019 Zhang al. 2020), naturally leading to multi-objective NAS HHsu
et al. 2018 Kim et al. 2021 Tan et al. 2019). Two-stage NAS methods can be adapted to this context by incorporating a multi-objective search in the second stage (Cai et al. 2018 [to & Von Zuben
(023). However, most two-stage methods depend on random sampling during supernet training. which doesn't prioritize promising architectures. Differentiable NAS methods, such as those in Cai et al. (2018): Fu et al. (020)): Jiang et al. (202T): Wang et al. (201); Wu et al. (2019) 2021 y; Xu et al. (2020b), use latency proxies like layer-wise latencies and FLOPS (Dudziak al. (020) to evaluate hardware performance, combining task and hardware objectives with fixed weighting to find single optimal solution. However, changing the objective weighting requires a complete search rerun, which
is computationally demanding.
In contrast, our proposed search algorithm offers the entire Pareto Front of objectives in a single run, making it more efficient. While our focus on multi-objective NAS for hardware constraints, our technique is applicable to other objectives such as fairness (Das & Dooley, 2023 Dooley et al. 2023 Martinez et al. 2020), suggesting promising avenues for future research.
Application to Other Tasks. Finally, we want to briefly discuss the potential application of MODNAS to other tasks not mentioned in this work. Object detection is very important application since it is probably one the most important use cases of neural networks on embedded devices (e.g., in self self-driving cars) (Baller al.) [021] Lazarevich et al. 2023 Lee et al. (002). An interesting benchmark is YOLOBench (Lazarevich et al. 2023). where the authors benchmark more than 550 architectures on four datasets and four different hardware platforms. One way leverage the Supernetwork here would be to parameterize the search space via the AutoDeepLab (Liu et al. 2019a) supernetwork model, which parameterizes the resolution too. This parameterization would not only leverage MODNAS to work for object detection, but for other computer vision tasks such as semantic segmentation, disparity estimation, etc (Mohan et al. 2023).
44


Reviews:
[{"review_id": "TdaDVyq7SI", "reviewer": "Reviewer_ei5k", "summary": "The paper (MODNAS) presents an approach to Neural Architecture Search (NAS) that balances competing objectives—like performance, latency, and energy efficiency—across multiple hardware devices. By encoding user preferences as a scalarization vector, MODNAS efficiently searches for Pareto-optimal solutions across diverse devices in a single run. The method employs a hypernetwork to generate architectures for specific hardware configurations, leveraging multiple gradient descents for optimization.  The method has been evaluated on MobileNetV3 on ImageNet-1k, an encoder-decoder transformer space for machine translation, and a decoder-only space for language modeling.", "questions": "__1-__ Using hypernetwork for the NAS and prate-frontier learning is well known. \n\nThat limits the novelty of the proposed method.\n\n[A]Brock, A., Lim, T., Ritchie, J., and Weston, N. (2018). SMASH: One-shot model architecture search through hypernetworks. In the International Conference on Learning Representations\n\n[B[ Lorraine, Jonathan, and David Duvenaud. \"Stochastic hyperparameter optimization through hypernetworks.\" arXiv preprint arXiv:1802.09419 (2018).\n\n[C] Pan, Z., Liang, Y., Zhang, J., Yi, X., Yu, Y., and Zheng, Y. (2018). Hyperst-net: Hypernetworks for spatio-temporal forecasting. arXiv preprint arXiv:1809.10889\n\n[D]  Zhang, C., Ren, M., and Urtasun, R. (2019). Graph hypernetworks for neural architecture search. In International Conference on Learning Representations.\n \nHowever, there are major concerns about the performance, initialization, and scalability as stated here (Section 6 in the following paper):\nChauhan, Vinod Kumar, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A. Clifton. \"A brief review of hypernetworks in deep learning.\" Artificial Intelligence Review 57, no. 9 (2024): 250.\n\n\n__2-__ Hypernrtwork is not performing well on unseen networks.\nhttps://arxiv.org/abs/2110.13100\nWith that, Could you elaborate on any specific constraints or limitations when transferring MODNAS to devices significantly different from the ones used in the training phase?\n\n\n__3-__ MiLeNAS paper (CVPR’20) shows that the gradient errors caused by approximations of second-order methods in bi-level optimization results in suboptimality, in the sense that the optimization procedure fails to converge to a (locally) optimal solution.  However, it seems that the authors still use a bi-level optimization technique.  It seems using advanced techniques like MiLeNAS might be more useful.\n \n__4-__ Can the MODNSD be applied to object detection tasks as well?\n\n__5-__ MobileNet search space is pretty small DNN. How does it work on more complex DNN search space?  \n\n__6-__ The paper mentions scalability but doesn’t mention potential bottlenecks. For example, are there search space complexities that MODNAS struggles with?\n\n\n\n__7-__ The paper doesn't provide the code. That limits reproducibility of the proposed method.\n\n\n\n__8-__ I am not sure how accurate is the energy model that the paper is using considering even Eyeriss is a pretty old paper (2016).  The authors need to provide more details about energy modeling. \n\n__9-__ I think it is better to use power rather than energy. Considering energy is power x  latency, when you minimize the latency, if the power is fixed, energy is minimized automatically. Considering power, accuracy, and latency can be a better metric.\n\n\n__10-__ The paper employs the Frank-Wolfe solver for optimizing scalarizations. Could you discuss any trade-offs in this choice, and if other optimizers were considered?\n\n__11-__ Given that preference vectors affect the optimization landscape, an analysis of how different preference configurations influence the final architectures would clarify MODNAS’s versatility.\n\n\n__12-__ MODNAS is tested for three objectives—accuracy, latency, and energy. Could this method scale effectively if additional objectives were introduced, or would this necessitate modifications?\n\n__13-__ The approach appears tailored to devices with GPU-like architectures. It would be valuable to understand how MODNAS performs on other architectures, like mobile hardware.\n\n\n\n\n__14-__ Minor: \n\nA)The paper is pretty dense. I hardly can read the legends in Figures 7, 8.\n\nB)Minor Typo:\nPareto front profiling in multi-objective optimization (MOO), i.e. finding a diverse set of Pareto optimal solutions, is challenging\" – A comma after \"i.e.\" would improve readability: \"i.e., finding a diverse set...\"\n\n\"To search across devices, we frame the problem as a multi-task multi-objective optimization problem\" – Add a comma, “multi-task, multi-objective optimization problem.”", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1-The method is adaptable to a range of devices by conditioning the hypernetwork on device embeddings, making it highly versatile for deployment on diverse hardware.\n\n2-MODNAS is tested on various hardware devices and tasks, including image classification, machine translation, and language modeling, showcasing its applicability across multiple domains.", "weaknesses": "1-Using Hypernetworks for NAS is well known but doesn’t seem a promising solution. It is like a heuristic solution.\n\n2- I think energy and latency are not necessarily conflicting metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mAJlhXEdZL", "reviewer": "Reviewer_ugEA", "summary": "This paper proposes a Neural Architecture Search (NAS) algorithm that optimizes several performance metrics on various hardware.\nThe paper adopts Multi Objective Optimization (MOO) to optimize multiple objectives simultaneously.\nThe key idea is to train $\\textbf{MetaHypernetwork}$ which takes hardware features and user preferences about performance metrics as inputs and returns optimized network architectures as outputs.\nTo train $\\textbf{MetaHypernetwork}$, the paper exploits $\\textbf{Supernetwork}$ for NAS and $\\textbf{MetaPredictor}$ for optimization in terms of efficiency.\n\n$\\textbf{MetaHypernetwork}$ takes user preferences and device features, and returns network architecture designs.\nThen, $\\textbf{Architect}$ makes those architecture designs into differentiable architecture parameters.\nWith these architecture parameters, $\\textbf{Supernetwork}$ computes accuracy-related loss of the given network design.\nAlso, $\\textbf{MetaPredictor}$ computes efficiency-related loss with the network architecture under the given hardware features.\nFurther, the paper proposes to update $\\textbf{MetaHypernetwork}$ using Multiple Gradient Descent (MGD) to get optimized network architectures that satisfy multiple objectives concurrently.\n\nFor NAS, the proposed method is adaptable to diverse pretrained $\\textbf{Supernetwork}$.\nFor efficiency, $\\textbf{MetaHypernetwork}$ can be updated with a lot of hardware-related loss functions at once.\nFor optimization, the paper achieves Pareto frontier solutions that are hard to reach with sequential or averaged gradient updates.", "questions": "- How are user preference and hardware device embeddings designed?\n- Please explain how $\\textbf{Architect}$ makes architecture parameters differentiable and propagate them to the $\\textbf{MetaHypernetwork}$.\n- In Figure 4, solutions on the Pareto frontier can be achieved by modulating user preferences. However, in HDX [1], which is a one-shot NAS with hardware constraints, claims that modulating hyperparameters linearly doesn’t lead to linearly distributed results. Is $\\textbf{MetaHypernetwork}$ free from this problem? Can real experimental results be plotted like Figure 4 to substantiate the integrity of $\\textbf{MetaHypernetwork}$?\n- To search network architectures with other works, the paper sets the search time budget up to 2.5 times compared to that of MODNAS. (Or fixed time budget, e.g., 192 hours) That is, MODNAS outperforms prior works in terms of search time, and the quality of solutions is better than that of others under given time budgets. What the review wonders is whether can other works find near-GT solutions with a larger time budget. If a target network is distributed extensively and used frequently, huge search costs can be tolerable if there are better solutions.\n\n[1] Deokki Hong, Kanghyun Choi, Hye Yoon Lee, Joonsang Yu, Noseong Park, Youngsok Kim, and Jinho Lee. Enabling Hard Constraints in Differentiable Neural Network and Accelerator Co-Exploration. In Proceedings of the 59th ACM/IEEE Design Automation Conference, 2022.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes to execute one-shot NAS while search networks satisfy multiple objectives about accuracy and hardware efficiency with target hardware.\n- The paper analyzes the proposed method with various aspects such as efficacy, and robustness of the training process.\n- The paper provides extensive experiments and visualizations to support the proposal and its analysis.", "weaknesses": "- It may be hard to regulate the trade-off among user preferences with scalarization. Figure 4 can be a support, but it is just an abstract depiction, not experimental results.\n- The proposed method can help search optimized network architectures quickly at low cost. However, network architectures the same as or near ground truth solutions may hard to be reach with $\\textbf{MetaHypernetwork}$, where other works can reach with huge search costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FqwaXgKXOK", "reviewer": "Reviewer_VHgP", "summary": "The authors propose a multi-objective HW aware NAS algorith that can emit optimized DNN architectures across multiple different target devices using a single search run.\nTo overcome the challange of having to solve multiple different optimization runs for each considered HW platform, the authors propose using a hypernetwork to generate an architectural distribution across multiple devices based on a \"preference vector\" and a \"hardware device feature vector\" from which discrete architectures can then be sampled.\nFor objective function evaluators, the authors use a Supernetwork as a standin for a accuracy and \"MetaPredictors\", one for each target device, to estimate hardware metrics.", "questions": "- What assumptions do the MetaPredictors make about the underlying devices when estimating latency and energy requirements? For example, do they assume a particular choice of operating system and process scheduler used, other parallel and system processes running, core utilisation of the DNN inference, runtime library, or HW design used to execute the DNN on the target device?\n\n- Especially on smaller systems, memory requirements are often a major bottleneck of DNN inference: Could the approach proposed by the authors be extended to include memory? So far, the evaluation seems to focus only on accuracy, latency and energy.\n\n- How exactly are \"preference vector\" and a \"hardware device feature vector\" defined? The authors cover many different types of HW in their evaluation: GPUs (e.g. 1080 ti), smarthpones (e.g. pixel 2, 3), edge devices (e.g. raspi 4) and dedicated HW accelerators (eyeriss, FPGA), which can be characterised by a number of different metrics, often exclusive to the type of HW considered. (e.g. #cores, processor speed, RAM, ... for the raspi4 vs. #streaming cores, VRAM, bus interface, ... for the 1080 ti vs. #LUTs, #BRAM, ... for the FPGA). So it would be really interesting to know how the authors put these metrics into relation and unified them into one vector.\n\n- Which FPGA did the authors use? Since an FPGA is just freely programmable hardware, it would also be interesting to know which DNN accelerator design the authors implemented on the FPGA to perform their evaluation.\n\n- Fig. 5: Why is MODNAS shown as an upper threashold line (+/- variance)? Based on the caption, I would have expected to see an HVI curve that stops after 24 evaluations, similar to the other curves in the plot.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The extension of existing zero-shot NAS techniques with Hypernetworks for HW aware NAS is motivated well and contextualized nicely for alredy existing techniques. \n- The authors provide an extensive evaluation for different applications (language modeling, vision, translation), multiple well known NAS searchspaces, and different target spaces (2-3 dimensional).", "weaknesses": "- The authors do not show how the proposed DNN architectures would actually perform on the different target systems. As far as I understand it, the HV results shown are calculated using the estimated results from the \"MetaPredictors\". While this still allows for a relative comparision with the other techniques and algorithms evaluated in the paper, it makes it hard to evaluate the actual usefulness and effectiveness of the apporach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9SEXfWqwuv", "reviewer": "Reviewer_KeBz", "summary": "The paper introduces a novel framework for differentiable neural architecture search (NAS) that integrates a generalizable hardware embedding with multi-objective embeddings during supernet training. This approach enables the NAS framework to generalize to unseen hardware platforms and efficiently sample Pareto-optimal subnets based on user preferences. To achieve this, the authors design a MetaHypernetwork that uses a sampled preference vector and representative hardware embeddings to guide architecture sampling within the supernet. During training, instead of randomly sampling subnets, the framework selects and trains subnets based on the hardware and objective embeddings, resulting in a better Pareto front. Extensive experiments demonstrate that this framework consistently produces superior Pareto-optimal subnets across various devices, outperforming several state-of-the-art NAS frameworks.", "questions": "- Could you provide the training and validation loss curves?\n- In Figure 13, do you quantize the probability of $r_m$​ to obtain the vector in the embedding layer $e_m$​? As I understand it, embedding layers typically take indices as inputs.\n- In Figure 13, if I understand correctly, $e_{\\phi_0}$ is a linear layer that maps the device feature vector to a k-dimensional vector. If this is the case, referring to it as an 'embedding' layer may be somewhat confusing, given the use of $e_m$ for embeddings.\n- Suppose an application requires a network with a latency of under 30 ms on a given platform. How does the user preference vector, where each element is a probability within [0, 1], map to specific latency (and energy consumption) in real-world scenarios?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The differentiable NAS framework proposed in the paper introduces a novel hypernetwork that enables zero-shot transferability to new devices, which, to the best of my knowledge, is an original contribution. I appreciate how this framework addresses key challenges faced by the NAS community. Previous approaches directly optimize network structures and hardware metrics using differentiable operators for one specific platform, while two-stage NAS frameworks separate supernet training from subnet searching. However, the search process in two-stage frameworks is often costly, as it relies on hardware-specific predictors and estimators for each objective on each platform. Although some works, such as  [1, 2], have attempted to reduce the search cost in two-stage frameworks, I believe that a unified solution that provides zero-shot transferability and multi-objective optimization across new platforms has been lacking. This paper fills that gap by proposing a unified framework that combines hardware and objective embeddings with differentiable NAS through the use of a hypernetwork. I believe this is an exciting progress for neural architecture search.\n\nThe paper is well-presented and well-organized. The extensive experiments that compare it to several state-of-the art NAS frameworks and the in-depth analysis of the framework fully support the claims made in the paper.\n\n[1] HELP: Hardware-Adaptive Efficient Latency Prediction for NAS via Meta-Learning\n\n[2] BRP-NAS: Prediction-based NAS using GCNs", "weaknesses": "My only concern is that training differentiable NAS with multi-objective hardware metrics can often be unstable and may struggle to converge, leading to issues with reproducibility. While the authors have shared the hyperparameters and plan to release the implementation, it would be beneficial to include a discussion of the training techniques in the Appendix to address these challenges.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)