Multi-objective Differentiable Neural Architecture Search

Authors: rhea  sanjay sukthanker
arber  zela
benedikt  staffler
samuel  dooley
josif  grabocka
frank  hutter
  bcai
abacus  ai
Venue: ICML_cc_2024_Workshop_WANT
Published: 2024-05-23 09:03:12

Abstract: Pareto front profiling in multi-objective optimization (MOO), i.e. finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives like neural network training. Typically, in MOO neural architecture search (NAS), we aim to balance performance and hardware metrics across devices. Prior NAS approaches simplify this task by incorporating hardware constraints into the objective function, but profiling the Pareto front necessitates a computationally expensive search for each constraint. In this work, we propose a novel NAS algorithm that encodes user preferences for the trade-off between performance and hardware metrics, and yields representative and diverse architectures across multiple devices in just one search run. To this end, we parameterize the joint architectural distribution across devices and multiple objectives via a hypernetwork that can be conditioned on hardware features and preference vectors, enabling zero-shot transferability to new devices. Extensive experiments with up to 19 hardware devices and 3 objectives showcase the effectiveness and scalability of our method. Finally, we show that, without extra costs, our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets, including MobileNetV3 on ImageNet-1k, an encoder-decoder transformer space for machine translation and a decoder-only transformer space for language modelling.

Main paper: 
 introduction: 
 The ability to make good tradeoffs between predictive accuracy and efficiency (in terms of latency and/or energy Accepted to the Workshop on Advancing Neural Network Training at International Conference on Machine Learning (WANT@ICML 2024). consumption) has become crucial in an age of ever increasing neural networks complexity and size (Kaplan et al., 2020; Hoffmann et al., 2022; Zhai et al., 2022; Alabdulmohsin et al., 2023) and a plethora of embedded devices. However, finding the right trade-off remains a challenging task that typically requires human intervention and a lot of trial-and-error across devices. With multiple conflicting objectives, it becomes infeasible to optimize all of them simultaneously and return a single solution. Ideally, NAS should empower users to choose from a set of diverse Pareto optimal solutions that represent their preferences regarding the trade-off between objectives.Neural Architecture Search (NAS) (White et al., 2023) provides a principled framework to search for network architectures in an automated fashion. Significant research (Elsken et al., 2019b; Cai et al., 2020; Wang et al., 2020; Chen et al., 2021a) has extended NAS for multi-objective optimization (MOO), considering performance and hardware efficiency metrics like latency and energy consumption. However, to the best of our knowledge, no existing gradient-based method returns the full Pareto front for the MOO problem at hand without repeating their search routine multiple times with different hardware constraints.In this work, we propose a scalable and hardware-aware Multi-Objective Differentiable Neural Architecture Search (MODNAS) algorithm that efficiently trains a single supernet which can be used to read off Pareto-optimal solutions for different user preferences and different target devices, without any extra search steps. To search across devices, we frame the problem at hand as a multi-task multi-objective optimization problem, where each task (device) has multiple (conflicting) objectives, e.g., classification accuracy and latency. The user's preferences are modelled by a preference vector that defines a scalarization of the different objectives. This preference vector, along with features of the hardware of interest, is fed to a hypernetwork (Ha et al., 2017) that outputs continuous architectural parameters α. To search in the space of architectures, we employ a one-shot model and a bi-level optimization scheme, as is typically done in gradient-based NAS. In our case, however, the upper-level parameters are the hypernetwork weights, optimized in expectation across different preference vectors and hardware devices via multiple gradient descent (Désidéri, 2012) .Figure 1 . MODNAS overview. Given a set of T devices, MODNAS seeks to optimize M (potentially conflicting) objectives across these devices. To this end, it employs a MetaHypernetwork HΦ(r, dt), that takes as input a scalarization r, representing the user preferences, and a device embedding dt, to yield an un-normalized architectural distribution α. The Architect uses α to sample differentiable discrete architectures, used in the Supernetwork to estimate accuracy and in the MetaPredictor to estimate the other M -1 loss functions (e.g. latency, energy consumption) for every device. By iterating over devices and sampling scalarizations uniformly from the M -dimensional simplex, at each iteration we update the MetaHypernetwork using multiple gradient descent (MGD).To evaluate our method, we conduct experiments on multiple NAS search spaces, including CNN and Transformer architectures, and up to 3 objectives across 19 hardware devices. While other NAS methods that utilize hardware constraints in their search objectives require substantial search costs both for each new constraint and each new hardware, MODNAS addresses both in a zero-shot manner, without extra search cost, while nevertheless yielding higher quality solutions. Our contributions can be summarized as follows:1. We present a principled and robust approach for Multiobjective Differentiable NAS, that leverages hypernetworks and multiple gradient descent to simultaneously find Pareto-optimal architectures across devices.2. This work is the first to provide a global view of the Pareto solutions with just a single model, without the need to repeat search or fine-tune on new target devices.3. Extensive evaluation of our method across 4 different search spaces (NAS-Bench-201, MobileNetV3, an encoder-decoder and a decoder-only Transformer space), 3 tasks (image classification, machine translation and language modeling), and up to 19 hardware devices and 3 objectives, show both improved efficiency and performance in comparison to previous approaches that use a constrained objective in their search.To facilitate reproducibility, we provide our code via the following link. 
background and related work: 
 In this section, before describing our algorithm, we introduce some basic concepts, definitions and related work. Refer to Appendix B for an extended related work.Multi-objective optimization (MOO) for Multi-Task Learning. Consider a multi-task dataset D consisting of N instances, where the feature vector of the i-th instance is denoted as x i ∈ X , and the M -many associated target variables as y1 i ∈ Y 1 , . . . , y M i ∈ Y M . Moreover, consider there exists a family of parametric models f (x; w) : X → {Y 1 × • • • × Y M }, parameterized by w, that maps the input x to the joint space of the multiple tasks. To simplify the notation, we denote the prediction of the m-th task as f m (x; w) : X → Y m , and the respective loss L m (w) ≜ 1 N N i ℓ m (y m i , f m (x i ; w)). The vector of the values of all loss functions is denoted as L(w) ≜ (L 1 (w), . . . , L M (w)). MOO then seeks to find a set of Pareto-optimal solutions w * that minimize L(w) 1 :EQUATIONDefinition 2.1. (Pareto Optimality): A solution w 2 dominates w 1 iff L m (w 2 ) ≤ L m (w 1 ), ∀m ∈ {1, . . . , M }, and L(w 1 ) ̸ = L(w 2 ). In other words, a dominating solution has a lower loss value on at least one task and no higher loss value on any task. A solution w * is called Pareto optimal iff there exists no other solution dominating w * . Definition 2.2. (Pareto front): The sets of Pareto optimal points and their function values are called Pareto set (P w ) and Pareto front (P L = {L(w) w∈Pw }), respectively.Linear Scalarization. In MOO, a standard technique to solve the M -dimensional problem is using a preference vector r ∈ S ≜ {R M | M m=1 r m = 1, r m ≥ 0, ∀m ∈ {1, . . . , M }} in the M -dimensional probability simplex (Lin et al., 2019; Mahapatra & Rajan, 2020; Ruchte & Grabocka, 2021) . Every r ∈ S yields a convex combination of the loss functions in Equation 1 as L r (w) = r T L(w). Given a preference vector r, one can apply standard, single-objective optimization algorithms to find a minimizer w * r = argmin w L r (w). By sampling multiple r vectors, one can compute Pareto-optimal solutions w * r that profile the Pareto front. Several methods (Lin et al., 2020; Navon et al., 2021; Hoang et al., 2023; Phan et al., 2022) employ a hypernetwork (Ha et al., 2017) to generate Pareto-optimal solutions given different preference vectors as input. Similarly, in this work, we utilize a hypernetwork conditioned on scalarizations, to generate architectures.Multiple Gradient Descent (MGD). MOO can be solved to local optimality via MGD (Désidéri, 2012) , as a natural extension of single-objective gradient descent, which iteratively updates w towards a direction that ensures that all tasks improve simultaneously (called Pareto improvement): w ′ ← w -ξg * w , where g * w is a vector field that needs to be determined. If we denote by g m w = ∇ w L m (w) the gradient of the m-th scalar loss function, via Taylor approximation, the decreasing direction of L m when we update w towards g * w is given by ⟨g m w , g * w ⟩ ≈ -(L m (w ′ ) -L m (w))/ξ. In MGD g * w is chosen to maximize the slowest update rate among all objectives:g * w ∝ argmax gw∈R d ,||gw||≤1 min m∈[M ]⟨g w , g m w ⟩ .(2)The early work of Désidéri ( 2012) has been extended in various settings, particularly multi-task learning, with great promise (Sener & Koltun, 2018; Lin et al., 2019; Mahapatra & Rajan, 2020; Liu & Vicente, 2021) , but these approaches are applied to mainly a fixed architecture and extending them to a search space of architectures is non-trivial.One-shot NAS and Bi-Level optimization. With the architecture space being intrinsically discrete, large and hence expensive to search on, most existing differentiable NAS approaches leverage the weight sharing paradigm and continuous relaxation to enable gradient descent (Liu et al., 2019; Pham et al., 2018; Bender et al., 2018; Xie et al., 2019; Xu et al., 2020a; Dong & Yang, 2019; Chen et al., 2021b; Liu et al., 2023; Movahedi et al., 2022; Zhang et al., 2021) . Typically, in these approaches, architectures are stacks of cells, where the cell structure is represented as a directed acyclic graph (DAG) with N nodes and E edges.Every transition from node i to j, i.e. edge (i, j), is associated with an operation o (i,j) ∈ O, where O is a predefined candidate operation set. Liu et al. (2019) proposed a continuous relaxation of the search space by parameterizing the discrete operation choices in the DAG edges via a learnable vector α. This allows to frame the NAS problem as a bilevel optimization one, with differentiable objectives w.r.t. all variables:EQUATIONwhere L train and L val are the empirical losses on the training and validation data, respectively, w are the supernetwork parameters, α ∈ A are the continuous architectural parameters, and w * (α) : A → R d is a best response function that maps architectures to their optimal weights.Comparison to single-objective constrained NAS. Early NAS methods predominantly targeted high accuracy, whereas contemporary hardware-aware differentiable NAS approaches (Wu et al., 2019; Wan et al., 2020; Cai et al., 2018; Wu et al., 2021; Fu et al., 2020; Xu et al., 2020b; Jiang et al., 2021; Wang et al., 2021) are designed to identify architectures optimized for target hardware efficiency. Typically, these methods integrate hardware constraints within their objectives, yielding a single optimal solution and necessitating multiple search iterations to construct the Pareto front. Our proposed algorithm addresses this by profiling the entire Pareto front in a single search iteration. While single-objective constrained optimization is advantageous in scenarios demanding optimization of one objective under a specific constraint, practical applications often require a suite of models adaptable to varying user preferences even on a single device. For example, meeting stringent hardware or task requirements (e.g., memory usage, real-time inference) while elucidating the trade-offs (i.e., Pareto front) between precision and performance is critical, as substantial runtime reductions with minimal performance degradation may be acceptable or mitigated by other components. 
hardware-aware multi-objective differentiable neural architecture search: 
 We first formalize the multi-objective bi-level optimization NAS problem across multiple hardware devices, and then introduce a scalable and differentiable method that combines MGD with linear scalarizations to efficiently solve this problem.In multi-objective NAS, the bi-level problem described in Equation 1 becomes more difficult, since we are not only concerned with finding w * given a fixed architecture, but we want to optimize in the space of architectures A as well.Assuming we have T hardware devices (target functions) and M objectives (e.g. accuracy, latency, etc.), similar to (3), for every t ∈ {1 . . . T }, the Pareto set of the multi-objective NAS problem is obtained by solving the following bi-level optimization problem:EQUATIONwhere the M -dimensional loss vector L t (w* (α), α) ≜ L 1 t (w * (α), α), . . . , L M t (w * (α), α) is evaluated ∀t ∈ Figure 2.Architecture overview of the MetaHypernetwork, which gets as input a device embedding dt (input to an embedding layer E) and a scalarization r (input to K hypernetworks) and yields an architecture encoding α.{1, . . . , T }. L train t and L valid t are the vectors with all M loss functions evaluated on the train and validation splits of D, used in the lower-and upper-level problems of (4), respectively.Our goal is to find Pareto-optimal architectures for each target device, covering diverse and representative preferences for different objectives. However, naively solving (4) for each device t requires T independent bi-level searches.To address this, we integrate a single MetaHypernetwork within the one-shot model (supernetwork) used in conventional NAS (Bender et al., 2018; Pham et al., 2018; Liu et al., 2019) , generating architectures based on device embeddings and preference vectors in a single search run.Our search model is composed of four modular components shown in Figure 1: (1) a MetaHypernetwork that generates the architectural distribution; (2) an Architect that samples discrete architectures from this distribution; (3) a Supernetwork that exploits the weight sharing paradigm for search efficiency and provides a proxy for the network accuracy; and (4) a MetaPredictor that predicts hardware metrics and enables gradient propagation. We now discuss each of these in turn.In order to generate architectures across multiple devices, inspired by Wang et al. (2022) and Lin et al. (2020) , we propose a MetaHypernetwork that can meta-learn across different hardware devices (see Figure 2 ). Hypernetworks are a class of neural networks that generate the parameters of another model. They were initially proposed for model compression (Ha et al., 2017) and were later adopted for NAS (Brock et al., 2018) and MOO (Navon et al., 2021; Lin et al., 2020) . Here, given a preference vector r = (r 1 , . . . , r M ) and a hardware de-vice feature vector d t , for device t ∈ {1, . . . , T }, we use the MetaHypernetwork H Φ (r, d t ), parameterized by Φ, to generate an un-normalized architecture distribution αΦ that is later used to compute the upper-level updates in (4). Similar to Lee et al. (2021b) , d t is a fixed-size feature vector that is obtained by evaluating a fixed set of reference architectures on device t. The MetaHypernetwork is composed of 2 main components (see Figure 2 ):1. A bank hpn 1 ,. . . ,hpn K of K independent hypernetworks, that parse the preference vector r and generate the architectural parameters α1 , . . . , αK , respectively.2. An embedding layer E, that learns a similarity map from device feature vectors to the bank of hpns. E takes as input the device feature vector d t and outputs an attention vector of size K.The final output, αΦ , of the MetaHypernetwork is computed as a weighted sum of the outputs of the K hypernetworks, where the vector of weights is the output of the embedding layer E. For a more detailed description of the MetaHypernetwork we refer the reader to Appendix F.2.In all experiments, we initialize the MetaHypernetwork to yield a uniform probability mass over all architectural parameters for all scalarizations and device embeddings. By using the preference vector r to create a linear scalarization of L t and the MetaHypernetwork to model the architectural distribution across T devices, the bi-level problem in (4) reduces to:EQUATIONwhere α Φ are the normalized architectural parameters obtained from the Architect Λ( αΦ ) and rT L t (•, α Φ ) = M m=1 r m L m t (•, α Φ )is the scalarized loss for device t. Conditioning the MetaHypernetwork on the hardware embeddings allows us to generate architectures on new test devices without extra finetuning or meta-learning steps. We use the Dirichlet distribution Dir(β), β = (β 1 , . . . , β M ), to sample the preference vectors and approximate the expectation over the scalarizations using Monte Carlo sampling. In our experiments we set β 1 = • • • = β M = 1, for a uniform sampling over the (M -1)-simplex, however, one can set these differently based on user priors or make it a learnable parameter (Chen et al., 2021b) .MetaPredictor. For the cheap-to-evaluate hardware objectives, such as latency, energy consumption, we employ a regression model p m θ (α, d m t ) that predicts the target labels y m t for objective m and device t, given an architecture α and device embedding d m t . We use the same Φ (r, d t ) 5 Sample α Φ ∼ Λ( αΦ ) from Architect 6 g t Φ ← M m=1 r m ∇ Φ L m t (D valid ; w, α Φ ) 7 γ ← FrankWolfeSolver(g 1 Φ , . . . , g T Φ ) ; // see Alg.4 g * Φ ← T t=1 γ t • g t Φ 9 Φ ← Φ -ξ 1 • g * Φ ;// update MetaHypernetwork10 for t ∈ {1, . . . , T } do 11 Sample scalarization r ∼ Dir(β) 12 Set arch params αΦ ← H Φ (r, d t ) 13 Sample α Φ ∼ Λ( αΦ ) from Architect 14 g t w ← M m=1 r m ∇ w L m t (D train ; w, α Φ ) 15 g * w ← 1 T T t=1 g t w 16 w ← w -ξ 2 • g * w ;// update Supernetwork 17 return H Φ predictors as Lee et al. (2021b) and optimize the MSE loss:min θ E α∼A,t∼[T ] y m t -p m θ (α, d m t )2 , as done in Lee et al. (2021a) for meta-learning performance metrics across datasets. In our experiments, we pretrain a separate MetaPredictor for every hardware objective m (e.g. latency, energy, etc.) on a subset of (α, y m t ) pairs, and use its predicted value directly in (5) asL m t (•, α Φ ) = p m θ (α Φ , d m t ).During search, we freeze and do not update further the MetaPredictor parameters θ.Supernetwork. For expensive objectives like neural network classification accuracy, we use a Supernetwork that encodes the architecture space and shares parameters between architectures, providing a best response function w * (α Φ ) for the scalarized loss in (5). While any parametric model could estimate this function, such as performance predictors (Lee et al., 2021a) , this requires an expensive prior step of creating the training dataset for the predictor. To reduce memory costs of Supernetwork training, we: (1) use a one-hot encoding of α Φ for differentiable architecture sampling (Dong & Yang, 2019; Cai et al., 2018; Xie et al., 2019) , activating only one architecture per step, and (2) entangle operation choice parameters in the Supernetwork, further increasing memory efficiency beyond weight sharing (Sukthanker et al., 2023) .Architect. The Architect Λ(α) samples discrete architectural configurations from the un-normalized distribution αΦ = H Φ (r, d t ) and enables gradient estimation through discrete variables for ∇ Φ L t (•, α Φ ). Methods such as GDAS (Dong & Yang, 2019) utilize the Straight-Through Gumbel-Softmax (STGS) estimator (Jang et al., 2017) , that integrates the Gumbel reparameterization trick to approximate the gradient. Here we employ the recently proposed ReinMax estimator (Liu et al., 2023) , that yields secondorder accuracy without the need to compute second-order derivatives. See Appendix C.1 for more details on these discrete samplers. Similar to the findings in Liu et al. (2023) , in our initial experiments, ReinMax outperformed the GDAS STGS estimator (see Figure 11 in the Appendix), therefore, we use ReinMax in all following experiments.We denote the gradient of the scalarized loss in (5) with respect to the MetaHypernetwork parameters Φ, shared across all devices t ∈ 1, . . . , T , as:g t Φ = r T ∇ Φ L t (•, α Φ ) = M m=1 r m ∇ Φ L m t (•, α Φ ), where α Φ is the discrete architectural sample from the Architect Λ( αΦ). Multiple Gradient Descent (MGD) (Désidéri, 2012; Sener & Koltun, 2018) provides a plausible approach to estimate the update directions for every task simultaneously by maximizing (2). Via the Lagrangian duality, the optimal solution to equation 2 isg * Φ ∝ T t=1 γ * t g t Φ , where {γ * t } T t=1is the solution of the following minimization problem:min γ1,...,γ T T t=1 γ t g t Φ 2 2 T t=1 γ t = 1, γ t ≥ 0, ∀tThe solution to this problem is either 0 or, given a small step size ξ, a descent direction that monotonically decreases all objectives at the same time and terminates when it finds a Pareto stationary point, i.e. g t Φ = 0, ∀t ∈ {1, . . . , T }. When T = 2, the problem above simplifies to min γ∈[0,1] γg 1 Φ + (1 -γ)g 2 Φ 22 , which is a quadratic function of γ with a closed form solution:γ * = max min (g 2 Φ -g 1 Φ ) T g 2 Φ ∥g 1 Φ -g 2 Φ ∥ 2 2, 1 , 0 .When T > 2, we utilize the Frank-Wolfe solver (Jaggi, 2013) as in Sener & Koltun (2018) , where the analytical solution in for T = 2 is used inside the line search. We provide the full algorithm for computing γ * in Algorithm 4 in Appendix C.2.In Algorithm 1 and Figure 1 we provide the pseudocode and an illustration of the overall search phase of MODNAS. generate the un-normalized architectural distribution αΦ (lines 3-4). We then compute the device-specific gradient in line 6 which is used to estimate the γ coefficients (line 7) used from MGD to update Φ (lines 8-9). Similarly to Liu et al. (2019) , we use the first-order approximation to obtain the best response function in the lower level (lines 10-14) and repeat the same procedure for the upper-level (lines 2-6), except now the Supernetwork weights w are updated with the mean gradient (line 15), over devices. 
experiments: 
 In this section, we firstly demonstrate the scalability and generalizability of our MODNAS approach on a NAS tabular benchmark (Section 4.1). To assess the quality of the Pareto set solutions, we use the hypervolume (HV) indicator, which is a standard metric in MOO. Given a reference point ρ = [ρ 1 , . . . , ρ m ] ∈ R M + that is an upper bound for all objectives {f m (•; w, α)} M m=1 , i.e. sup α f m (•; w, α) ≤ ρ m , ∀m ∈ [M ], and a Pareto set P α ⊂ A, HV(P α ) measures the region of non-dominated points bounded above from ρ:λ q ∈ R M + | ∃α ∈ P α : q ∈ M m=1 [f m (•; w, α), ρ m ] ,where λ(•) is the Euclidean volume. HV can be interpreted as the total volume of the union of the boxes created by the Pareto front.We firstly validate the scalability and learning capability of MODNAS by evaluating on the NAS-Bench-201 cell-based convolutional space. Here we want to optimize both latency and classification accuracy on all devices. We utilize the same set of 19 heterogeneous devices as Lee et al. (2021b) , from which we use 13 for search and 6 at test time. For the latency predictor, we use the one from HELP, namely a graph convolutional network (GCN), which we pretrain for 3 GPU hours on the ground truth latencies on the 13 search devices as described in Section 3. We run the MODNAS search (see Appendix F for more details on the search hyperparameters), as described in Algorithm 1, for 100 epochs (22 GPU hours on a single NVidia RTX2080Ti) and show the HV in Figure 3 of the evaluated Pareto front in comparison to the baselines, for which we allocated the same search time budget across all devices equivalent to the MODNAS search + evaluation.Most notably, MODNAS consistently outperforms all other baselines across every device. For the baselines, we conducted 19 separate search runs (one for each device), whereas MODNAS leverages meta-learning to generate the Pareto set on each device using the same MetaHypernetwork in a single search run. Interestingly, the trained MODNAS attention-based MetaHypernetwork significantly outperforms the RHPN baseline in profiling the Pareto front, demonstrating its effectiveness in optimizing across multiple devices and conflicting objectives simultaneously. In Figure 14a in the Appendix, we compare MODNAS with additional baselines, running them at double the budget used for the experiments in Figure 3 . Figure 4 (left) (see Figure 17 in the appendix for all devices) shows that most baselines require more than twice the number of architecture evaluations to reach the same HV as MODNAS.Results show that MODNAS remains the top performer across hardware devices on average. Furthermore, Figure 14 presents radar plots for four additional metrics.MetaHypernetwork update schemes: robustness of MGD. We now compare the MGD update scheme for the MetaHypernetwork Φ (line 9 in Alg. 1) against (1) the mean gradient over tasks:Φ ← Φ -ξ 1 T T t=1 g t Φ ;(2) sequential updates with all single tasks' gradients:Φ ← Φ -ξg t Φ , ∀t;(3) single updates using gradients of MC samples over tasks: Φ ← Φ -ξg t Φ , t ∼ {1, . . . T }. Figure 4 (right) (see Figure 18 in Appendix J for more results) shows the HV over search epochs for these schemes. MGD, by accounting for inter-task dependencies, achieves higher final HV, better anytime performance, and faster convergence than the other schemes.Scalability to three objectives. We now demonstrate the scalability of MODNAS to 3 objectives, namely, accuracy, latency and energy consumption. For this experiment we use the FPGA and Eyeriss tabular energy usage values from HW-NAS-Bench (Li et al., 2021) . In addition to the MetaPredictor for latency, we pretrain a second predictor on the energy usage objective. We then run MODNAS and the MOO baselines with the same exact settings as for 2 objectives. Results shown in Figure 6 indicate that MOD-NAS can scale to M > 2 without additional search costs or hyperparameter tuning and yet achieves HV close to the G lo b a l o p t. M O D N A S R S M O -A S H A N S G A -I I R H P N q E H V I M O -R E R S -B O L S -To compare against single-objective NAS with hardware constraints in the objective, we run MetaD2A+HELP (Lee et al., 2021b) . Since MetaD2A + HELP is not able to profile the Pareto front directly, we run the NAS search 24 times with different constraints, which we compute by denormalizing the same 24 equidistant preference vectors we use to evaluate MODNAS. We also extend MODNAS to incorporate user prior constraints over the multiple objectives being optimized during search Namely, we add a normalized constraint c m , such that if the predicted value from the MetaPredictor during search satisfies this constraint, i.e. p m θ (α Φ , d m t ) ≤ c m , we remove the gradient w.r.t. to that objective in lines 6 and 14 of Algorithm 1. In Figure 5 (other devices in Figure 15 ) we can see that when increasing the latency constraint to 1 (only cross-entropy optimized), though the HV decreases, MODNAS returns Pareto sets with more performant architectures. MetaD2A+HELP, though conducting multiple search runs per constraint, focuses more on highly performant architectures, and is not able to return a diverse solution set.To demonstrate its effectiveness beyond image classification and CNN spaces, we apply MODNAS to the hardware- aware Transformer (HAT) search space from Wang et al.M O D N A S q E H V I N S G A -I I M O -A S H A L S M O -R E R S R H P N L S -B O H A T R S -(2020) on the WMT'14 En-De (Jean et al., 2015; Macháček & Bojar, 2014 ) machine translation task. We pretrained the MetaPredictor (details in Appendix F.1) for 5 GPU hours on 2000 architecture samples from the search space and then conducted the search for 110 epochs (6 days on 8 NVIDIA RTX A6000 GPUs) using 2 search devices, adhering to the same hyperparameters as Wang et al. (2020) to optimize for latency and validation cross entropy loss.Each baseline was allocated 2.5× more runtime budget than MODNAS, resulting in 1300 (RS-BO) to 6000 (MO-ASHA) total architecture evaluations, whereas MODNAS evaluated only 24 generated architectures. Details on the HAT search space and search hyperparameters are in Appendix G. We evaluated MODNAS on all 3 devices (2 search and 1 test) using the BLEU score, and results in Figure 7 show that MODNAS outperforms all baselines, achieving a higher hypervolume (left plot) of the generated Pareto fronts (right plot). For HAT, we evaluated the architectures provided in their paper. Additional results on other training devices and evaluation metrics are presented in Figures 20, 21 and 22 in the Appendix.Image Classification on ImageNet-1k. We now validate MODNAS on ImageNet-1k using the MovileNetV3 search space from Once-for-All (OFA) (Cai et al., 2020) . For this experiment, we run MODNAS using 11 search (and 1 test) devices starting with the pretrained OFA supernetwork and run the search further for 1 day on 8 RTX2080Ti GPUs.During the search, we only update the MetaHypernetwork weights and keep the pretrained Supernetwork weights frozen. Details on the search space and hyperparameters are in Appendices G and F.3. We use the simple MLP from Lee et al. (2021b) as our MetaPredictor, pretraining it for 6 hours on 5000 sampled architecture-latency pairs. To evaluate the 24 points generated by our MetaHypernetwork and baselines, we use the OFA pretrained Supernetwork.Results in Figure 8 (left) show that MODNAS achieves a higher average HV across all devices compared to baselines, which we ran for 192 hours using the OFA pretrained accuracy predictor (see Figure 25 for all results and Figure 24 for the Pareto fronts).M O D N A S L S -B O M O -A S H A R S -B O q E H V I M O -R E N S G A -I I R SLanguage Modeling with GPT-2. With the rapid growth of language model sizes, it is crucial to identify transformer variants that are efficient during inference (latency) while maintaining competitive performance. We applied MOD-NAS to the GPT-S space from HW-GPT-Bench (Sukthanker et al., 2024) , which features a non-convex Pareto front between perplexity and hardware metric objectives. Using pretrained Supernetwork weights from HW-GPT-Bench, we conducted a single 6-hour search on 4 Nvidia A100 GPUs, optimizing for energy consumption (Wh) and perplexity across 8 different GPU devices. See Appendix F for details on the MetaHypernetwork architecture and other search hyperparameters. The Supernetwork weights were kept frozen while updating the MetaHypernetwork. Figure 8 (right) shows that, within the same time budget, MODNAS matches or surpasses other MOO baselines, demonstrating its effectiveness in optimizing beyond convex Pareto fronts. 
conclusion: 
 In this paper, we propose a novel hardware-aware differentiable NAS algorithm for profiling the Pareto front in multiobjective problems. In contrast to constraint-based NAS methods, ours can generate Pareto optimal architectures across multiple devices with a single hypernetwork that is conditioned on preference vectors encoding the trade-off between objectives. Experiments across various hardware devices (up to 19), objectives (accuracy, latency and energy usage), search spaces (CNNs and Transformers), and applications (classification, machine translation, language modeling) demonstrate the effectiveness and efficiency of our method. Brock, A., Lim, T., Ritchie, J., and Weston, N.  
 
 Appendix: This work focuses on an area, neural architecture search (NAS), which has, rightfully, received criticism for its energy consumption. NAS techniques, particularly black-box techniques, require significant computational resources to train the sampled neural network architectures. Speeding up the search and training cost for different neural architectures is an important aspect of responsible research in NAS. The main goal of this work is to improve the search time in NAS, as well as the efficiency of the found architectures in terms of various hardware metrics, such as latency, energy usage, therefore reducing the energy consumption and CO 2 footprint.Specifically, this paper presents a more efficient optimization procedure than previously proposed methods and achieves comparable or better solutions. Additionally, the more hardware-efficient architectures we find, correspond to more efficient neural networks, e.g. in terms of energy consumption. The energy savings of these architectures will be amplified as they might be deployed on a large number of devices. Ultimately, our method leads to decreased energy consumption during product development, as well as during the usage of the product.Multi-objective optimization. Multi-objective optimization (MOO) (Gunantara, 2018) ). However, these methods are primarily applied to fixed architectures, and adapting them to architecture search spaces is complex. This adaptation would require retraining each architecture with multiple objectives, which is impractically expensive for large search spaces. Another major challenge in MOO is balancing the different objectives. To address this, preference vectors have been proposed to guide the prioritization of objectives on the Pareto Front (Ye & Liu, 2022; Momma et al., 2022 ). An emerging approach to mitigate the retraining issue involves hypernetworks, which determine the weights of the main network in MOO scenarios (Lin et al., 2020) , often incorporating preference vector (Navon et al., 2021; Hoang et al., 2023; Phan et al., 2022) .Neural Architecture Search. A major challenge in the automated design of neural network architectures is the efficient exploration of vast search spaces. Early NAS methods relied on Reinforcement Learning (Zoph & Le, 2017) , evolutionary algorithms (Deb et al., 2002; Lu et al., 2020; Elsken et al., 2019b) , and other black-box optimization techniques (Daulton et al., 2022) to train and evaluate numerous architectures from scratch. The advent of one-shot NAS introduced weight sharing among architectures by training an over-parameterized network, known as a supernet, to expedite the evaluation of individual networks within the search space (Saxena & Verbeek, 2016; Bender et al., 2018; Pham et al., 2018; Liu et al., 2019) . Differentiable one-shot NAS methods (Wu et al., 2019; Cai et al., 2018; Wu et al., 2021; Fu et al., 2020) further improved efficiency by applying a continuous relaxation to the search space, enabling the use of gradient descent to identify optimal sub-models within the supernet. In contrast, two-stage NAS methods initially train a supernet, often through random sampling of subnetworks, and subsequently employ black-box optimization to identify optimal subnetworks (Bender et al., 2018; Li & Talwalkar, 2020; Guo et al., 2020) .Hardware-aware and Multi-objective Neural Architecture Search. Early NAS methods primarily focused on maximizing accuracy for a given task. In contrast, hardware-aware NAS aims to optimize architectures for efficient performance on specific hardware devices (Benmeziane et al., 2021; Zhang et al., 2020; Lee et al., 2020; Shaw et al., 2019) , naturally leading to multi-objective NAS (Hsu et al., 2018; Kim et al., 2021; Tan et al., 2019 (Dudziak et al., 2020) to evaluate hardware performance, combining task and hardware objectives with fixed weighting to find a single optimal solution. However, changing the objective weighting requires a complete search rerun, which is computationally demanding.In contrast, our proposed search algorithm offers the entire Pareto Front of objectives in a single run, making it more efficient. While our focus is on multi-objective NAS for hardware constraints, our technique is applicable to other objectives such as fairness (Martinez et al., 2020; Dooley et al., 2023; Das & Dooley, 2023) , suggesting promising avenues for future research.In this section, we provide the pseudocodes for some of the algorithmic components we use in MODNAS.Given the architecture parameters αΦ from the MetaHypernetwork, we obtain a differentiable discrete architecture sample from the Architect as α Φ ← π -stop_g(π) + α Φ , where α Φ ∼ Cat softmax 1 (α Φ ) andπ ← 2 • softmax 1 stop_g ln( α Φ + softmax τ (α Φ ) 2 ) -αΦ + αΦ - softmax 1 (α Φ ) 2 .Here, Cat is the categorical distribution, τ is the temperature in the tempered softmax softmax τ (α) i = exp(αi/τ ) |O| j=1 exp(αj /τ ), and stop_g(•) duplicates its input and detaches it from backpropagation. Refer to the ReinMax paper (Liu et al., 2023) for more details. The algorithm pseudocode on how a one-hot encoded (discrete) architecture is sampled given an architectural unnormalized distribution α is given in Algorithm 2 and Algorithm 3, for the Straight-Through (Jang et al., 2017) and ReinMax (Liu et al., 2023) gradient estimators, respectively.Algorithm 2: Straight -Through (Jang et al., 2017) Data: α: softmax input, τ : temperature Result: α: one-hot samples1 π 0 ← softmax 1 (α) 2 α ∼ Cat(π 0 ) 3 π 1 ← softmax τ (α) 4 α ← π 1 -stop_g(π 1 ) + α 5 return αAlgorithm 3: ReinMax (Liu et al., 2023) Data: α: softmax input, τ : temperature Result: α: one-hot samples1 π 0 ← softmax 1 (α) 2 α ∼ Cat(π 0 ) 3 π 1 ← α+softmaxτ ( α) 2 4 π 1 ← softmax 1 stop_g ln(π 1 ) -α + α 5 π 2 = 2 • π 1 -1 2 • π 0 6 α ← π 2 -stop_g(π 2 ) + α 7 return α C.2. Frank-Wolfe SolverIn this section, we provide the pseudocode of the Frank-Wolfe solver (Jaggi, 2013) used to compute the gradient coefficients used for the MGD updates. To solve the constrained optimization problem, the Frank-Wolfe solver uses analytical solution for the line search with T = 2 (Algorithm 5).Algorithm 4: FrankWolfeSolver (Jaggi, 2013)Data: g 1 Φ , . . . , g T Φ Result: γ = (γ 1 , . . . , γ T ) 1 Initialize γ ← ( 1 T , . . . , 1 T ) 2 Precompute M s.t. M i,j = (g i Φ ) T (g j Φ ) 3 repeat 4 t ← argmin r T t=1 γ t M rt 5 e t ← M t,• ; // t-th row of M 6 δ ← argmin δ (1 -δ)γ + δe t T M (1 -δ)γ + δe t ;// using Algorithm 57 γ ← (1 -δ)γ + δe t 8 until δ ∼ 0 or Number of Iterations Limit; 9 return γ Algorithm 5: Solver min δ∈[0,1] ||δθ + (1 -δ) θ|| 2 2 1 if θ T θ ≥ θ T θ then 2 δ ← 1 3 else if θ T θ ≥ θT θ then 4 δ ← 0 5 else 6 δ ← ( θ-θ) T θ ||θ-θ|| 2 2 7 return δThis section elaborates on the multi-objective NAS methods we utilize as baselines in Section 4.• Random Search (RS) is a robust baseline for both single-objective (Bergstra & Bengio, 2012; Li & Talwalkar, 2020) and multi-objective (Cai et al., 2020; Chen et al., 2021a) architecture searches. This baseline involves randomly sampling architectures from the search space and computing the Pareto front from these samples. While RS is computationally efficient and often effective, it may not always find the optimal architectures, especially in larger search spaces.• Local Search (LS) is adapted to refine solutions near Pareto-optimal points in multi-objective optimization, iteratively improving solutions within defined neighborhoods.• Multi-objective Asynchronous Successive Halving (MO-ASHA) (Schmucker et al., 2021 ) is a multi-fidelity method that utilizes an asynchronous successive halving scheduler (Li et al., 2018) and non-dominating sorting for budget allocation. MO-ASHA uses the NSGA-II selection mechanism and the ϵ-net (Salinas et al., 2021) exploration strategy that ranks candidates in the same Pareto set by iteratively selecting the one with the largest Euclidian distance from the previous set of candidates.• Multi-Objective Regularized Evolution (MO-RE) builds on Regularized Evolution (RE) (Real et al., 2019) , which evolves a population of candidates through mutation and periodically removes the oldest individuals, thus regularizing the population. MO-RE adapts this by using multi-objective non-dominated sorting to score candidates, with parents sampled based on these scores.• Non-dominated Sorting Genetic Algorithm II (NSGA-II) (Deb et al., 2002) is a multi-objective evolutionary algorithm designed to find a Pareto set of architectures. It ranks architectures using non-dominated sorting and maintains diversity with crowding distance. Through selection, crossover, and mutation, NSGA-II evolves populations towards the Pareto front, although it is known for being sample inefficient.• Covariance Matrix Adaptation Evolution Strategy (CMA-ES) (Igel et al., 2007) is an evolutionary algorithm particularly effective in continuous optimization problems. In a multi-objective context, it adapts its covariance matrix to the shape of the search space, iteratively updating its sampling distribution to favor promising regions. This method efficiently handles complex, non-linear optimization landscapes and can be adapted to multi-objective scenarios by using techniques such as Pareto-based selection to maintain a diverse set of solutions.• Latent Action MOO (LaMOO) (Zhao et al., 2022 ) uses a parametric model and Monte Carlo Tree Search (MCTS) to learn to partition the objective space based on the dominance number, which indicates the vicinity of a point to the Pareto front relative to the other samples. qEHVI+LaMOO and CMA-ES+LaMOO use the original qEHVI and CMA-ES, respectively, as an inner routine in the learned subspaces.• Bayesian Optimization with Random Scalarizations (RS-BO) (Paria et al., 2020) uses an acquisition function based on random linear scalarizations of objectives across multiple points to find the Pareto-optimal set that minimizes Bayesian regret.• Bayesian Optimization with Linear Scalarizations (LS-BO) is similar to RS-BO but optimizes a single objective derived from a fixed linear combination of two objectives instead of using randomized linear scalarizations.• Expected Hypervolume Improvement (qEHVI) (Daulton et al., 2020) is a Bayesian optimization acquisition function that explores the Pareto front by quantifying potential hypervolume improvement. This approach measures the volume dominated by Pareto-optimal solutions and guides the search towards regions likely to offer better trade-offs, aiding in the discovery of diverse Pareto-optimal solutions.For NAS-Bench-201, in addition, we evaluate the generational distance (GD) and inverse generational distance (IGD) (see Appendix E). See Figure 14 for the results complementary to the hypervolume radar plot in Figure 3 of the main paper.Generational Distance (GD) and Inverse Generational Distance (IGD). Given a reference set S ⊂ A and a Pareto set P α ⊂ A with dim(A) = K, the GD indicator is defined as the distance between every point α ∈ P α and the closest point in s ∈ S, averaged over the size of P α :GD(P α , S) = 1 |P α | α∈Pα min s∈S d(α, s) 2 1/2 , where d(α, s) = K k=1 (α k -s k ) 2 is the Euclidean distance from α to its nearest reference point in S.The inverted generational distance (IGD) is computed as IGD(P α , S) = GD(S, P α ).Generational Distance Plus (GD + ) and Inverse Generational Distance Plus (IGD + ). GD + (P α , S) = IGD + (S, P α ) replaces the euclidean distance d(α, s) in GD with:d + (α, s) = K k=1 (max{α k -s k , 0}) 2On the NAS-Bench-201 search space, since the architectures evaluated with the supernetwork weights are not highly correlated to the ones trained independently from scratch, we employ the Sum of Training Losses (SoTL) proxy from Ru et al. (2021) . To profile the Pareto front with SoTL, we firstly evaluate the 24 architectures using the exponential moving average of the sum of training losses for the initial 12 epochs of training as 12 e=1 0.9 12-e L train (w, α), and then train from scratch only the subset of architectures in the Pareto set built using the SoTL evaluations. We present the results of MODNAS-SoTL in Figure 14 , where we compare to the other baselines as well. As we see, we can further decrease the evaluation cost via MODNAS-SoTL, by trading off the number of solutions in the Pareto set with HV.For all search spaces we set the dimensionality of the hardware embedding to 10. This corresponds to latency evaluations on a set of 10 reference architectures, which are the same used by Lee et al. (2021b) .NAS-Bench-201. For the NAS-Bench-201 (Dong & Yang, 2020) search space we use a Graph Convolutional Network (GCN) as proposed in Dudziak et al. (2020) . Furthermore, in addition to the one-hot operation encoding and adjacency matrix corresponding to the architecture cells, we also input the hardware embedding to this predictor, as done by Lee et al. (2021b) . The number of nodes in the GCN is 8 and the dimensionality of the layers is set to 100 following HELP (Lee et al., 2021b) .MobileNetV3 (OFA). Following HELP (Lee et al., 2021b) , we employ a simple feedforward neural network in the MobileNetV3 search space. The input dimension of the MetaPredictor is set to 160, matching the concatenated architecture encoding dimension. We set the size of the hidden layers to 100. Specifically, the MetaPredictor comprises 2 linear layers with ReLU activation for processing the 160-dimensional one-hot architecture encoding and 2 linear layers for processing the hardware embedding. The outputs from these two paths are concatenated and passed through a final linear layer to predict the latency.Figure 9 . MetaHypernetwork architecture overview in the case of M objectives. Note that m = 1 is reserved for the accuracy objective, which we model through the cross-entropy loss in the Supernetwork. The initial embedding layer e ϕ 0 gets the dt hardware embedding and outputs a weight that scales each of the K hypernetworks' (orange boxes) outputs from the hypernetwork bank. The scaled architectural parameters are then summed up element-wise. All individual hypernetwork h ϕ k get as input the same scalarization r. Each of them has M -1 embedding layers with dimensions nm × dim(A) M -1 , ∀m ∈ {2, . . . , M } that gets as input the scalarizations for objectives m = 2, . . . , m = M , and yields a vector of size dim(A) M -1 . The output from the M -1 embedding layers are concatenated to give the architecture encoding α.Seq-Seq Transformer (HAT). HELP3 does not release the architecture or the meta-learned pretrained predictor for HAT (Wang et al., 2020) . However, HAT4 releases code and pretrained models for each of the devices and tasks trained independently. Hence, we build our single per-task MetaPredictor based on the architecture of the HAT predictor, i.e. a simple feedforward neural network. The input dimension corresponds to the onehot architecture encoding of the candidate Transformer architecture. Additionally, to condition on the hardware embedding, we include 2 extra linear layers for processing the hardware embedding, which is then concatenated with the processed architecture encoding to produce the final latency prediction. The hidden dimension of the MetaHypernetwork is set to 400, with 6 hidden layers. The predictor's input feature dimension is 130.We utilize the raw energy observations released in (Sukthanker et al., 2024) to train a single hardware-aware meta-predictor across energy observations from eight GPU types. Our meta-predictor is a simple MLP, similar to the one in HAT, with 4 hidden layers, 2 layers for processing the hardware embedding (which the network is conditioned on). The MLP's hidden dimension is 256, and the input feature dimension matches the one-hot encoded architecture feature map for this space, i.e., 80.Given a preference vector r ∈ R M , we use the hypernetwork h ϕ (r) : R M → A, parameterized by ϕ ∈ R n , to generate an un-normalized architecture distribution α that is later used to compute the upper-level updates in (4). In our experiments, h ϕ is composed of M -15 embedding layers e m , m ∈ {2, . . . , M } with n m possible learnable vectors of size dim(A) M -1 . The output of h ϕ is the concatenation of all M -1 outputs of e m , such that its size matches dim(A). See Figure 2 for details.In order to enable the hypernetwork to generate architectures across multiple devices, inspired by Wang et al. (2022) and Lin et al. (2020) , we propose a MetaHypernetwork H Φ (r, d t ) : R M × H M -1 → A that can meta-learn across T different hardware devices (see Figure 1 ). The input to H Φ is a concatenation of device feature vectors across all metrics, i.e. d t = ⊕ M m=2 d m t . Similar to Lee et al. (2021b) , d m t ∈ H is a fixed-size feature vector representative of device t ∈ {1, . . . , T } and objective m ∈ {2, . . . , M }, that is obtained by evaluating a fixed set of reference architectures for a given metric. The MetaHypernetwork, with Φ = ∪ K k=0 ϕ k parameters, contains a bank of K > T hypernetworks {h ϕ k (r)} K k=1 and an additional embedding layer e ϕ0 (d t ) : H M -1 → R K at the beginning, that learns a similarity map for every device feature to the hypernetworks' bank. If we denote by h ϕ 1:k = (h ϕ1 • • • h ϕ k ) T the vector of all hypernetworks in the bank, then, given a preference vector r, to obtain α for device t, we compute a weighted mixture of predictions of all h ϕ in the hypernetwork bank as follows:αΦ = H Φ (r, d t ) = K k=1 e ϕ0 (d t )[k] • h ϕ k (r) = e ϕ0 (d t ) • h ϕ 1:k (r).We keep the MetaHypernetwork architecture similar across search spaces. The only thing we adapt is the output dimensionality of the hypernetwork (in the hypernetwork bank of MetaHypernetwork), which corresponds to the dimensionality of the architecture parameters of the respective search space. We set the size of the initial hardware embedding layer and the hypernetwork bank to 50 for all search spaces. Furthermore, each hypernetwork has 100 possible learnable embeddings e m , for every objective m ∈ {2, . . . , M }, to map the scalarization vector to an architecture. See Figure 2 for an illustration of the MetaHypernetwork architecture.For the NAS-Bench-201 search space, we use a single embedding layer of dimensionality 30, i.e. corresponding to the dimensionality of the architecture space: 6×5 (6 edges and 5 operation choices on each edge). For the 3-objective experiment, we include an additional embedding for the energy usage objective, concatenated with the latency embedding before passing it to the MetaHypernetwork. The individual hypernetworks in the MetaHypernetwork bank have 2 embedding layers with dimensionality 15, whose outputs are concatenated to match the architecture space dimensions.In the MobileNetV3 space, we use 4 embedding layers -for depth, expansion ratio, kernel size, and resolution. The space comprises 5 blocks, each with 3 depth choices, making the depth embedding layer dimensionality 5 × 3. The kernel and expansion embedding layers have dimensions 5 × 4 × 3, corresponding to 5 blocks with a maximum depth of 4 and 3 possible kernel size or expansion ratio choices. The resolution embedding layer has a dimension of 25, representing 25 possible resolution choices.In the Seq-Seq Transformer (HAT) space, the individual hypernetworks of the MetaHypernetwork utilize 9 embedding layers (the encoder layer count is fixed; see Table 1 ):• 2 embedding layers of size 2 for the encoder and decoder blocks to map the scalarization to the embedding dimension architecture parameter, held constant throughout the encoder or decoder block.• 2 embedding layers with dimensions 6 × 3 (6 encoder/decoder layers, 3 choices) for the linear layer size in every attention block for both encoder and decoder.• 2 embedding layers with dimensions 6 × 2 for the number of heads in each attention block.• 1 embedding layer of size 6 to encode the 6 possible choices for the number of layers in the decoder.• 1 embedding layer of size 6 × 3 (6 encoder layers, 3 choices) for the arbitrary encoder layer choice for attention.• 1 embedding layer of size 6 × 2 (6 encoder layers, 2 choices) for the number of heads in the encoder-decoder attention.For the HW-GPT-Bench space, the individual hypernetworks of the MetaHypernetwork contain 5 embedding layers:• 1 embedding layer of dimension 1 × 3 for mapping the scalarization to the embedding dimension architecture parameter of the language model, with 3 choices.• 1 embedding layer of dimension 1 × 3 for mapping the scalarization to the layer number dimension architecture parameter of the language model, with 3 choices.• 1 embedding layer of dimension 12 × 3 for mapping the scalarization to the mlp_ratio dimension architecture parameter of the language model, with 12 layers and 3 mlp_ratio choices per layer.• 1 embedding layer of dimension 12×3 for mapping the scalarization to the num_heads dimension architecture parameter of the language model, with 12 layers and 3 choices per layer.• 1 embedding layer of dimension 2 for toggling the bias in linear layers on or off.In Table 4 , we show the search hyperparameters and their corresponding values we use to conduct our experiments with MODNAS. For the convolutional spaces we subtract a cosine similarity penalty from the scalarized loss following (Ruchte & Grabocka, 2021) :EQUATIONwhere || • || is the l 2 norm. We set λ to 0.001. Empirically we did not observe significant differences on disabling the cosine penalty term.Since our method relies on a scalarization of different objectives, it is important that the objectives being optimized are on the same scale. For simplicity, lets consider the scenario where the two objectives of interest are the cross-entropy loss and latency. Since we pretrain and freeze our MetaPredictor, the latency-scale remains constant throughout the search, while the cross-entropy loss of the Supernetwork (likely) decreases over time. To this end, we use the following max-min normalization to normalize the objectives:EQUATIONwhereL = N i=1 stop_g L m t (•, α i ) iis the set of losses evaluated on N architectures and potentially N previous steps. For the latency objective, we precompute these sample-statistics using N samples (ground-truth for NAS-Bench-201 and predicted for OFA and HAT spaces) from the search space, whilst for the cross-entropy loss we compute them throughout the search. Furthermore, to take into account the decreasing cross-entropy, we reset the cross-entropy loss statistics after every epoch. NAS-Bench-201 (Dong & Yang, 2020) is a convolutional, cell-based search space. The search space consists of 3 stages, each with number of channels 16, 32 and 64, respectively. Each stage contains a convolutional cell repeated 5 times. Here, every cell is represented as a directed acyclic graph (DAG) which has 4 nodes, densely connected with 6 edges. Each edge has 5 possible operation choices: a skip connection, a zero operation, a 3×3 convolution, a 5×5 convolution or an average pooling operation. NAS-Bench-201 is a tabular benchmark exhaustively constructed, where the objective is finding the optimal cell for the given macro skeleton.MobileNetV3 proposed in OFA (Cai et al., 2020 ) is a macro convolutional search space. The different searchable dimensions in the search space are the depth (per block), the kernel size (for every layer in every block) and the channel expansion ratio (for every layer in every block). There are a total of 5 blocks, each with 3 possible depth choices and every layer in this block has 3 possible kernel sizes and channel expansion ratio choices. This amounts to a total search space size of ((3 × 3) 2 + (3 × 3) 3 + (3 × 3) 4 ) 5 ≈ 2 × 10 19 . Additionally, every architecture has 25 possible choices for the size of the input resolution. The 3 possible choices for depth, kernel size and expansion ratio are {2, 3, 4}, {3, 5, 7} and {3, 4, 6}, respectively. The input resolution choices are {128, 132, 136, 140, 144, 148, 152, 156, 160, 164, 168, 172, 176, 180, 184, 188, 192, 196, 200, 204, 208, 212, 216, 220, 224}. We use a width factor of 1.2 similar to OFA (Cai et al., 2020) .Seq-Seq Encoder-Decoder Transformer (HAT) (Wang et al., 2020) for the En-De machine translation task has a searchable number of layers, embedding dimension, feedforward expansion layer dim per-layer, number of heads per-layer for both the encoder and the decoder sub-modules. In addition to this, the number of encoder layers the decoder attends to, and the number of attention heads in the encoder-decoder attention is also searchable. We present the details of the search space in Table 1 . (Sukthanker et al., 2024) is a decoder-only transformer space designed for autoregressive language modeling. The search space includes choices for embedding dimensions {768, 384, 192}, the number of layers from {10, 11, 12}, the MLP expansion ratio per layer from {2,3,4}, the number of heads per layer from {12,8,4}, and the option to toggle the bias parameter on or off in the layers.This section describes the hardware devices and tasks used to evaluate MODNAS and the MOO baselines throughout the paper. We assess our methods across small-and large-scale image classification datasets, including CIFAR-10 and ImageNet-1K. For the machine translation task, we evaluate our method on the WMT'14 En-De dataset (Macháček & Bojar, 2014) , and we use the OpenWebText (Gokaslan & Cohen, 2019) dataset for language modeling. Furthermore, we evaluate MODNAS across 19 devices on NAS-Bench-201, 12 devices on MobileNetV3, three devices on Seq-Seq Transformer, and eight devices from HW-GPT-Bench (Sukthanker et al., 2024) , with zero-shot generalization to test devices. Table 2 lists the devices used. For more details on the devices, we refer readers to Lee et al. (2021b ), Cai et al. (2020) , Wang et al. (2020) , Li et al. (2021), and Sukthanker et al. (2024) . titan_xp_32, titan_xp_64, v100_1, v100_32, v100_64, titan_rtx_1, titan_rtx_32 Seq-Seq Transformer (HAT) titanxp gpu, cpu xeon cpu raspberrypi WMT14.en-de HW-GPT-Bench a40, v100, rtx2080, rtx3080 a100, h100, P100, a6000 OpenWebTextIn Table 3 we provide the number of GPU hours we ran MODNAS and baselines on every search space. We ran the search on NAS-Bench-201, OFA, together with the evaluations on Nvidia RTX2080Ti, while for HAT we used NVidia A6000. For both OFA and HAT, we used 8 GPUs in parallel. Similar as in Sukthanker et al. (2024) , on the HW-GPT-Bench space we ran the MODNAS search and evaluations on 4 Nvidia A100 GPUs. (Deb et al., 2002; Daulton et al., 2020; Zhao et al., 2022) O(NT) O(1) ProxylessNAS (Cai et al., 2018) O(CT) O(1) MetaD2A + HELP (Lee et al., 2021a; b) O(N) O(CT) OFA (Cai et al., 2020) + HELP (Lee et al., 2021b) O( 1 5 and 3 in the Appendix for more details. For every device we optimize for 2 objectives, namely latency (ms) and test accuracy on CIFAR-10. For method, metric and device we report the mean of 3 independent search runs. Higher area in the radar indicates better performance for every metric. Test devices are colored in red around the radar plot.Here we allocate double the budget to baselines, i.e. we run all baselines for 50 function evaluations. w can be replaced with any other parameter here, also architectural ones (see Section 3).We use the implementations from SyneTune(Salinas et al., 2022): https://github.com/awslabs/syne-tunehttps://github.com/HayeonLee/HELPhttps://github.com/mit-han-lab/hardware-aware-transformersm = 1 (CE loss) does not have an hardware embedding. 


Decision: The paper received a single review. Upon checking the paper, the AC is positive about the proposed method, MODNAS, a new approach for hardware-aware NAS that encodes user preferences for the trade-off between performance and hardware metrics, and yields representative and diverse architectures in just one run. The method is tested on a large number of hardware targets and benchmarks, where it shows promising results.