Multi-objective Differentiable Neural Architecture Search

## Abstract

Abstract Pareto front profiling in multi-objective optimization (MOO), i.e. finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives like neural network training. Typically, in MOO neural architecture search (NAS), we aim to balance performance and hardware metrics across devices. Prior NAS approaches simplify this task by incorporating hardware constraints into the objective function, but profiling the Pareto front necessitates a search for each constraint. In this work, we propose a novel NAS algorithm that encodes user preferences for the trade-off between performance and hardware metrics, and yields representative and diverse architectures across multiple devices in just one search run. To this end, we parameterize the joint architectural distribution across devices and multiple objectives via a hypernetwork that can be conditioned on hardware features and preference vectors, enabling zero-shot transferability to new devices. Extensive experiments with up to 19 hardware devices and 3 objectives showcase the effectiveness and scalability of our method. Finally, we show that, without additional costs, our method outperforms existing MOO NAS methods across qualitatively different search spaces and datasets, including MobileNetV3 on ImageNet-1k and a Transformer space on machine translation.

## 

### 1 Introduction

The ability to make good tradeoffs between predictive accuracy and efficiency (in terms of latency and/or energy consumption) has become crucial in an age of ever increasing neural networks complexity and size (Kaplan et al., ; Hoffmann et al., ; Zhai et al., ; Alabdulmohsin et al., ) and a plethora of embedded devices. However, finding the right trade-off remains a challenging task that typically requires human intervention and a lot of trial-and-error across devices. With multiple conflicting objectives, it becomes infeasible to optimize all of them simultaneously and return a single solution.
Ideally, the user should be able to choose from a set of diverse Pareto optimal solutions that represent their preferences regarding the trade-off between objectives.

Neural Architecture Search (NAS) (White et al., ) provides a principled framework to search for neural network architectures in an automated fashion. Substantial research effort (Elsken et al., ; Cai et al., ; Wang et al., ; Chen et al., ) has been invested in extending NAS for multi-objective optimization (MOO) and taking into consideration the performance and various hardware efficiency constraints, such as latency, energy consumption, etc.
However, to the best of our knowledge, no existing gradient-based method returns the full Pareto front for the MOO problem at hand without running their search routine multiple times with different hardware constraints.

In this work, we propose a scalable and hardware-aware Multi-Objective Differentiable Neural Architecture Search (MODNAS) algorithm that efficiently trains a single supernet that can be used to read off
Pareto-optimal solutions for both different user preferences and different target devices,
without any extra search steps.
To search across devices, we frame the problem at hand as a multi-task multi-objective optimization problem, where each task (device) has multiple (conflicting) objectives, e.g., classification accuracy and latency.
The user's preferences are modelled by a *preference vector* that defines a scalarization of the different objectives. This preference vector, along with features of the hardware of interest, is fed to a hypernetwork (Ha et al., ) that outputs continuous architectural parameters $\alpha$.
To search in the space of architectures, we employ a one-shot model and use a bi-level optimization scheme, as is typically done in gradient-based NAS. Here, the upper-level parameters are the hypernetwork weights, optimized in expectation across different preference vectors and
hardware devices via
multiple gradient descent (MGD) (Désidéri, ).

Figure: Figure 1: MODNAS overview. Given a set of $T$ devices, MODNAS seeks to optimize $M$ (potentially conflicting) objectives across these devices. To do this, it employs a MetaHypernetwork $H_{\Phi}(r,d_{t})$, that takes as input a scalarization $r$, representing the user preferences, and a device embedding $d_{t}$, to yield an un-normalized architectural distribution $\tilde{\alpha}$. The Architect uses $\tilde{\alpha}$ to sample differentiable discrete architectures, used in the Supernetwork to estimate accuracy and in the MetaPredictor to estimate the other $M-1$ loss functions (e.g. latency, energy consumption) for every device. By iterating over devices and sampling scalarizations uniformly from the $M$-dimensional simplex, at each iteration we update the MetaHypernetwork using multiple gradient descent (MGD).
Refer to caption: /html/2402.18213/assets/x1.png

To evaluate our method, we conduct experiments on multiple NAS search spaces, including CNN and Transformer architectures, and up to 3 objectives across 19 hardware devices.
While other NAS methods that utilize hardware constraints in their search objectives require substantial search costs both for each new constraint and each new hardware, MODNAS addresses both in a zero-shot manner, without extra search cost, while nevertheless yielding higher quality solutions.

Our contributions can be summarized as follows:

- 1.
We present a principled and robust approach for Multi-objective Differentiable NAS, that leverages *hypernetworks* and *multiple gradient descent* to simultaneously find Pareto-optimal architectures across devices.
- 2.
This work is the *first* to provide a global view of the Pareto solutions with just a single model, without the need to search or fine-tune on new target devices.
- 3.
Extensive evaluation of our method across 3 different search spaces (NAS-Bench-201, MobileNetV3, Transformer-based), 2 tasks (Image classification and machine translation), and up to 19 hardware devices and 3 objectives, show both improved *efficiency* and *performance* in comparison to previous approaches that use a constrained objective in their search.

We provide our code via the following link: [https://github.com/automl/MODNAS](https://github.com/automl/MODNAS).

### 2 Background and Related Work

In this section, before describing our algorithm, we introduce some basic concepts, definitions and related work. Refer to Appendix [A](#A1) for an extended related work.

Multi-objective optimization (MOO) for Multi-Task Learning. Consider a multi-task dataset $\mathcal{D}$ consisting of $N$ instances, where the feature vector of the $i-$th instance is denoted as $x_{i}\in\mathcal{X}$, and the $M$-many associated target variables as $y_{i}^{1}\in\mathcal{Y}^{1},\dots,y_{i}^{M}\in\mathcal{Y}^{M}$. Moreover, consider there exists a family of parametric models $f(\bm{x};\bm{w}):\mathcal{X}\rightarrow\{\mathcal{Y}^{1}\times\dots\times\mathcal{Y}^{M}\}$, parameterized by $\bm{w}$, that maps the input $\bm{x}$ to the joint space of the multiple tasks. To simplify the notation, we denote the prediction of the $m$-th task as $f^{m}(\bm{x};\bm{w}):\mathcal{X}\rightarrow\mathcal{Y}^{m}$, and the respective loss $\mathcal{L}^{m}(\bm{w})\triangleq\frac{1}{N}\sum_{i}^{N}\ell^{m}(y_{i}^{m},f^{m}(x_{i};\bm{w}))$.
The vector of the values of all loss functions is denoted as $\mathbf{L}(\bm{w})\triangleq(\mathcal{L}^{1}(\bm{w}),\dots,\mathcal{L}^{M}(\bm{w}))$.
MOO then seeks to find a set of Pareto-optimal solutions $\bm{w}^{*}$ that jointly minimize $\mathbf{L}(\bm{w})$(^1^11$\bm{w}$ can be replaced with any other parameter here, also architectural ones (see Section [3](#S3)).):

$$ $\bm{w}^{*}\in\operatorname*{argmin}_{\bm{w}}\mathbf{L}(\bm{w})$ (1) $$

###### Definition 2.1 .

(Pareto Optimality):
A solution $\bm{w}_{2}$ dominates $\bm{w}_{1}$ iff $\mathcal{L}^{m}(\bm{w}_{2})\leq\mathcal{L}^{m}(\bm{w}_{1})$, $\forall m\in\{1,\dots,M\}$, and $\mathbf{L}(\bm{w}_{1})\neq\mathbf{L}(\bm{w}_{2})$. In other words, a dominating solution has a lower loss value on at least one task and no higher loss value on any task. A solution $\bm{w}^{*}$ is called *Pareto optimal* iff there exists no other solution dominating $\bm{w}^{*}$.

###### Definition 2.2 .

(Pareto front):
The sets of Pareto optimal points and their function values
are called *Pareto set* ($\mathcal{P}_{\bm{w}}$) and *Pareto front* ($\mathcal{P}_{\mathbf{L}}=\{\mathbf{L}(\bm{w})_{\bm{w}\in\mathcal{P}_{\bm{w}}}\}$), respectively.

Linear Scalarization. In MOO, a standard technique to solve the $M$-dimensional problem is using a preference vector $\bm{r}\in\mathcal{S}\triangleq\{\mathbb{R}^{M}|\sum_{m=1}^{M}r_{m}=1,r_{m}\geq 0,\forall m\in\{1,\dots,M\}\}$ in the $M$-dimensional probability simplex (Lin et al., ; Mahapatra & Rajan, ; Ruchte & Grabocka, ).
Every $\bm{r}\in\mathcal{S}$ yields a convex combination of the loss functions in Equation [1](#S2.E1) as $\mathcal{L}_{\bm{r}}(\bm{w})=\bm{r}^{\mathbf{T}}\mathbf{L}(\bm{w})$. Given a preference vector $\bm{r}$, one can apply standard, single-objective optimization algorithms to find a minimizer $\bm{w}_{\bm{r}}^{*}=\operatorname*{argmin}_{\bm{w}}\mathcal{L}_{\bm{r}}(\bm{w})$. By sampling multiple $\bm{r}$ vectors, one can compute Pareto-optimal solutions $\bm{w}_{\bm{r}}^{*}$ that profile the Pareto front.
Several methods (Lin et al., ; Navon et al., ; Hoang et al., ; Phan et al., ) employ a hypernetwork (Ha et al., ) that can generate Pareto-optimal solutions given different preference vectors as input.
Similarly, in this work, we utilize a hypernetwork conditioned on scalarizations, to generate Pareto-optimal architectures.

Multiple Gradient Descent (MGD). MOO can be solved to local optimality via MGD (Désidéri, ), as a natural extension of single-objective gradient descent, which iteratively updates $\bm{w}$ towards a direction that ensures that all tasks improve simultaneously (called *Pareto improvement*): $\bm{w}^{\prime}\leftarrow\bm{w}-\xi g_{\bm{w}}^{*}$, where $g_{\bm{w}}^{*}$ is a vector field that needs to be determined. If we denote by $g_{\bm{w}}^{m}=\nabla_{\bm{w}}\mathcal{L}^{m}(\bm{w})$ the gradient of the $m$-th scalar loss function, via Taylor approximation, the decreasing direction of $\mathcal{L}^{m}$ when we update $\bm{w}$ towards $g_{\bm{w}}^{*}$ is given by $\langle g_{\bm{w}}^{m},g_{\bm{w}}^{*}\rangle\approx-(\mathcal{L}^{m}(\bm{w}^{\prime})-\mathcal{L}^{m}(\bm{w}))/\xi$.
In MGD $g_{\bm{w}}^{*}$ is chosen to maximize the slowest update rate among all objectives:

$$ $g_{\bm{w}}^{*}\propto\operatorname*{argmax}_{g_{\bm{w}}\in\mathbb{R}^{d},||g_{\bm{w}}||\leq 1}\Big{\{}\min_{m\in[M]}\langle g_{\bm{w}},g_{\bm{w}}^{m}\rangle\Big{\}}.$ (2) $$

The early work of Désidéri () has been extended in various settings, particularly multi-task learning, with great promise (Sener & Koltun, ; Lin et al., ; Mahapatra & Rajan, ; Liu & Vicente, ), but these approaches are applied to mainly a fixed architecture and extending them to a search space of architectures is non-trivial.

One-shot NAS and Bi-Level optimization.
With the architecture space being intrinsically discrete and hence expensive to search on, most existing differentiable NAS approaches leverage the weight sharing paradigm and continuous relaxation to enable gradient descent (Liu et al., ; Pham et al., ; Bender et al., ; Xie et al., ; Xu et al., ; Dong & Yang, ; Chen et al., ; Liu et al., ; Movahedi et al., ; Zhang et al., ). Typically, in these approaches, architectures are stacks of cells, where the cell structure is represented as a directed acyclic graph (DAG) with $N$ nodes and $E$ edges. Every transition from node $i$ to $j$, i.e. edge $(i,j)$, is associated with an operation $o^{(i,j)}\in\mathcal{O}$, where $\mathcal{O}$ is a predefined candidate operation set. The DAG node $j\in[N]$ contains a latent representation $x^{j}$ that is computed as the summation of all input tensors to that node: $x^{j}=\sum_{i<j}o^{(i,j)}(x^{i})$. With this formulation, the NAS task reduces to searching for the optimal operations in edges and the cell topology. Liu et al. () first proposed a continuous relaxation of the discrete operation choices in the DAG edges to form a $\mathtt{MixOp}$: $\tilde{o}^{(i,j)}(x^{i})=\sum_{o\in\mathcal{O}}\alpha_{o}^{(i,j)}o(x^{i})$, where $\alpha^{(i,j)}\in\mathbb{R}^{|\mathcal{O}|}$ is defined over the probability simplex and each entry in it represents the operation strength in edge $(i,j)$. This allows to frame the NAS problem as a bi-level optimization one, with differentiable objectives w.r.t. all variables:

$$ $\displaystyle\operatorname*{argmin}_{\alpha}\mathcal{L}^{val}(\bm{w}^{*}(\alpha),\alpha)$ (3) $\displaystyle s.t.\quad\bm{w}^{*}(\alpha)=\operatorname*{argmin}_{\bm{w}}\mathcal{L}^{train}(\bm{w},\alpha),$ $$

where $\mathcal{L}^{train}$ and $\mathcal{L}^{val}$ are the empirical losses on the training and validation data, respectively, $\bm{w}$ are the supernetwork parameters, $\alpha\in\mathcal{A}\triangleq\mathbb{R}^{|\mathcal{O}|^{E}}$ are the continuous architectural parameters, and $\bm{w}^{*}(\alpha):\mathcal{A}\rightarrow\mathbb{R}^{d}$ is a best response function that maps architectures to their optimal weights.

While early NAS approaches focused mainly on achieving high accuracy, hardware-aware differentiable NAS methods (Wu et al., ; Wan et al., ; Cai et al., ; Wu et al., ; Fu et al., ; Xu et al., ; Jiang et al., ; Wang et al., ) aim to search for architectures that run efficiently on a target hardware device.
However, most of them incorporate hardware constraints directly in their search, hence, resulting in a single optimal solution, and a necessity to repeat the search procedure in order to profile the Pareto front.
Here, we propose an algorithm that can profile the entire Pareto front in a single search run.

###### Definition 2.1 .

###### Definition 2.2 .

### 3 Hardware-aware Multi-objective Differentiable Neural Architecture Search

We now formalize the NAS multi-objective bi-level optimization problem across multiple hardware devices and then introduce a scalable method that combines MGD with linear scalarizations to efficiently solve this problem.

#### 3.1 Problem Definition & Sketch of Solution Approach

In multi-objective NAS, the problem described in Equation [1](#S2.E1) becomes more difficult, since we are not only concerned with finding $\bm{w}^{*}$ given a fixed architecture, but we want to optimize in the space of architectures $\mathcal{A}$ as well.
Assuming we have $T$ hardware devices (target functions) and $M$ objectives (e.g. accuracy, latency, energy usage, etc.), similarly to ([3](#S2.E3)), for every $t\in\{1\dots T\}$, the Pareto set $\mathcal{P}_{\alpha_{t}}$ of the multi-objective NAS problem is obtained by solving the following bi-level optimization problem:

$$ $\displaystyle\operatorname*{argmin}_{\alpha}\mathbf{L}_{t}^{valid}(\bm{w}^{*}(\alpha),\alpha)$ (4) $\displaystyle s.t.\quad\bm{w}^{*}(\alpha)=\operatorname*{argmin}_{\bm{w}}\mathbf{L}_{t}^{train}(\bm{w},\alpha),$ $$

where the $M$-dimensional loss vector $\mathbf{L}_{t}(\bm{w}^{*}(\alpha),\alpha)\triangleq\big{(}\mathcal{L}_{t}^{1}(\bm{w}^{*}(\alpha),\alpha),\dots,\mathcal{L}_{t}^{M}(\bm{w}^{*}(\alpha),\alpha)\big{)}$ is evaluated $\forall t\in\{1,\dots,T\}$. $\mathbf{L}_{t}^{train}$ and $\mathbf{L}_{t}^{valid}$ are the vectors with all $M$ loss functions evaluated on the train and validation splits of $\mathcal{D}$, respectively.
Our goal is to search for a set of Pareto-optimal architectures for every target device that covers a diverse and representative set of preferences for the different objectives.
However, solving ([4](#S3.E4)) for every target device $t$, when done naively, requires running bi-level search independently $T$ times.
To overcome this limitation, we integrate a single hypernetwork inside the one-shot model (supernetwork) used in conventional NAS (Bender et al., ; Pham et al., ; Liu et al., ), to generate architectures conditioned on the various device (target function) embeddings and preference vectors at the cost of a *single search run*.

#### 3.2 Algorithm Design and Components

Our search model is composed of four main modular components: (1) a MetaHypernetwork that generates the architectural distribution; (2) an Architect that samples discrete architectures from this distribution; (3) a Supernetwork that exploits the weight sharing paradigm for search efficiency and provides a proxy for the accuracy; and (4) a MetaPredictor that predicts hardware metrics and enables gradient propagation. We now discuss each of these in turn.

Figure: Algorithm 1 MODNAS

MetaHypernetwork.
Hypernetworks are a class of neural networks that can generate an arbitrary number of parameters of another model. They were initially proposed for model compression (Ha et al., ) and were later adopted for NAS (Brock et al., ) and MOO (Navon et al., ; Lin et al., ).
Given a preference vector $\bm{r}\in\mathbb{R}^{M}$, we use the hypernetwork $h_{\phi}(\bm{r}):\mathbb{R}^{M}\rightarrow\mathcal{A}$, parameterized by $\phi\in\mathbb{R}^{n}$, to generate an un-normalized architecture distribution $\tilde{\alpha}$ that is later used to compute the upper-level updates in ([4](#S3.E4)).
In our experiments, $h_{\phi}$ is composed of $M-1$ (^2^22$m=1$ (CE loss) does not have an hardware embedding.) embedding layers $e^{m}$, $m\in\{2,\dots,M\}$ with $n_{m}$ possible learnable vectors of size $\frac{dim(\mathcal{A})}{M-1}$. The output of $h_{\phi}$ is the concatenation of all $M-1$ outputs of $e^{m}$, such that its size matches $dim(\mathcal{A})$. See Figure [8](#A3.F8) in the appendix for details.

In order to enable the hypernetwork to generate architectures across multiple devices, inspired by Wang et al. () and Lin et al. (), we propose a MetaHypernetwork $H_{\Phi}(\bm{r},d_{t}):\mathbb{R}^{M}\times\mathcal{H}^{M-1}\rightarrow\mathcal{A}$ that can meta-learn across $T$ different hardware devices (see Figure [1](#S1.F1)). The input to $H_{\Phi}$ is a concatenation of device feature vectors across all metrics, i.e. $d_{t}=\oplus_{m=2}^{M}d_{t}^{m}$. Similar to Lee et al. (), $d_{t}^{m}\in\mathcal{H}$ is a fixed-size feature vector representative of device $t\in\{1,\dots,T\}$ and objective $m\in\{2,\dots,M\}$, that is obtained by evaluating a fixed set of reference architectures for a given metric.
The MetaHypernetwork, with $\Phi=\cup_{k=0}^{K}\phi_{k}$ parameters, contains a bank of $K>T$ hypernetworks $\{h_{\phi_{k}}(\bm{r})\}_{k=1}^{K}$ and an additional embedding layer $e_{\mathcal{\phi}_{0}}(d_{t}):\mathcal{H}^{M-1}\rightarrow\mathbb{R}^{K}$ at the beginning, that learns a similarity map for every device feature to the hypernetworks' bank.
Given a preference vector $\bm{r}$, to obtain $\tilde{\alpha}$ for device $t$, we then compute a weighted mixture of predictions of all $h_{\phi}$ in the hypernetwork bank as follows:

$$ $\tilde{\alpha}_{\Phi}=H_{\Phi}(\bm{r},d_{t})=\sum_{k=1}^{K}e_{\phi_{0}}(d_{t})[k]\cdot h_{\phi_{k}}(\bm{r}).$ (5) $$

In all experiments, we pretrain the MetaHypernetwork to yield an equal probability mass over all architectural parameters for all scalarizations and device embeddings.
By using the preference vector $\bm{r}$ to create a linear scalarization of $\mathbf{L}_{t}$ and the MetaHypernetwork to model the architectural distribution across $T$ devices, the bi-level problem in ([4](#S3.E4)) reduces to:

$$ $\displaystyle\operatorname*{argmin}_{\Phi}\mathbb{E}_{\bm{r}\sim\mathcal{S}}\big{[}\bm{r}^{\mathbf{T}}\mathbf{L}_{t}^{valid}(\bm{w}^{*}(\alpha_{\Phi}),\alpha_{\Phi})\big{]}$ (6) $\displaystyle s.t.\quad\bm{w}^{*}(\alpha_{\Phi})=\operatorname*{argmin}_{\bm{w}}\mathbb{E}_{\bm{r}\sim\mathcal{S}}\big{[}\bm{r}^{\mathbf{T}}\mathbf{L}_{t}^{train}(\bm{w},\alpha_{\Phi})\big{]},$ $$

where $\alpha_{\Phi}$ are the normalized architectural parameters obtained from the Architect $\Lambda(\tilde{\alpha}_{\Phi})$ and $\bm{r}^{\mathbf{T}}\mathbf{L}_{t}(\cdot,\alpha_{\Phi})=\sum_{m=1}^{M}r_{m}\mathcal{L}_{t}^{m}(\cdot,\alpha_{\Phi})$ is the scalarized loss for device $t$. Conditioning the MetaHypernetwork on the hardware embeddings allows us to generate architectures on new test devices without extra finetuning or meta-learning steps. We use the *Dirichlet* distribution $Dir(\beta)$, $\beta=(\beta_{1},\dots,\beta_{M})$, to sample the preference vectors and approximate the expectation over the scalarizations using Monte Carlo sampling. In our experiments we set $\beta_{1}=\dots=\beta_{M}=1$, for a uniform sampling over the $(M-1)$-simplex, however, one can set these differently based on user priors or make it a learnable parameter (Chen et al., ).

MetaPredictor.
For the cheap-to-evaluate hardware objectives, such as latency, energy consumption, we employ a regression model $p_{\theta}^{m}(\alpha,d_{t}^{m}):\mathcal{A}\times\mathcal{H}\rightarrow\mathbb{R}$, parameterized by $\theta$, that predicts the target labels $y_{t}^{m}$ for objective $m$ and device $t$, given an architecture $\alpha$ and device embedding $d_{t}^{m}$. We use the same predictors as Lee et al. () and optimize the MSE loss:

$$ $\min_{\theta}\mathbb{E}_{\alpha\sim\mathcal{A},t\sim[T]}\big{(}y_{t}^{m}-p_{\theta}^{m}(\alpha,d_{t}^{m})\big{)}^{2},$ (7) $$

as done in Lee et al. () for meta-learning performance metrics across datasets.
In our experiments, we pretrain a separate MetaPredictor for every hardware objective $m$ (e.g. latency, energy, etc.) on a subset of $(\alpha,y_{t}^{m})$ pairs, and use its predicted value directly in ([6](#S3.E6)) as $\mathcal{L}_{t}^{m}(\cdot,\alpha_{\Phi})=p_{\theta}^{m}(\alpha_{\Phi},d_{t}^{m})$. During the search we freeze and do not update further the MetaPredictor parameters $\theta$.

Supernetwork.
For the expensive objective, such as neural network classification accuracy, we utilize a Supernetwork, that encodes the architecture space and shares the parameters between architectures. The Supernetwork provides a best response function $\bm{w}^{*}(\alpha_{\Phi})$ for the scalarized loss in ([6](#S3.E6)). Generally speaking, one could use any parametric model to estimate the best response function, including performance predictors as done in Lee et al. (), however, this would require an expensive prior step of creating the training dataset for the predictor.
To circumvent the large memory cost coming with the Supernetwork training, we further employ two other components: (1) at each training step, $\alpha_{\Phi}$ is a one-hot encoding of the architecture sampled in a differentiable manner from the Architect (Dong & Yang, ; Cai et al., ; Xie et al., ). This way only a single architecture is activated at each step.; (2) the parameters of the operation choices in the Supernetwork are entangled, increasing even further the memory efficiency compared to only weight-sharing (Sukthanker et al., ).

Architect.
The Architect $\Lambda(\tilde{\alpha})$ samples discrete architectural configurations from the un-normalized distribution $\tilde{\alpha}_{\Phi}=H_{\Phi}(\bm{r},d_{t})$ and enables gradient estimation through discrete variables for $\nabla_{\Phi}\mathbf{L}_{t}(\cdot,\alpha_{\Phi})$. Methods such as GDAS (Dong & Yang, ) utilize the Straight-Through Gumbel-Softmax (STGS) estimator (Jang et al., ), that integrates the Gumbel reparameterization trick to approximate the gradient. Here we employ the recently proposed *ReinMax* estimator (Liu et al., ), that yields second-order accuracy without the need to compute second-order derivatives. Similar to the findings in Liu et al. (), in our initial experiments, ReinMax outperformed the GDAS STGS estimator (see Appendix [G.2](#A7.SS2)). Therefore, we use ReinMax in all following experiments. Given the architecture parameters $\tilde{\alpha}_{\Phi}$ from the MetaHypernetwork, we obtain a differentiable discrete architecture sample from the Architect as follows:

$$ $\displaystyle\mathtt{\alpha_{\Phi}\sim Cat\big{(}\sigma_{1}(\tilde{\alpha}_{\Phi})\big{)}}$ $\displaystyle\mathtt{\pi\leftarrow 2\cdot\sigma_{1}\Big{(}stop_g\big{(}ln(\frac{\alpha_{\Phi}+\sigma_{\tau}(\tilde{\alpha}_{\Phi})}{2})-\tilde{\alpha}_{\Phi}\big{)}+\tilde{\alpha}_{\Phi}\Big{)}-\frac{\sigma_{1}(\tilde{\alpha}_{\Phi})}{2}}$ $\displaystyle\mathtt{\alpha_{\Phi}\leftarrow\pi-stop_g(\pi)+\alpha_{\Phi}},$ $$

where $\mathtt{Cat}$ is the categorical distribution, $\tau$ is the temperature in the tempered softmax $\sigma_{\tau}(\alpha)_{i}=\frac{exp(\alpha_{i}/\tau)}{\sum_{j=1}^{|\mathcal{O}|}exp(\alpha_{j}/\tau)}$, and $\mathtt{stop_g(\cdot)}$ duplicates its input and detaches it from backpropagation.

#### 3.3 Optimizing the MetaHypernetwork via MGD

We denote the gradient of the scalarized loss in ([6](#S3.E6)) w.r.t. the MetaHypernetwork parameters $\Phi$, which are shared across all devices $t\in\{1,\dots,T\}$, as:

$$ $\displaystyle g_{\Phi}^{t}=\bm{r}^{\mathbf{T}}\nabla_{\Phi}\mathbf{L}_{t}(\cdot,\alpha_{\Phi})=\sum_{m=1}^{M}r_{m}\nabla_{\Phi}\mathcal{L}_{t}^{m}(\cdot,\alpha_{\Phi}),$ (8) $$

where $\alpha_{\Phi}$ is the discrete architectural sample obtained from the Architect $\Lambda(\tilde{\alpha}_{\Phi})$.
A simple, though naive way to optimize over all devices is to use the mean gradient $\hat{g}_{\Phi}=\frac{1}{T}\sum_{t=1}^{T}g_{\Phi}^{t}$ over tasks: $\Phi\leftarrow\Phi-\xi\hat{g}_{\Phi}$, or just simply update sequentially for every task independently: $\Phi\leftarrow\Phi-\xi g_{\Phi}^{t}$, $\forall t$. However, these updates disregard the interdependence between tasks and might lead to slow convergence, or even high bias in the parameter estimate. On the other hand, Multiple Gradient Descent (MGD) (Désidéri, ; Sener & Koltun, ) provides a more plausible approach to estimate the update directions for every task simultaneously by maximizing ([2](#S2.E2)).
Via the Lagrangian duality, the optimal solution to equation [2](#S2.E2) is $g_{\Phi}^{*}\propto\sum_{t=1}^{T}\gamma_{t}^{*}g_{\Phi}^{t}$, where $\{\gamma_{t}^{*}\}_{t=1}^{T}$ is the solution of the following minimization problem:

$$ $\min_{\gamma_{1},\dots,\gamma_{T}}\Big{\{}\left\|\sum_{t=1}^{T}\gamma_{t}g_{\Phi}^{t}\right\|_{2}^{2}\bigg{|}\sum_{t=1}^{T}\gamma_{t}=1,\gamma_{t}\geq 0,\forall t\Big{\}}.$ (9) $$

The solution to this problem is either 0 or, given a small step size $\xi$, a descent direction that monotonically decreases all objectives at the same time and terminates when it finds a Pareto stationary point, i.e. $g_{\Phi}^{t}=0,\forall t\in\{1,\dots,T\}$. When $T=2$, problem [9](#S3.E9) can be written as $\min_{\gamma\in[0,1]}\left\|\gamma g_{\Phi}^{1}+(1-\gamma)g_{\Phi}^{2}\right\|_{2}^{2}$, which is a quadratic function of $\gamma$ with a closed form solution:

$$ $\gamma^{*}=\max\Bigg{(}\min\Big{(}\frac{(g_{\Phi}^{2}-g_{\Phi}^{1})^{\mathbf{T}}g_{\Phi}^{2}}{\left\|g_{\Phi}^{1}-g_{\Phi}^{2}\right\|_{2}^{2}},1\Big{)},0\Bigg{)}.$ (10) $$

When $T>2$, we utilize the Frank-Wolfe solver (Jaggi, ) as in Sener & Koltun (), where the analytical solution in ([10](#S3.E10)) is used as a subroutine inside the line search. We provide the full algorithm for $\gamma^{*}$ computation in Algorithm [2](#alg2) in Appendix [B](#A2).

In Algorithm [1](#alg1) and Figure [1](#S1.F1) we provide the pseudocode and an illustration of the overall search phase of MODNAS. For every mini-batch sample from $\mathcal{D}_{valid}$, we iterate over the device features $d_{t}$ (line 2), sample one scalarization $\bm{r}$ and condition the MetaHypernetwork on both $\bm{r}$ and $d_{t}$ to generate the un-normalized architectural distribution $\tilde{\alpha}_{\Phi}$ (lines 3-4). We then compute the device-specific gradient in line 6 which is used to estimate the $\gamma$ coefficients (line 7) used from MGD to update $\Phi$ (lines 8-9).
Similarly to Liu et al. (), we use the first-order approximation to obtain the best response function in the lower level (lines 10-14) and repeat the same procedure for the upper-level (lines 2-6), except now the Supernetwork weights $\bm{w}$ are updated with the mean gradient (line 15).

**Table 1: Cost of MODNAS compared to other methods. N is the number of trained architectures during search, T is the number of devices and C is the number of constraints.**
| Method | Search Cost | Pareto Set Build Cost |
| --- | --- | --- |
| LEMONADE (Elsken et al., ) | $\mathcal{O}$(NT) | $\mathcal{O}$(1) |
| ProxylessNAS (Cai et al., ) | $\mathcal{O}$(CT) | $\mathcal{O}$(1) |
| MetaD2A + HELP (Lee et al., , ) | $\mathcal{O}$(N) | $\mathcal{O}$(CT) |
| OFA (Cai et al., ) + HELP (Lee et al., ) | $\mathcal{O}$(1) | $\mathcal{O}$(CT) |
| MODNAS (Ours) | $\mathcal{O}$(1) | $\mathcal{O}$(1) |

Computational Complexity. By ignoring the cost needed to train the final architectures in the Pareto set, methods such as MetaD2A + HELP (Lee et al., , ), have a worst case time complexity of $\mathcal{O}$(CT) to build the Pareto set, i.e. the time complexity required to find the Pareto set architectures using a pretrained supernetwork or performance predictor, where T is the number of devices and C is the number of constraints. On the other hand, MODNAS reduces it to $\mathcal{O}$(1), since a single MetaHypernetwork can be conditioned on both device types and constraints. Methods such as LEMONADE (Elsken et al., ) and ProxylessNAS (Cai et al., ) apply the constraints directly during the search phase, e.g. in the supernetwork training, thus requiring an independent search run per device. Moreover, black-box methods, such as LEMONADE and MetaD2A + HELP, either train $\mathcal{O}$(NT) architectures or a surrogate based on $\mathcal{O}$(N) architectures, respectively, whereas MODNAS and OFA have a cost of $\mathcal{O}$(1) since they train a single supernetwork. Even though MODNAS iterates over T devices to compute $g_{\Phi}^{*}$ and $g_{\bm{w}}^{*}$, we show in Figure [17](#A7.F17) of Appendix [G.2](#A7.SS2) that even with 2 search devices MODNAS can generalize well on 17 test devices due to its meta-learning capabilities. See Table [1](#S3.T1) for more details.

### 4 Experiments

In this section, we firstly demonstrate the scalability and generalizability of our MODNAS approach on a NAS tabular benchmark (Section [4.1](#S4.SS1)).
Then, we validate MODNAS on larger search spaces for Machine Translation (Section [4.2](#S4.SS2)) and ImageNet classification (Section [4.3](#S4.SS3)).

Search Spaces and Datasets.
We evaluate MODNAS on 3 search spaces: (1) NAS-Bench-201 (Dong & Yang, ; Li et al., ) with 19 devices and CIFAR-10 dataset; (2) MobileNetV3 from Once-for-All (OFA) (Cai et al., ) with 12 devices and ImageNet-1k dataset; (3) Hardware-Aware-Transformer (HAT) (Wang et al., ) on the machine translation benchmark WMT'14 En-De
across 3 different hardware devices. We refer to Appendices [E](#A5) and [F](#A6) for more details.

Figure: Figure 2: Hypervolume (HV) of MODNAS and baselines across 19 devices on NAS-Bench-201. For every device, we optimize for 2 objectives, namely latency (ms) and test accuracy on CIFAR-10. For each method, metric and device we report the mean of 3 independent search runs. Higher area in the radar plot indicates better performance for every metric. The dashed black contour represents the global Pareto front. Test devices are colored in red around the plot.
Refer to caption: /html/2402.18213/assets/x2.png

Evaluation.
At test time, in order to profile the Pareto front with MODNAS on unseen devices, we sample 24 equidistant preference vectors $\bm{r}$ from the $M$-dimensional probability simplex and pass them through the pretrained MetaHypernetwork $H_{\Phi}(\bm{r},d_{t})$ to get 24 architectures. Here the device feature $d_{t}$ is obtained similarly as for the train devices. Moreover, to reduce further the Pareto set evaluation cost of training 24 architectures from scratch, we employ various proxies, such as the Sum of Training Losses (SoTL) (Ru et al., ) for NAS-Bench-201 and weight inheritance from the Supernetwork for the MovileNetV3 and HAT spaces. For details on SoTL see Appendix [C.2](#A3.SS2).

Baselines.
We compare MODNAS against two random baselines: (1) Random Search (RS), where we sample uniformly at random 24 architectures from $\mathcal{A}$ and build the Pareto front out of those, and (2) Random MetaHypernetwork (RHPN) with randomly initialized weights and evaluated with the same protocol as the MODNAS trained MetaHypernetwork. Note that RHPN is different to RS, due to the inherent implicit bias of the hypernetwork model. Finally, we also run MetaD2A + HELP (Lee et al., ) as a hardware-aware NAS method with hardware metrics' constraints. Since MetaD2A + HELP is not able to profile the Pareto front directly, we run the NAS search 24 times with different constraints, which we compute by denormalizing the same 24 equidistant preference vectors we use to evaluate MODNAS.

Metrics. To assess the quality of the Pareto set solutions, we use the hypervolume (HV) indicator, which is a standard metric in MOO.
Given a *reference point* $\rho=[\rho^{1},\dots,\rho^{m}]\in\mathbb{R}_{+}^{M}$ that is an upper bound for all objectives $\{f^{m}(\cdot;\bm{w},\alpha)\}_{m=1}^{M}$, i.e. $sup_{\alpha}f^{m}(\cdot;\bm{w},\alpha)\leq\rho^{m}$, $\forall m\in[M]$, and a Pareto set $\mathcal{P}_{\alpha}\subset\mathcal{A}$, HV($\mathcal{P}_{\alpha}$) measures the region of non-dominated points bounded above from $\rho$:

$$ $\lambda\Big{(}\big{\{}q\in\mathbb{R}_{+}^{M}\ |\ \exists\alpha\in\mathcal{P}_{\alpha}:q\in\prod_{m=1}^{M}[f^{m}(\cdot;\bm{w},\alpha),\rho^{m}]\big{\}}\Big{)},$ $$

where $\lambda(\cdot)$ is the Euclidean volume. HV can be interpreted as the total volume of the union of the boxes created by the points in the Pareto set.

#### 4.1 Simultaneous Pareto Set Learning across 19 devices

We firstly validate the scalability and learning capability of MODNAS by evaluating on the NAS-Bench-201 cell-based convolutional space. Here we want to optimize both latency and classification accuracy on all devices. We utilize the same set of 19 heterogeneous devices as Lee et al. (), from which we use 13 for search and 6 at test time. For the latency predictor, we use the one from HELP, namely a graph convolutional network (GCN), which we pretrain on the ground truth latencies on the 13 search devices as described in Section [3](#S3).
We run the MODNAS search (see Appendix [D](#A4) for more details on the search hyperparameters), as described in Algorithm [1](#alg1), for 100 epochs (22 GPU hours on a single NVidia RTX2080Ti) and show the HV in Figure [2](#S4.F2) of the evaluated Pareto front in comparison to the baselines. In addition, Figure [13](#A7.F13) and [13](#A7.F13) (in the appendix) show radar plots for two other qualitative metrics, namely, GD$+$ and IGD$+$, respectively.
Most importantly, we notice that MODNAS outperforms all the other baselines across all devices. Interestingly, the trained MODNAS attention-based MetaHypernetwork is significantly better in profiling the Pareto front than the RHPN baseline, validating our method's capability in simultaneously optimizing across multiple devices and conflicting objectives. As expected, MetaD2A + HELP, though conducting multiple search runs, focuses more on highly performant architectures, and is not able to return a diverse solution set. Finally, we find that we can decrease the evaluation cost even further via MODNAS-SoTL, by trading off the number of solutions in the Pareto set with performance (lower HV).

Figure: Figure 3: Hypervolume over search epochs of different gradient schemes in MODNAS.
Refer to caption: /html/2402.18213/assets/x3.png

MetaHypernetwork update schemes: robustness of MGD.
We now compare the MGD update scheme for the MetaHypernetwork $\Phi$ (line 9 in Alg. [1](#alg1)) against (1) the mean gradient over tasks: $\Phi\leftarrow\Phi-\xi\frac{1}{T}\sum_{t=1}^{T}g_{\Phi}^{t}$; (2) sequential updates with all single tasks' gradients: $\Phi\leftarrow\Phi-\xi g_{\Phi}^{t}$, $\forall t$; (3) single updates using gradients of MC samples over tasks: $\Phi\leftarrow\Phi-\xi g_{\Phi}^{t}$, $t\sim\{1,\dots T\}$. In Figure [3](#S4.F3) (see Figure [16](#A7.F16) in Appendix [G](#A7) for more results) we plot the HV over the search epochs for these 4 update schemes. MGD not only achieves higher final HV, but has better anytime performance and faster convergence rate compared to the other schemes.

Figure: Figure 4: HV (left) and Pareto front (right) of MODNAS and baselines on FPGA with 3 normalized objectives: error, latency and energy usage. HV was computed using the $(1,1,1)$ reference point on the right 3D plot. See Fig. [11](#A7.F11) for results on Eyeriss.
Refer to caption: /html/2402.18213/assets/figures/3d/fpga/fpga_3d_bar.png

Scalability to three objectives.
We now demonstrate the scalability of MODNAS to 3 objectives, namely, accuracy, latency and energy consumption. For this experiment we use the FPGA and Eyeriss tabular energy usage values from HW-NAS-Bench (Li et al., ). In addition to the MetaPredictor for latency, we pretrain a second predictor on the energy usage objective. We then run MODNAS with the same exact settings as for 2 objectives and compare it to RS and RHPN. Results shown in Figure [4](#S4.F4) indicate that MODNAS can scale to $M>2$ without additional search costs or hyperparameter tuning and yet achieves HV close to the global optimum front of the NAS-Bench-201 space.

Figure: Figure 5: Pareto front on Eyeriss (*left*) and HV across devices (*right*) of MODNAS ran with various latency constraints on NAS-Bench-201. See Fig. [14](#A7.F14) in Appendix [G](#A7) for all results.
Refer to caption: /html/2402.18213/assets/x5.png

MODNAS with user priors.
MODNAS can easily be extended to incorporate user priors over the multiple objectives being optimized, through the use of hardware metric constraints during search. Namely, we add a normalized constraint $c^{m}$, such that if the predicted value from the MetaPredictor during search satisfies this constraint, i.e. $p_{\theta}^{m}(\alpha_{\Phi},d_{t}^{m})\leq c^{m}$, we remove the gradient w.r.t. to that objective in lines 6 and 14 of Algorithm [1](#alg1). In Figure [5](#S4.F5) we can see that by increasing the latency constraint from 0 (all objectives being optimized) to 1 (only cross-entropy optimized), though the HV decreases, MODNAS returns Pareto sets with more performant architectures.

#### 4.2 Pareto Front Profiling on Transformer Space

Given the ever-growing size of language models it becomes of utmost importance to discover transformer variants which are efficient at inference time (latency) while remaining competitive in terms of different performance metrics.
To that end, we apply MODNAS to the hardware-aware Transformer (HAT) search space from Wang et al. () on the WMT'14 En-De (Jean et al., ; Macháček & Bojar, ) machine translation task. We pretrain the MetaPredictor (see Appendix [D.1](#A4.SS1) for details) on 2000 different architecture samples from the search space and then ran the search for 110 epochs (6 days on 8 NVidia RTX A6000) with the exact same hyperparameters as Wang et al. ().
We refer to Appendix [E](#A5) for details on the HAT search space and the search hyperparameters.
We evaluate MODNAS on all 3 devices (2 search and 1 test) and results shown in Figure [6](#S4.F6) indicate that MODNAS outperforms all baselines with a higher hypervolume of the generated Pareto set. See Figure [18](#A7.F18) for the corresponding Pareto fronts. Here, for HAT and HELP (Lee et al., ), we evaluate the architectures provided in their papers.

Figure: Figure 6: Hypervolume of MODNAS and baselines across devices on the HAT space. Leftmost plot is for the test device.
Refer to caption: /html/2402.18213/assets/x7.png

#### 4.3 Efficient MOO on ImageNet-1k starting from Pretrained Supernetworks

We now validate MODNAS on ImageNet-1k using the MovileNetV3 search space from Once-for-All (OFA) (Cai et al., ).
For this experiment, we run MODNAS using 11 search (and 1 test) devices starting with the pretrained OFA supernetwork and run the search further for 1 day on 8 RTX2080Ti GPUs. See Appendix [E](#A5) for details on the search space and Appendix [D.3](#A4.SS3) for the hyperparameters.
We use the simple MLP from Lee et al. () as our MetaPredictor, which we pretrain on 5000 sampled architecture-latency pairs. To evaluate the 24 points generated by our MetaHypernetwork and baselines, we use the OFA pretrained supernetwork weights. Results in Figure [7](#S4.F7) indicate that MODNAS achieves a higher total hypervolume across all devices, by being better than OFA + HELP (see Figure [20](#A7.F20) for the Pareto fronts), with *$\sim 13\times$ less search costs* (HELP needs $\sim 37$ GPU days to evaluate 16k architectures with the OFA supernetwork weights).

Figure: Figure 7: Hypervolume (HV) of MODNAS and baselines across 12 devices on OFA space. For every device we optimize for 2 objectives, namely latency (ms) and test accuracy on ImageNet-1k. Higher area in the radar indicates better performance for every metric. Test devices are colored in red around the radar plot.
Refer to caption: /html/2402.18213/assets/x10.png

### 5 Conclusion

In this paper, we propose a novel hardware-aware differentiable NAS algorithm for profiling the Pareto front in multi-objective problems.
In contrast to constraint-based NAS methods, ours can generate Pareto optimal architectures across multiple devices with a single hypernetwork that is conditioned on preference vectors encoding the trade-off between objectives.
Experiments across various hardware devices (up to 19), objectives (accuracy, latency and energy usage), search spaces (CNNs and Transformers), and applications (classification and machine translation) demonstrate the effectiveness and efficiency of our method.

### Acknowledgments

This research was partially supported by the following sources:
TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No
952215; the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under
grant number 417962828; the European Research Council (ERC) Consolidator Grant “Deep Learning
2.0” (grant no. 101045765). Robert Bosch GmbH is acknowledged for financial support. The authors acknowledge support from ELLIS and ELIZA. Funded by
the European Union. Views and opinions expressed are however those of the author(s) only and do
not necessarily reflect those of the European Union or the ERC. Neither the European Union nor the
ERC can be held responsible for them.

.

## Appendix

### Appendix A Extended Related Work

##### Multi-objective optimization.

Multi-objective optimization (MOO) (Gunantara, ) is a crucial field in optimization theory that tackles decision-making by simultaneously considering multiple conflicting objectives.
Two main categories of MOO have developed: those that use gradients and those that don't.
Gradient-free MOO approaches include
evolutionary algorithms, specifically dominance-based techniques like NSGA-II (Deb et al., ), but are often sample inefficient and ill-suited for deep learning problems.

The other form of MOO algorithms employs gradients.
The early gradient-based work of Désidéri () has been extended in various settings, particularly multi-task learning, with great promise (Sener & Koltun, ; Lin et al., ; Mahapatra & Rajan, ; Liu & Vicente, ), but these approaches are applied to mainly a fixed architecture and extending them to a search space of architectures is non-trivial. Applying these to a search space would correspond to retraining each architecture in the space with multiple objectives, which is prohibitively expensive for larger search spaces.
Another challenge of MOO is balancing the different objectives; so solutions to address this have proposed to incorporate preference vectors which dictate how to favor the multiple objectives on the Pareto-Front (Ye & Liu, ; Momma et al., ).
Another line of work has developed which tries to alleviate the retraining problem by using hypernetworks.
In the MOO case, hypernetworks determine the weights of the main network (Lin et al., ), with many using preference vector techniques (Navon et al., ; Hoang et al., ; Phan et al., ).

##### Neural Architecture Search.

A key challenge for the automatic design of neural network architectures is devising methods that can efficiently explore large search spaces. Initial approaches to NAS used Reinforcement Learning (Zoph & Le, ), evolutionary algorithms (Deb et al., ; Lu et al., ; Elsken et al., ), or other black box optimization techniques (Daulton et al., ) to train and evaluate a large number of neural network architectures from scratch. One-shot NAS introduced weight sharing between architectures by training a single over-parameterized network called a supernet, or one-shot model, to speed up the performance evaluation of individual networks in the search space (Saxena & Verbeek, ; Bender et al., ; Pham et al., ; Liu et al., ). Differentiable one-shot NAS methods (Wu et al., ; Cai et al., ; Wu et al., ; Fu et al., ) introduce a continuous relaxation of the search space to search for optimal sub-models of the supernet using gradient descent (Wu et al., ; Cai et al., ; Wu et al., ; Fu et al., ). In contrast, two-stage NAS methods first train a supernet, e.g. by random sampling of subnetworks, and in a second stage use black-box optimization to search for optimal subnetworks (Bender et al., ; Li & Talwalkar, ; Guo et al., ).

##### Hardware-aware and multi-objective Neural Architecture Search.

While early NAS approaches focused on achieving high accuracy on a target task, hardware-aware NAS aims to also optimize architectures to run efficiently on a target hardware device (Benmeziane et al., ; Zhang et al., ; Lee et al., ; Shaw et al., ), which naturally leads to multi-objective NAS (Hsu et al., ; Kim et al., ; Tan et al., ).
Two-stage NAS approaches, which involve a training and a search stage, can be easily adapted to this setting by using a multi-objective search procedure in the second stage (Cai et al., ; Ito & Von Zuben, ).
However, most two-stage methods rely on random sampling during the supernet training phase, which does not favor promising architectures.
Differential neural architecture search methods like (Wu et al., ; Cai et al., ; Wu et al., ; Fu et al., ; Xu et al., ; Jiang et al., ; Wang et al., ) use latency proxies like layer-wise latencies, FLOPS (Dudziak et al., ) etc. to calculate the hardware objective and a scalarization of task and hardware objective with a fixed weighting, resulting in a single optimal solution.
However, re-calculating the solution for a different weighting of the objectives requires repeating the search, making it computationally expensive.
In contrast to these approaches, we propose a search algorithm that provides the user with the entire Pareto-Front of objectives in a single run.
While this work focuses on multi-objective NAS for hardware constraints, there are applications to other objectives like fairness (Martinez et al., ; Dooley et al., ; Das & Dooley, ); applying our technique to these approaches is worthy future work.

### Appendix B Frank-Wolfe Solver

In this section, we provide the pseudocode of the Frank-Wolfe solver (Jaggi, ) used to compute the gradient coefficients used for the MGD updates. To solve the constrained optimization problem, the Frank-Wolfe solver uses analytical solution for the line search with $T=2$ (Algorithm [3](#alg3)).

Figure: Algorithm 2 $\mathtt{FrankWolfeSolver}$

Figure: Algorithm 3 Solver $\min_{\delta\in[0,1]}{||\delta\theta+(1-\delta)\bar{\theta}||}_{2}^{2}$

### Appendix C Evaluation Details

#### C.1 Other Metrics

For NAS-Bench-201, in addition, we evaluate the *generational distance* (GD) and *inverse generational distance* (IGD) (see Appendix [C](#A3)). See Figure [13](#A7.F13) for the results complementary to the hypervolume radar plot in Figure [2](#S4.F2) of the main paper.

##### Generational Distance ( G ​ D 𝐺 𝐷 GD ) and Inverse Generational Distance ( I ​ G ​ D 𝐼 𝐺 𝐷 IGD ).

Given a *reference set* $\mathcal{S}\subset\mathcal{A}$ and a Pareto set $\mathcal{P}_{\alpha}\subset\mathcal{A}$ with $dim(\mathcal{A})=K$, the GD indicator is defined as the distance between every point $\alpha\in\mathcal{P}_{\alpha}$ and the closest point in $s\in\mathcal{S}$, averaged over the size of $\mathcal{P}_{\alpha}$:

$$ $GD(\mathcal{P}_{\alpha},\mathcal{S})=\frac{1}{|\mathcal{P}_{\alpha}|}\bigg{(}\sum_{\alpha\in\mathcal{P}_{\alpha}}\min_{s\in\mathcal{S}}d(\alpha,s)^{2}\bigg{)}^{1/2},$ $$

where $d(\alpha,s)=\sqrt{\sum_{k=1}^{K}(\alpha_{k}-s_{k})^{2}}$ is the Euclidean distance from $\alpha$ to its nearest reference point in $\mathcal{S}$.

The inverted generational distance (IGD) is computed as $IGD(\mathcal{P}_{\alpha},\mathcal{S})=GD(\mathcal{S},\mathcal{P}_{\alpha})$.

##### Generational Distance Plus ( G ​ D + 𝐺 superscript 𝐷 GD^{+} ) and Inverse Generational Distance Plus ( I ​ G ​ D + 𝐼 𝐺 superscript 𝐷 IGD^{+} ).

$GD^{+}(\mathcal{P}_{\alpha},\mathcal{S})=IGD^{+}(\mathcal{S},\mathcal{P}_{\alpha})$ replaces the euclidean distance $d(\alpha,s)$ in GD with:

$$ $d^{+}(\alpha,s)=\sqrt{\sum_{k=1}^{K}(\max\{\alpha_{k}-s_{k},0\})^{2}}$ $$

Figure: Figure 8: Overview of the MetaHypernetwork architecture in the case of $M$ objectives. Note that $m=1$ is reserved for the accuracy objective which we model through the cross-entropy loss in the Supernetwork. The initial embedding layer $e_{\phi_{0}}$ gets the $d_{t}$ hardware embedding and outputs a weight that scales each of the $K$ hypernetworks' (orange boxes) outputs from the hypernetwork bank. The scaled architectural parameters are then summed up element-wise. All individual hypernetwork $h_{\phi_{k}}$ get as input the same scalarization $\bm{r}$. Each of them has $M-1$ embedding layers with dimensions $n_{m}\times\frac{dim(\mathcal{A})}{M-1}$, $\forall m\in\{2,\dots,M\}$ that gets as input the scalarizations for objectives $m=2,\dots,m=M$, and yields a vector of size $\frac{dim(\mathcal{A})}{M-1}$. The output from the $M-1$ embedding layers are concatenated to give the architecture encoding $\tilde{\alpha}$.
Refer to caption: /html/2402.18213/assets/x11.png

#### C.2 MODNAS-SoTL

On the NAS-Bench-201 search space, since the architectures evaluated with the supernetwork weights are not highly correlated to the ones trained independently from scratch, we employ the Sum of Training Losses (SoTL) proxy from Ru et al. ().
To profile the Pareto front with SoTL, we firstly evaluate the 24 architectures using the exponential moving average of the sum of training losses for the initial 12 epochs of training as $\sum_{e=1}^{12}0.9^{12-e}\mathcal{L}^{train}(\bm{w},\alpha)$, and then train from scratch only the subset of architectures in the Pareto set built using the SoTL evaluations.

### Appendix D Experimental Details

#### D.1 MetaPredictor Architectures

For all search spaces we set the dimensionality of the hardware embedding to 10. This corresponds to latency evaluations on a set of 10 reference architectures, which are the same used by Lee et al. ().

##### NAS-Bench-201.

For the NAS-Bench-201 (Dong & Yang, ) search space we use a Graph Convolutional Network (GCN) as proposed in Dudziak et al. (). Furthermore, in addition to the one-hot operation encoding and adjacency matrix corresponding to the architecture cells, we also input the hardware embedding to this predictor, as done by Lee et al. (). The number of nodes in the GCN is 8 and the dimensionality of the layers is set to 100 following HELP (Lee et al., ).

##### MobileNetV3 (OFA).

Following HELP (Lee et al., ), in the MobileNetV3 search space, we use a simple feedforward neural network. The input dimension of the MetaPredictor is set $160$, equivalent to the dimension of the concatenated architecture encoding. We set the size of the hidden layers to 100. Precisely, the MetaPredictor contains 2 linear layers with ReLU activation to process the input one-hot architecture encoding of dimension 160, and 2 linear layers to process the hardware embedding. The outputs from these two paths are finally concatenated and passed through a final linear layer to get the predicted latency.

##### Seq-Seq Transformer (HAT).

HELP (^3^33[https://github.com/HayeonLee/HELP](https://github.com/HayeonLee/HELP)) does not release the architecture or the meta-learned pretrained predictor for HAT(Wang et al., ). However, HAT (^4^44[https://github.com/mit-han-lab/hardware-aware-transformers](https://github.com/mit-han-lab/hardware-aware-transformers)) releases code and pretrained models for each of the devices and tasks trained independently. Hence, we build our single per-task MetaPredictor based on the architecture of the HAT predictor, i.e. a simple feedforward neural network. The input dimensionality corresponds to the dimensionality of the one-hot architecture encoding of the candidate Transformer architecture. In addition to this, since we condition on the hardware embedding choice, we add 2 additional linear layers to process the hardware embedding, which is then concatenated with the processed architecture encoding to output the final latency prediction. We set the hidden dimension of the MetaHypernetwork to 400 and the number of hidden layers to 6. The input feature dimension of the predictor is 130.

#### D.2 MetaHypernetwork Architecture

We keep the MetaHypernetwork architecture similar across search spaces. The only thing we adapt is the output dimensionality of the hypernetwork (in the hypernetwork bank of MetaHypernetwork), which corresponds to the dimensionality of the architecture parameters of the respective search space. We set the size of the initial hardware embedding layer and the hypernetwork bank to 50 for all search spaces. Furthermore, each hypernetwork has 100 possible learnable embeddings $e^{m}$, for every objective $m\in\{2,\dots,M\}$, to map the scalarization vector to an architecture. See Figure [8](#A3.F8) for an illustration of the MetaHypernetwork architecture.

For the NAS-Bench-201 search space, we use a single embedding layer of dimensionality 30, i.e. corresponding to the dimensionality of the architecture space: $6\times 5$ (6 edges and 5 operation choices on each edge). For the experiment with 3 objectives, we have an additional embedding of for the energy usage objective, which we concatenate with the latency embedding before passing it to the MetaHypernetwork. Moreover, the individual hypernetworks in the MetaHypernetwork bank, now have 2 embedding layers with dimensionality 15. Their output is concatenated to match again with the architecture space dimensions.

In the MobileNetV3 space, we use 4 different embedding layers – for the depth, expansion ratio, kernel size and resolution. The number of blocks in this space is set to 5 and hence the depth embedding layer has a dimensionality of $5\times 3$ i.e. 3 depth choices per block. Furthermore, the kernel and expansion embedding layers have a dimension of $5\times 4\times 3$, for 5 blocks, each with a maximum depth of 4 and 3 possible kernel size or expansion ratio choices. Finally, we also have a resolution embedding layer of dimension 25 for the 25 possible resolution choices.

For the Seq-Seq Transformer (HAT) space, we use 9 embedding layers in the individual hypernetworks of the MetaHypernetwork (the number of layers in the encoder is fixed; see Table [4](#A6.T4)):

- •
2 embedding layers of size 2 (corresponding to the encoder and decoder blocks) to map the scalarization to the embedding dimension architecture parameter. The chosen embedding size is held constant throughout the encoder or decoder block.
- •
2 embedding layers of dimensions $6\times 3$ (where 6 corresponds to the maximum possible number of encoder or decoder layers and 3 the number of choices) for the linear layer size in every attention block for both the encoder and the decoder.
- •
2 embedding layers, with size $6\times 2$, for the number of heads in every attention block.
- •
1 embedding layer of size $6$ to encode the $6$ possible choices for the number of layers in the decoder.
- •
1 embedding layer of size $6\times 3$ (6 possible encoder layers and 3 possible choices), corresponding to the arbitrary encoder layer choice for attention.
- •
1 embedding layer of size $6\times 2$ (6 possible encoder layers and 2 possible choices) for the number of heads in the encoder-decoder attention.

#### D.3 MODNAS Hyperparameter Configurations

In Table [2](#A4.T2), we show the search hyperparameters and their corresponding values we use to conduct our experiments with MODNAS. For the convolutional spaces we subtract a cosine similarity penalty from the scalarized loss following (Ruchte & Grabocka, ):

$$ $g_{\Phi}^{t}\leftarrow\bm{r}^{\mathbf{T}}\nabla_{\Phi}\mathbf{L}_{t}(\mathcal{D}_{valid},\bm{w},\alpha_{\Phi})-\lambda\nabla_{\Phi}\frac{\bm{r}^{\mathbf{T}}\mathbf{L}_{t}(\mathcal{D}_{valid},\bm{w},\alpha_{\Phi})}{||\bm{r}||\hskip 2.84526pt||\mathbf{L}_{t}(\mathcal{D}_{valid},\bm{w},\alpha_{\Phi})||},$ (11) $$

where $||\cdot||$ is the $l_{2}$ norm. We set $\lambda$ to 0.001. Empirically we did not observe significant differences on disabling the cosine penalty term.

**Table 2: Hyperparameters used on different search spaces**
| Search Space | Hyperparameter Type | Value |  |
| --- | --- | --- | --- |
| NAS-Bench-201 | MetaHypernetwork | learning rate | 3e-4 |
| weight decay | 1e-3 |  |  |
| embedding layer size | 100 |  |  |
| hypernetwork bank size | 50 |  |  |
| optimizer | Adam |  |  |
| ReinMax temperature | 1 |  |  |
| Supernetwork | learning rate | 0.025 |  |
| momentum | 0.9 |  |  |
| weight decay | 0.0027 |  |  |
| learning rate scheduler | cosine |  |  |
| epochs | 100 |  |  |
| batch size | 256 |  |  |
| gradient clipping | 5 |  |  |
| cutout | true |  |  |
| cutout length | 16 |  |  |
| initial channels | 16 |  |  |
| optimizer | SGD |  |  |
| train portion | 0.5 |  |  |
| MobileNetV3 (OFA) | MetaHypernetwork | learning rate | 1e-5 |
| weight decay | 1e-3 |  |  |
| embedding layer size | 100 |  |  |
| hypernetwork bank size | 50 |  |  |
| optimizer | Adam |  |  |
| ReinMax temperature | 1 |  |  |
| Supernetwork | learning rate | 1e-3 |  |
| momentum | 0.9 |  |  |
| weight decay | 3e-5 |  |  |
| learning rate scheduler | cosine |  |  |
| epochs | 50 |  |  |
| batch size | 32 |  |  |
| bn_momentum | 0.1 |  |  |
| bn_eps | 1e-5 |  |  |
| dropout | 0.1 |  |  |
| width | 1.2 |  |  |
| optimizer | SGD |  |  |
| train portion | 1.0 |  |  |
| Seq-Seq Transformer (HAT) | MetaHypernetwork | learning rate | 3e-4 |
| weight decay | 1e-3 |  |  |
| embedding layer size | 100 |  |  |
| hypernetwork bank size | 50 |  |  |
| optimizer | Adam |  |  |
| ReinMax temperature | 1 |  |  |
| Supernetwork | learning rate | 1e-7 |  |
| momentum | 0.9 |  |  |
| weight decay | 0.0 |  |  |
| learning rate scheduler | cosine |  |  |
| epochs | 110 |  |  |
| batch size/max-tokens | 4096 |  |  |
| criterion | label_smoothed_cross_entropy |  |  |
| attention-dropout | 0.1 |  |  |
| dropout | 0.3 |  |  |
| precision | float32 |  |  |
| optimizer | Adam |  |  |
| train portion | 1.0 |  |  |

#### D.4 Normalization of objectives

Since our method relies on a scalarization of different objectives, it is important that the objectives being optimized are on the same scale. For simplicity, lets consider the scenario where the two objectives of interest are the cross-entropy loss and latency. Since we pretrain and freeze our MetaPredictor, the latency-scale remains constant throughout the search, while the cross-entropy loss of the Supernetwork (likely) decreases over time. To this end, we use the following max-min normalization to normalize the objectives:

$$ $\mathcal{L}_{t}^{m}(\cdot,\alpha_{\Phi})=\frac{\mathcal{L}_{t}^{m}(\cdot,\alpha_{\Phi})-\min(\mathbf{\bar{L}})}{\max(\mathbf{\bar{L}})-\min(\mathbf{\bar{L}})},$ (12) $$

where $\mathbf{\bar{L}}=\bigcup_{i=1}^{N}\mathtt{stop_g}\big{(}\mathcal{L}_{t}^{m}(\cdot,\alpha_{i})^{i}\big{)}$ is the set of losses evaluated on $N$ architectures and potentially $N$ previous steps. For the latency objective, we precompute these sample-statistics using N samples (ground-truth for NAS-Bench-201 and predicted for OFA and HAT spaces) from the search space, whilst for the cross-entropy loss we compute them throughout the search. Furthermore, to take into account the decreasing cross-entropy, we reset the cross-entropy loss statistics after every epoch.

### Appendix E Details on Search Spaces

NAS-Bench-201 (Dong & Yang, ) is a convolutional, cell-based search space. The search space consists of 3 stages, each with number of channels 16, 32 and 64, respectively. Each stage contains a convolutional cell repeated 5 times. Here, every cell is represented as a directed acyclic graph (DAG) which has 4 nodes, densely connected with 6 edges. Each edge has 5 possible operation choices: a skip connection, a zero operation, a 3$\times$3 convolution, a 5$\times$5 convolution or an average pooling operation. NAS-Bench-201 is a tabular benchmark exhaustively constructed, where the objective is finding the optimal cell for the given macro skeleton.

MobileNetV3 proposed in OFA (Cai et al., ) is a macro convolutional search space. The different searchable dimensions in the search space are the depth (per block), the kernel size (for every layer in every block) and the channel expansion ratio (for every layer in every block). There are a total of 5 blocks, each with 3 possible depth choices and every layer in this block has 3 possible kernel sizes and channel expansion ratio choices. This amounts to a total search space size of $((3\times 3)^{2}+(3\times 3)^{3}+(3\times 3)^{4})^{5}\approx 2\times 10^{19}$. Additionally, every architecture has 25 possible choices for the size of the input resolution. The 3 possible choices for depth, kernel size and expansion ratio are $\{2,3,4\}$, $\{3,5,7\}$ and $\{3,4,6\}$, respectively. The input resolution choices are $\{128,132,136,140,144,148,152,156,160,164,168,172,176,180,184,188,192,196,200,204,208,212,216,220,224\}$. We use a width factor of 1.2 similar to OFA (Cai et al., ).

Seq-Seq Encoder-Decoder Transformer (HAT) (Wang et al., ) for the En-De machine translation task has a searchable number of layers, embedding dimension, feedforward expansion layer dim per-layer, number of heads per-layer for both the encoder and the decoder sub-modules. In addition to this, the number of encoder layers the decoder attends to, and the number of attention heads in the encoder-decoder attention is also searchable. We present the details of the search space in Table [4](#A6.T4).

**Table 3: Search-test split for hardware devices and datasets for different search spaces.**
| Search Space | Train-devices | Test devices | Dataset |
| --- | --- | --- | --- |
| NAS-Bench-201 | 1080ti_1, 1080ti_32, 1080ti_256, silver_4114, | titan_rtx_256, gold_6226, | CIFAR10 |
| silver_4210r, samsung_a50, pixel3, essential_ph_1, | fpga, pixel2, |  |  |
| samsung_s7, titanx_1, titanx_32, titanx_256, gold_6240 | raspi4, eyeriss |  |  |
| MobileNetV3 (OFA) | 2080ti_1, 2080ti_32, 2080ti_64, titan_xp_1, | titan_rtx_64 | ImageNet-1k |
| titan_xp_32, titan_xp_64, v100_1, v100_32, |  |  |  |
| v100_64, titan_rtx_1, titan_rtx_32 |  |  |  |
| Seq-Seq Transformer (HAT) | titanxp gpu, cpu xeon | cpu raspberrypi | WMT14.en-de |

### Appendix F Datasets and Devices

In this section, we describe the suite of hardware devices and tasks we use to evaluate MODNAS throughout the paper. We evaluate our methods across small- and large-scale image classification datasets, such as CIFAR10 and ImageNet1-k. Moreover, for the machine translation task, we evaluate our method on the WMT'14 En-De dataset (Macháček & Bojar, ). Finally, we evaluate MODNAS across 19 devices on NAS-Bench-201, 12 devices on MobileNetV3 and 3 devices on Seq-Seq Transformer, with zero-shot generalization to test devices. See Table [3](#A5.T3) for the list of devices used. We refer the reader to Lee et al. (), Cai et al. (), Wang et al. () and Li et al. () for more details on the devices.

**Table 4: Encoder-Decoder Search Space for HAT.**
| Module | Searchable Dim | Choices |
| --- | --- | --- |
| Encoder | No. of Layers | [6] (fixed) |
|  | Embedding dim | [640, 512] |
|  | No. of heads | [8, 4] |
|  | FFN dim | [3072, 2048, 1024] |
| Decoder | No. of layers | [6, 5, 4, 3, 2, 1] |
|  | Embedding dim | [640, 512] |
|  | No. of heads | [8, 4] |
|  | FFN dim | [3072, 2048, 1024] |
|  | Arbitrary-Encoder-Layer | [-1, 1, 2] |
|  | Enc-Dec attention num heads | [8, 4] |

### Appendix G Additional Experiments

#### G.1 Predicted v/s Ground-Truth Latencies

In Figure 8, we present the scatter plots of the predictions of our hardware-aware MetaPredictor vs. the ground-truth latencies of different architectures. In the figure title we also report the kendall-tau correlation coefficient for every device. As observed, our predictor achieves high kendall-$\tau$ correlation coefficient across all devices.

Figure: Figure 9: Scatter plots of predicted latencies from our pretrained MetaPredictor vs. ground-truth latencies (test devices in red).
Refer to caption: /html/2402.18213/assets/x12.png

Figure: (a) Hypervolume
Refer to caption: /html/2402.18213/assets/x31.png

Figure: Figure 11: HV (left) and Pareto front (right) of MODNAS and baselines on Eyeriss with 3 normalized objectives: error, latency and energy usage. HV was computed using the $(1,1,1)$ reference point on the right 3D plot.
Refer to caption: /html/2402.18213/assets/figures/3d/eyeriss/eyeriss_3d_bar.png

#### G.2 Additional Results on NAS-Bench-201

In Figure [12](#A7.F12), we present the Pareto fronts obtained by our method in comparison to different baselines on the NAS-Bench-201 search space. In Figure [13](#A7.F13), we present different additional metrics, such as GD and IGD (see Section [C](#A3)), to evaluate the quality of the Pareto fronts obtained on NAS-Bench-201. Figure [14](#A7.F14) presents the Pareto front MODNAS yields when applying different latency constraints during the search phase. Figure [10](#A7.F10) compares our method using the ReinMax gradient estimator to the GDAS estimator (Dong & Yang, ). As we can see, ReinMax obtains a qualitatively better hypervolume coverage compared to GDAS. Figure [11](#A7.F11) presents the 3D Pareto front and hypervolume obtained by MODNAS compared to other baselines when optimizing for accuracy, latency and energy usage on NAS-Bench-201. Figure [16](#A7.F16) presents the comparison of MODNAS with MGD to other gradient aggregation schemes, such as mean, sequential and MC sampling (see Section [4.1](#S4.SS1)), across multiple hardware devices. Finally, in Figure [17](#A7.F17) we present the robustness of MODNAS to the fraction of devices used for the predictor training and the search phase.

Figure: Figure 12: Pareto fronts of MODNAS and baselines on NAS-Bench-201. MODNAS-SoTL is not shown for better visibility.
Refer to caption: /html/2402.18213/assets/x34.png

Figure: (a) GD
Refer to caption: /html/2402.18213/assets/x53.png

Figure: Figure 14: Pareto fronts of MODNAS ran with different latency constraints during search.
Refer to caption: /html/2402.18213/assets/x57.png

Figure: (a) GD
Refer to caption: /html/2402.18213/assets/x76.png

Figure: Figure 16: HV over time on NAS-Bench-201 of MODNAS with different gradient update schemes.
Refer to caption: /html/2402.18213/assets/x80.png

Figure: Figure 17: HV over time on NAS-Bench-201 of MODNAS with different number of devices during search. For number of devices less than 13 (default one) we randomly select a subset from these 13 devices.
Refer to caption: /html/2402.18213/assets/x99.png

#### G.3 Additional Results on Hardware-aware Transformers (En-De)

We show the Pareto fronts of MODNAS compared to baselines for the Transformer space in Figure [18](#A7.F18), as well as their comparison with respect to hypervolume for the SacreBLEU metric in Figure [19](#A7.F19). These results demonstrate the superior performance of our method compared to the other baselines on this benchmark. All evaluations are done by inheriting the weights of a pretrained supernet.

Figure: Figure 18: Pareto fronts of MODNAS and baselines on HAT space.
Refer to caption: /html/2402.18213/assets/x118.png

Figure: Figure 19: Hypervolume across devices on the HAT search space.
Refer to caption: /html/2402.18213/assets/x124.png

#### G.4 Additional Results on MobileNetV3

In Figure [20](#A7.F20), we present the Pareto fronts of our method compared to different baselines for 12 different hardware devices on the MobileNetV3 space. Additionally, in Figure [21](#A7.F21) and [22](#A7.F22) we show the Pareto fronts of MODNAS ran with 0.2 latency constraint (similar to the "MODNAS with user uriors" experiment on NAS-Bench-201 in Section [4.1](#S4.SS1)) and where we only update the MetaHypernetwork parameters during search and freeze the Supernetwork weights, respectively.

Figure: Figure 20: Pareto fronts of MODNAS and baselines on the MobileNetV3 space.
Refer to caption: /html/2402.18213/assets/x127.png

Figure: Figure 21: Pareto fronts of MODNAS with 0.2 latency constraint during search and baselines on the MobileNetV3 space.
Refer to caption: /html/2402.18213/assets/x139.png

Figure: Figure 22: Pareto fronts of MODNAS and baselines on the MobileNetV3 space. Here, when running MODNAS we only update the MetaHypernetwork parameters and freeze the Supernetwork ones.
Refer to caption: /html/2402.18213/assets/x151.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: __1-__ Using hypernetwork for the NAS and prate-frontier learning is well known. 

That limits the novelty of the proposed method.

[A]Brock, A., Lim, T., Ritchie, J., and Weston, N. (2018). SMASH: One-shot model architecture search through hypernetworks. In the International Conference on Learning Representations

[B[ Lorraine, Jonathan, and David Duvenaud. "Stochastic hyperparameter optimization through hypernetworks." arXiv preprint arXiv:1802.09419 (2018).

[C] Pan, Z., Liang, Y., Zhang, J., Yi, X., Yu, Y., and Zheng, Y. (2018). Hyperst-net: Hypernetworks for spatio-temporal forecasting. arXiv preprint arXiv:1809.10889

[D]  Zhang, C., Ren, M., and Urtasun, R. (2019). Graph hypernetworks for neural architecture search. In International Conference on Learning Representations.
 
However, there are major concerns about the performance, initialization, and scalability as stated here (Section 6 in the following paper):
Chauhan, Vinod Kumar, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A. Clifton. "A brief review of hypernetworks in deep learning." Artificial Intelligence Review 57, no. 9 (2024): 250.


__2-__ Hypernrtwork is not performing well on unseen networks.
https://arxiv.org/abs/2110.13100
With that, Could you elaborate on any specific constraints or limitations when transferring MODNAS to devices significantly different from the ones used in the training phase?


__3-__ MiLeNAS paper (CVPR’20) shows that the gradient errors caused by approximations of second-order methods in bi-level optimization results in suboptimality, in the sense that the optimization procedure fails to converge to a (locally) optimal solution.  However, it seems that the authors still use a bi-level optimization technique.  It seems using advanced techniques like MiLeNAS might be more useful.
 
__4-__ Can the MODNSD be applied to object detection tasks as well?

__5-__ MobileNet search space is pretty small DNN. How does it work on more complex DNN search space?  

__6-__ The paper mentions scalability but doesn’t mention potential bottlenecks. For example, are there search space complexities that MODNAS struggles with?



__7-__ The paper doesn't provide the code. That limits reproducibility of the proposed method.



__8-__ I am not sure how accurate is the energy model that the paper is using considering even Eyeriss is a pretty old paper (2016).  The authors need to provide more details about energy modeling. 

__9-__ I think it is better to use power rather than energy. Considering energy is power x  latency, when you minimize the latency, if the power is fixed, energy is minimized automatically. Considering power, accuracy, and latency can be a better metric.


__10-__ The paper employs the Frank-Wolfe solver for optimizing scalarizations. Could you discuss any trade-offs in this choice, and if other optimizers were considered?

__11-__ Given that preference vectors affect the optimization landscape, an analysis of how different preference configurations influence the final architectures would clarify MODNAS’s versatility.


__12-__ MODNAS is tested for three objectives—accuracy, latency, and energy. Could this method scale effectively if additional objectives were introduced, or would this necessitate modifications?

__13-__ The approach appears tailored to devices with GPU-like architectures. It would be valuable to understand how MODNAS performs on other architectures, like mobile hardware.




__14-__ Minor: 

A)The paper is pretty dense. I hardly can read the legends in Figures 7, 8.

B)Minor Typo:
Pareto front profiling in multi-objective optimization (MOO), i.e. finding a diverse set of Pareto optimal solutions, is challenging" – A comma after "i.e." would improve readability: "i.e., finding a diverse set..."

"To search across devices, we frame the problem as a multi-task multi-objective optimization problem" – Add a comma, “multi-task, multi-objective optimization problem.”
soundness: 2
strengths: 1-The method is adaptable to a range of devices by conditioning the hypernetwork on device embeddings, making it highly versatile for deployment on diverse hardware.

2-MODNAS is tested on various hardware devices and tasks, including image classification, machine translation, and language modeling, showcasing its applicability across multiple domains.
summary: The paper (MODNAS) presents an approach to Neural Architecture Search (NAS) that balances competing objectives—like performance, latency, and energy efficiency—across multiple hardware devices. By encoding user preferences as a scalarization vector, MODNAS efficiently searches for Pareto-optimal solutions across diverse devices in a single run. The method employs a hypernetwork to generate architectures for specific hardware configurations, leveraging multiple gradient descents for optimization.  The method has been evaluated on MobileNetV3 on ImageNet-1k, an encoder-decoder transformer space for machine translation, and a decoder-only space for language modeling.
weaknesses: 1-Using Hypernetworks for NAS is well known but doesn’t seem a promising solution. It is like a heuristic solution.

2- I think energy and latency are not necessarily conflicting metrics.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - How are user preference and hardware device embeddings designed?
- Please explain how $\textbf{Architect}$ makes architecture parameters differentiable and propagate them to the $\textbf{MetaHypernetwork}$.
- In Figure 4, solutions on the Pareto frontier can be achieved by modulating user preferences. However, in HDX [1], which is a one-shot NAS with hardware constraints, claims that modulating hyperparameters linearly doesn’t lead to linearly distributed results. Is $\textbf{MetaHypernetwork}$ free from this problem? Can real experimental results be plotted like Figure 4 to substantiate the integrity of $\textbf{MetaHypernetwork}$?
- To search network architectures with other works, the paper sets the search time budget up to 2.5 times compared to that of MODNAS. (Or fixed time budget, e.g., 192 hours) That is, MODNAS outperforms prior works in terms of search time, and the quality of solutions is better than that of others under given time budgets. What the review wonders is whether can other works find near-GT solutions with a larger time budget. If a target network is distributed extensively and used frequently, huge search costs can be tolerable if there are better solutions.

[1] Deokki Hong, Kanghyun Choi, Hye Yoon Lee, Joonsang Yu, Noseong Park, Youngsok Kim, and Jinho Lee. Enabling Hard Constraints in Differentiable Neural Network and Accelerator Co-Exploration. In Proceedings of the 59th ACM/IEEE Design Automation Conference, 2022.
soundness: 3
strengths: - The paper proposes to execute one-shot NAS while search networks satisfy multiple objectives about accuracy and hardware efficiency with target hardware.
- The paper analyzes the proposed method with various aspects such as efficacy, and robustness of the training process.
- The paper provides extensive experiments and visualizations to support the proposal and its analysis.
summary: This paper proposes a Neural Architecture Search (NAS) algorithm that optimizes several performance metrics on various hardware.
The paper adopts Multi Objective Optimization (MOO) to optimize multiple objectives simultaneously.
The key idea is to train $\textbf{MetaHypernetwork}$ which takes hardware features and user preferences about performance metrics as inputs and returns optimized network architectures as outputs.
To train $\textbf{MetaHypernetwork}$, the paper exploits $\textbf{Supernetwork}$ for NAS and $\textbf{MetaPredictor}$ for optimization in terms of efficiency.

$\textbf{MetaHypernetwork}$ takes user preferences and device features, and returns network architecture designs.
Then, $\textbf{Architect}$ makes those architecture designs into differentiable architecture parameters.
With these architecture parameters, $\textbf{Supernetwork}$ computes accuracy-related loss of the given network design.
Also, $\textbf{MetaPredictor}$ computes efficiency-related loss with the network architecture under the given hardware features.
Further, the paper proposes to update $\textbf{MetaHypernetwork}$ using Multiple Gradient Descent (MGD) to get optimized network architectures that satisfy multiple objectives concurrently.

For NAS, the proposed method is adaptable to diverse pretrained $\textbf{Supernetwork}$.
For efficiency, $\textbf{MetaHypernetwork}$ can be updated with a lot of hardware-related loss functions at once.
For optimization, the paper achieves Pareto frontier solutions that are hard to reach with sequential or averaged gradient updates.
weaknesses: - It may be hard to regulate the trade-off among user preferences with scalarization. Figure 4 can be a support, but it is just an abstract depiction, not experimental results.
- The proposed method can help search optimized network architectures quickly at low cost. However, network architectures the same as or near ground truth solutions may hard to be reach with $\textbf{MetaHypernetwork}$, where other works can reach with huge search costs.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - What assumptions do the MetaPredictors make about the underlying devices when estimating latency and energy requirements? For example, do they assume a particular choice of operating system and process scheduler used, other parallel and system processes running, core utilisation of the DNN inference, runtime library, or HW design used to execute the DNN on the target device?

- Especially on smaller systems, memory requirements are often a major bottleneck of DNN inference: Could the approach proposed by the authors be extended to include memory? So far, the evaluation seems to focus only on accuracy, latency and energy.

- How exactly are "preference vector" and a "hardware device feature vector" defined? The authors cover many different types of HW in their evaluation: GPUs (e.g. 1080 ti), smarthpones (e.g. pixel 2, 3), edge devices (e.g. raspi 4) and dedicated HW accelerators (eyeriss, FPGA), which can be characterised by a number of different metrics, often exclusive to the type of HW considered. (e.g. #cores, processor speed, RAM, ... for the raspi4 vs. #streaming cores, VRAM, bus interface, ... for the 1080 ti vs. #LUTs, #BRAM, ... for the FPGA). So it would be really interesting to know how the authors put these metrics into relation and unified them into one vector.

- Which FPGA did the authors use? Since an FPGA is just freely programmable hardware, it would also be interesting to know which DNN accelerator design the authors implemented on the FPGA to perform their evaluation.

- Fig. 5: Why is MODNAS shown as an upper threashold line (+/- variance)? Based on the caption, I would have expected to see an HVI curve that stops after 24 evaluations, similar to the other curves in the plot.
soundness: 3
strengths: - The extension of existing zero-shot NAS techniques with Hypernetworks for HW aware NAS is motivated well and contextualized nicely for alredy existing techniques. 
- The authors provide an extensive evaluation for different applications (language modeling, vision, translation), multiple well known NAS searchspaces, and different target spaces (2-3 dimensional).
summary: The authors propose a multi-objective HW aware NAS algorith that can emit optimized DNN architectures across multiple different target devices using a single search run.
To overcome the challange of having to solve multiple different optimization runs for each considered HW platform, the authors propose using a hypernetwork to generate an architectural distribution across multiple devices based on a "preference vector" and a "hardware device feature vector" from which discrete architectures can then be sampled.
For objective function evaluators, the authors use a Supernetwork as a standin for a accuracy and "MetaPredictors", one for each target device, to estimate hardware metrics.
weaknesses: - The authors do not show how the proposed DNN architectures would actually perform on the different target systems. As far as I understand it, the HV results shown are calculated using the estimated results from the "MetaPredictors". While this still allows for a relative comparision with the other techniques and algorithms evaluated in the paper, it makes it hard to evaluate the actual usefulness and effectiveness of the apporach.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: - Could you provide the training and validation loss curves?
- In Figure 13, do you quantize the probability of $r_m$​ to obtain the vector in the embedding layer $e_m$​? As I understand it, embedding layers typically take indices as inputs.
- In Figure 13, if I understand correctly, $e_{\phi_0}$ is a linear layer that maps the device feature vector to a k-dimensional vector. If this is the case, referring to it as an 'embedding' layer may be somewhat confusing, given the use of $e_m$ for embeddings.
- Suppose an application requires a network with a latency of under 30 ms on a given platform. How does the user preference vector, where each element is a probability within [0, 1], map to specific latency (and energy consumption) in real-world scenarios?
soundness: 4
strengths: The differentiable NAS framework proposed in the paper introduces a novel hypernetwork that enables zero-shot transferability to new devices, which, to the best of my knowledge, is an original contribution. I appreciate how this framework addresses key challenges faced by the NAS community. Previous approaches directly optimize network structures and hardware metrics using differentiable operators for one specific platform, while two-stage NAS frameworks separate supernet training from subnet searching. However, the search process in two-stage frameworks is often costly, as it relies on hardware-specific predictors and estimators for each objective on each platform. Although some works, such as  [1, 2], have attempted to reduce the search cost in two-stage frameworks, I believe that a unified solution that provides zero-shot transferability and multi-objective optimization across new platforms has been lacking. This paper fills that gap by proposing a unified framework that combines hardware and objective embeddings with differentiable NAS through the use of a hypernetwork. I believe this is an exciting progress for neural architecture search.

The paper is well-presented and well-organized. The extensive experiments that compare it to several state-of-the art NAS frameworks and the in-depth analysis of the framework fully support the claims made in the paper.

[1] HELP: Hardware-Adaptive Efficient Latency Prediction for NAS via Meta-Learning

[2] BRP-NAS: Prediction-based NAS using GCNs
summary: The paper introduces a novel framework for differentiable neural architecture search (NAS) that integrates a generalizable hardware embedding with multi-objective embeddings during supernet training. This approach enables the NAS framework to generalize to unseen hardware platforms and efficiently sample Pareto-optimal subnets based on user preferences. To achieve this, the authors design a MetaHypernetwork that uses a sampled preference vector and representative hardware embeddings to guide architecture sampling within the supernet. During training, instead of randomly sampling subnets, the framework selects and trains subnets based on the hardware and objective embeddings, resulting in a better Pareto front. Extensive experiments demonstrate that this framework consistently produces superior Pareto-optimal subnets across various devices, outperforming several state-of-the-art NAS frameworks.
weaknesses: My only concern is that training differentiable NAS with multi-objective hardware metrics can often be unstable and may struggle to converge, leading to issues with reproducibility. While the authors have shared the hyperparameters and plan to release the implementation, it would be beneficial to include a discussion of the training techniques in the Appendix to address these challenges.

Meta-review: This paper presents a hypernetwork-based method for hardware-aware neural architecture search and demonstrates zero-shot generalization to new devices. The strengths of this paper include the detailed experiments that tested the proposed MODNAS on 19 hardware devices and showed good performance. The main weaknesses of this paper include the details of the approach were not initially very clear, and reviewers generally do not favor hyper network solutions to NAS. After the rebuttal, the missing details were addressed as can be seen in the edits of the appendix. All the reviewers ended up with a positive score for this paper.

Decision: Accept (Poster)