Multi-objective Differentiable Neural Architecture Search

Authors: Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler, Samuel Dooley, Josif Grabocka, Frank Hutter
Primary area: optimization
Keywords: hardware efficiency, neural architecture search, network compression

Abstract:
Pareto front profiling in multi-objective optimization (MOO), i.e., finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives that require training a neural network. Typically, in MOO for neural architecture search (NAS), we aim to balance performance and hardware metrics across devices. Prior NAS approaches simplify this task by incorporating hardware constraints into the objective function, but profiling the Pareto front necessitates a computationally expensive search for each constraint. In this work, we propose a novel NAS algorithm that encodes user preferences to trade-off performance and hardware metrics, yielding representative and diverse architectures across multiple devices in just a single search run. To this end, we parameterize the joint architectural distribution across devices and multiple objectives via a hypernetwork that can be conditioned on hardware features and preference vectors, enabling zero-shot transferability to new devices. Extensive experiments involving up to 19 hardware devices and 3 different objectives demonstrate the effectiveness and scalability of our method. Finally, we show that, without any additional costs, our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets, including MobileNetV3 on ImageNet-1k, an encoder-decoder transformer space for machine translation and a decoder-only space for language modelling.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1729478587293

thread_type:
official_review

<thread_object>
creator: reviewer_kebz

<content>
summary:
The paper introduces a novel framework for differentiable neural architecture search (NAS) that integrates a generalizable hardware embedding with multi-objective embeddings during supernet training. This approach enables the NAS framework to generalize to unseen hardware platforms and efficiently sample Pareto-optimal subnets based on user preferences. To achieve this, the authors design a MetaHypernetwork that uses a sampled preference vector and representative hardware embeddings to guide architecture sampling within the supernet. During training, instead of randomly sampling subnets, the framework selects and trains subnets based on the hardware and objective embeddings, resulting in a better Pareto front. Extensive experiments demonstrate that this framework consistently produces superior Pareto-optimal subnets across various devices, outperforming several state-of-the-art NAS frameworks.

strengths:
The differentiable NAS framework proposed in the paper introduces a novel hypernetwork that enables zero-shot transferability to new devices, which, to the best of my knowledge, is an original contribution. I appreciate how this framework addresses key challenges faced by the NAS community. Previous approaches directly optimize network structures and hardware metrics using differentiable operators for one specific platform, while two-stage NAS frameworks separate supernet training from subnet searching. However, the search process in two-stage frameworks is often costly, as it relies on hardware-specific predictors and estimators for each objective on each platform. Although some works, such as  [1, 2], have attempted to reduce the search cost in two-stage frameworks, I believe that a unified solution that provides zero-shot transferability and multi-objective optimization across new platforms has been lacking. This paper fills that gap by proposing a unified framework that combines hardware and objective embeddings with differentiable NAS through the use of a hypernetwork. I believe this is an exciting progress for neural architecture search.

The paper is well-presented and well-organized. The extensive experiments that compare it to several state-of-the art NAS frameworks and the in-depth analysis of the framework fully support the claims made in the paper.

[1] HELP: Hardware-Adaptive Efficient Latency Prediction for NAS via Meta-Learning

[2] BRP-NAS: Prediction-based NAS using GCNs

weaknesses:
My only concern is that training differentiable NAS with multi-objective hardware metrics can often be unstable and may struggle to converge, leading to issues with reproducibility. While the authors have shared the hyperparameters and plan to release the implementation, it would be beneficial to include a discussion of the training techniques in the Appendix to address these challenges.

questions:
- Could you provide the training and validation loss curves?
- In Figure 13, do you quantize the probability of $r_m$​ to obtain the vector in the embedding layer $e_m$​? As I understand it, embedding layers typically take indices as inputs.
- In Figure 13, if I understand correctly, $e_{\phi_0}$ is a linear layer that maps the device feature vector to a k-dimensional vector. If this is the case, referring to it as an 'embedding' layer may be somewhat confusing, given the use of $e_m$ for embeddings.
- Suppose an application requires a network with a latency of under 30 ms on a given platform. How does the user preference vector, where each element is a probability within [0, 1], map to specific latency (and energy consumption) in real-world scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730120268162

thread_type:
official_review

<thread_object>
creator: reviewer_vhgp

<content>
summary:
The authors propose a multi-objective HW aware NAS algorith that can emit optimized DNN architectures across multiple different target devices using a single search run.
To overcome the challange of having to solve multiple different optimization runs for each considered HW platform, the authors propose using a hypernetwork to generate an architectural distribution across multiple devices based on a "preference vector" and a "hardware device feature vector" from which discrete architectures can then be sampled.
For objective function evaluators, the authors use a Supernetwork as a standin for a accuracy and "MetaPredictors", one for each target device, to estimate hardware metrics.

strengths:
- The extension of existing zero-shot NAS techniques with Hypernetworks for HW aware NAS is motivated well and contextualized nicely for alredy existing techniques. 
- The authors provide an extensive evaluation for different applications (language modeling, vision, translation), multiple well known NAS searchspaces, and different target spaces (2-3 dimensional).

weaknesses:
- The authors do not show how the proposed DNN architectures would actually perform on the different target systems. As far as I understand it, the HV results shown are calculated using the estimated results from the "MetaPredictors". While this still allows for a relative comparision with the other techniques and algorithms evaluated in the paper, it makes it hard to evaluate the actual usefulness and effectiveness of the apporach.

questions:
- What assumptions do the MetaPredictors make about the underlying devices when estimating latency and energy requirements? For example, do they assume a particular choice of operating system and process scheduler used, other parallel and system processes running, core utilisation of the DNN inference, runtime library, or HW design used to execute the DNN on the target device?

- Especially on smaller systems, memory requirements are often a major bottleneck of DNN inference: Could the approach proposed by the authors be extended to include memory? So far, the evaluation seems to focus only on accuracy, latency and energy.

- How exactly are "preference vector" and a "hardware device feature vector" defined? The authors cover many different types of HW in their evaluation: GPUs (e.g. 1080 ti), smarthpones (e.g. pixel 2, 3), edge devices (e.g. raspi 4) and dedicated HW accelerators (eyeriss, FPGA), which can be characterised by a number of different metrics, often exclusive to the type of HW considered. (e.g. #cores, processor speed, RAM, ... for the raspi4 vs. #streaming cores, VRAM, bus interface, ... for the 1080 ti vs. #LUTs, #BRAM, ... for the FPGA). So it would be really interesting to know how the authors put these metrics into relation and unified them into one vector.

- Which FPGA did the authors use? Since an FPGA is just freely programmable hardware, it would also be interesting to know which DNN accelerator design the authors implemented on the FPGA to perform their evaluation.

- Fig. 5: Why is MODNAS shown as an upper threashold line (+/- variance)? Based on the caption, I would have expected to see an HVI curve that stops after 24 evaluations, similar to the other curves in the plot.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645240849

thread_type:
official_review

<thread_object>
creator: reviewer_ugea

<content>
summary:
This paper proposes a Neural Architecture Search (NAS) algorithm that optimizes several performance metrics on various hardware.
The paper adopts Multi Objective Optimization (MOO) to optimize multiple objectives simultaneously.
The key idea is to train $\textbf{MetaHypernetwork}$ which takes hardware features and user preferences about performance metrics as inputs and returns optimized network architectures as outputs.
To train $\textbf{MetaHypernetwork}$, the paper exploits $\textbf{Supernetwork}$ for NAS and $\textbf{MetaPredictor}$ for optimization in terms of efficiency.

$\textbf{MetaHypernetwork}$ takes user preferences and device features, and returns network architecture designs.
Then, $\textbf{Architect}$ makes those architecture designs into differentiable architecture parameters.
With these architecture parameters, $\textbf{Supernetwork}$ computes accuracy-related loss of the given network design.
Also, $\textbf{MetaPredictor}$ computes efficiency-related loss with the network architecture under the given hardware features.
Further, the paper proposes to update $\textbf{MetaHypernetwork}$ using Multiple Gradient Descent (MGD) to get optimized network architectures that satisfy multiple objectives concurrently.

For NAS, the proposed method is adaptable to diverse pretrained $\textbf{Supernetwork}$.
For efficiency, $\textbf{MetaHypernetwork}$ can be updated with a lot of hardware-related loss functions at once.
For optimization, the paper achieves Pareto frontier solutions that are hard to reach with sequential or averaged gradient updates.

strengths:
- The paper proposes to execute one-shot NAS while search networks satisfy multiple objectives about accuracy and hardware efficiency with target hardware.
- The paper analyzes the proposed method with various aspects such as efficacy, and robustness of the training process.
- The paper provides extensive experiments and visualizations to support the proposal and its analysis.

weaknesses:
- It may be hard to regulate the trade-off among user preferences with scalarization. Figure 4 can be a support, but it is just an abstract depiction, not experimental results.
- The proposed method can help search optimized network architectures quickly at low cost. However, network architectures the same as or near ground truth solutions may hard to be reach with $\textbf{MetaHypernetwork}$, where other works can reach with huge search costs.

questions:
- How are user preference and hardware device embeddings designed?
- Please explain how $\textbf{Architect}$ makes architecture parameters differentiable and propagate them to the $\textbf{MetaHypernetwork}$.
- In Figure 4, solutions on the Pareto frontier can be achieved by modulating user preferences. However, in HDX [1], which is a one-shot NAS with hardware constraints, claims that modulating hyperparameters linearly doesn’t lead to linearly distributed results. Is $\textbf{MetaHypernetwork}$ free from this problem? Can real experimental results be plotted like Figure 4 to substantiate the integrity of $\textbf{MetaHypernetwork}$?
- To search network architectures with other works, the paper sets the search time budget up to 2.5 times compared to that of MODNAS. (Or fixed time budget, e.g., 192 hours) That is, MODNAS outperforms prior works in terms of search time, and the quality of solutions is better than that of others under given time budgets. What the review wonders is whether can other works find near-GT solutions with a larger time budget. If a target network is distributed extensively and used frequently, huge search costs can be tolerable if there are better solutions.

[1] Deokki Hong, Kanghyun Choi, Hye Yoon Lee, Joonsang Yu, Noseong Park, Youngsok Kim, and Jinho Lee. Enabling Hard Constraints in Differentiable Neural Network and Accelerator Co-Exploration. In Proceedings of the 59th ACM/IEEE Design Automation Conference, 2022.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731024382444

thread_type:
official_review

<thread_object>
creator: reviewer_ei5k

<content>
summary:
The paper (MODNAS) presents an approach to Neural Architecture Search (NAS) that balances competing objectives—like performance, latency, and energy efficiency—across multiple hardware devices. By encoding user preferences as a scalarization vector, MODNAS efficiently searches for Pareto-optimal solutions across diverse devices in a single run. The method employs a hypernetwork to generate architectures for specific hardware configurations, leveraging multiple gradient descents for optimization.  The method has been evaluated on MobileNetV3 on ImageNet-1k, an encoder-decoder transformer space for machine translation, and a decoder-only space for language modeling.

strengths:
1-The method is adaptable to a range of devices by conditioning the hypernetwork on device embeddings, making it highly versatile for deployment on diverse hardware.

2-MODNAS is tested on various hardware devices and tasks, including image classification, machine translation, and language modeling, showcasing its applicability across multiple domains.

weaknesses:
1-Using Hypernetworks for NAS is well known but doesn’t seem a promising solution. It is like a heuristic solution.

2- I think energy and latency are not necessarily conflicting metrics.

questions:
__1-__ Using hypernetwork for the NAS and prate-frontier learning is well known. 

That limits the novelty of the proposed method.

[A]Brock, A., Lim, T., Ritchie, J., and Weston, N. (2018). SMASH: One-shot model architecture search through hypernetworks. In the International Conference on Learning Representations

[B[ Lorraine, Jonathan, and David Duvenaud. "Stochastic hyperparameter optimization through hypernetworks." arXiv preprint arXiv:1802.09419 (2018).

[C] Pan, Z., Liang, Y., Zhang, J., Yi, X., Yu, Y., and Zheng, Y. (2018). Hyperst-net: Hypernetworks for spatio-temporal forecasting. arXiv preprint arXiv:1809.10889

[D]  Zhang, C., Ren, M., and Urtasun, R. (2019). Graph hypernetworks for neural architecture search. In International Conference on Learning Representations.
 
However, there are major concerns about the performance, initialization, and scalability as stated here (Section 6 in the following paper):
Chauhan, Vinod Kumar, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A. Clifton. "A brief review of hypernetworks in deep learning." Artificial Intelligence Review 57, no. 9 (2024): 250.


__2-__ Hypernrtwork is not performing well on unseen networks.
https://arxiv.org/abs/2110.13100
With that, Could you elaborate on any specific constraints or limitations when transferring MODNAS to devices significantly different from the ones used in the training phase?


__3-__ MiLeNAS paper (CVPR’20) shows that the gradient errors caused by approximations of second-order methods in bi-level optimization results in suboptimality, in the sense that the optimization procedure fails to converge to a (locally) optimal solution.  However, it seems that the authors still use a bi-level optimization technique.  It seems using advanced techniques like MiLeNAS might be more useful.
 
__4-__ Can the MODNSD be applied to object detection tasks as well?

__5-__ MobileNet search space is pretty small DNN. How does it work on more complex DNN search space?  

__6-__ The paper mentions scalability but doesn’t mention potential bottlenecks. For example, are there search space complexities that MODNAS struggles with?



__7-__ The paper doesn't provide the code. That limits reproducibility of the proposed method.



__8-__ I am not sure how accurate is the energy model that the paper is using considering even Eyeriss is a pretty old paper (2016).  The authors need to provide more details about energy modeling. 

__9-__ I think it is better to use power rather than energy. Considering energy is power x  latency, when you minimize the latency, if the power is fixed, energy is minimized automatically. Considering power, accuracy, and latency can be a better metric.


__10-__ The paper employs the Frank-Wolfe solver for optimizing scalarizations. Could you discuss any trade-offs in this choice, and if other optimizers were considered?

__11-__ Given that preference vectors affect the optimization landscape, an analysis of how different preference configurations influence the final architectures would clarify MODNAS’s versatility.


__12-__ MODNAS is tested for three objectives—accuracy, latency, and energy. Could this method scale effectively if additional objectives were introduced, or would this necessitate modifications?

__13-__ The approach appears tailored to devices with GPU-like architectures. It would be valuable to understand how MODNAS performs on other architectures, like mobile hardware.




__14-__ Minor: 

A)The paper is pretty dense. I hardly can read the legends in Figures 7, 8.

B)Minor Typo:
Pareto front profiling in multi-objective optimization (MOO), i.e. finding a diverse set of Pareto optimal solutions, is challenging" – A comma after "i.e." would improve readability: "i.e., finding a diverse set..."

"To search across devices, we frame the problem as a multi-task multi-objective optimization problem" – Add a comma, “multi-task, multi-objective optimization problem.”
</content>
</thread_object>

</thread>
<thread>
creation date:
1731935240634

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to all reviewers

comment:
We would like to thank all the reviewers for reading our paper and their insightful feedback. We also appreciate the positive average score. As a general response, we want to emphasize the following main changes in the submission PDF (highlighted in blue):
1. **Appendix J**: additional discussion on the robustness of MODNAS (*addressing Reviewer KeBz*)
2. **Appendix K**: alignment of preference vectors with pareto front (*addressing Reviewer ugEA and ei5k*)
3. **Appendix L**: training and validation loss curves (*addressing Reviewer KeBz*)
4. **Appendix M**: Multi-objective optimization baselines with more budget (*addressing Reviewer ugEA*)
5. **Appendix N**: Additional Details on the Architect (*addressing Reviewer ugEA*)
6. Updated Figure 15 with the MODNAS + MiLeNAS baseline (*addressing Reviewer ei5k*)

We hope that after reading our responses your concerns will be addressed and you will consider increasing your scores. Thank you very much for your time.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733202501342

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
As the discussion period closes soon, we would like to again thank all the reviewers for their thorough evaluation of our work and their active engagement during the discussion phase. Their valuable feedback has significantly enhanced both the clarity of our paper and the depth of our experimental analysis. We also appreciate the overall positive reception reflected in the scores of our paper. 

We would like to highlight the following additional updates, which are incorporated into the revised version of our paper (marked in blue):

1. **Appendix O**: New experiments analyzing perplexity and memory usage objectives.
2. **Appendix P**: Discussion on the applicability of our approach to the object detection task.

Moreover, as suggested by reviewer **ei5k**, we will refine **Section 5** of the paper to include a discussion on potential search space complexities.

If there are any further questions, we remain available for discussion. Thank you once again for your thoughtful contributions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a hypernetwork-based method for hardware-aware neural architecture search and demonstrates zero-shot generalization to new devices. The strengths of this paper include the detailed experiments that tested the proposed MODNAS on 19 hardware devices and showed good performance. The main weaknesses of this paper include the details of the approach were not initially very clear, and reviewers generally do not favor hyper network solutions to NAS. After the rebuttal, the missing details were addressed as can be seen in the edits of the appendix. All the reviewers ended up with a positive score for this paper.