Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The authors demonstrate how multi-objective NAS can be formulated as a Quality Diversity Optimization (QDO). 

overall_review: The authors demonstrate how multi-objective NAS can be formulated as a Quality Diversity Optimization (QDO). The formulation of the proposal is sound. Unfortunately, the authors don't demonstrate the idea of creating niches with actual hardware devices. It would have been great to show experiments on many target devices. 


review_summary: The paper is marginally above the acceptance threshold. The quality of the paper is good but the experiments and results section could be improved, particularly when the authors show the creation of niches. It would have been good to see how the proposed approach would be used in actual hardware devices.
review_title: null

final_score: rating: 3: Marginally below the acceptance threshold (use sparsely)
review_content: summary_of_contributions: This paper introduces a new approach for multi-objective NAS that solves a quality diversity optimization (QDO) problem instead of the traditional pareto curve optimization problem.  The main insight presented by the authors is that we usually care about the best architecture in multiple perhaps overlapping subspaces instead of those on the pareto front.  The authors combine a few existing methods to form a multi-fidelity QDO solver called BOP-ElitesHB which combines Bayesian optimization with Hyperband for QDO.  Experiments show BOP-ElitesHB to outperform other methods for QDO on NASBench-101, NASBench201, and a model compression task.
overall_review: Positives:
- It is natural to pose NAS for multiple deployment scenarios as a QDO problem.
- The proprosed QDO optimizer BOP-EliteHB appears to work well on studied benchmarks.

Negatives:
- Technical novelty is limited since the main technical contribution is a small modification to Hyperband to adapt it for quality diversity optimization.  Then, the authors modify an existing QDO method called BOP-Elites to make it suitable for NAS search spaces by borrowing insights from SMAC and BANANAS.  
- Benchmarks studied are not used in practice.  Need to evaluate QDO methods on a more realistic NAS search space like those based on MobileNetV3.
review_summary: While I agree with QDO as the right problem to solve for multi-objective NAS, I think the paper would benefit from additional experiments on more practical NAS search spaces with niches corresponding to actual deployment settings (e.g. mobile platforms like Google Pixel, Samsung S7, etc).  In lieu of such experiments, the paper does not sufficiently establish QDO optimization as a meaningful approach to NAS in practice.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: Quality-diversity optimization is a new branch of stochastic optimization, the author(s) has attempted to introduce QDO to the neural architecture search field. The paper framed the search for neuronal architecture as a QDO problem, and compared QDNAS and classic multi-object NAS whose performances are better. In the experimental results showed that QDNAS outperformed and more efficient multi-object NAS in the NAS-Bench-101 and NAS-Bench-201 with specified niches.
overall_review: Advantage:
The Quality-Diversity Optimization is a novel branch of stochastic optimization, the author(s) attempted to introduce QDO to neural architecture search field. Also, the experimental outcomes are sufficient, that a number of QDO and multi-object optimization methods have been tested on multiple benchmarks.

Key weakness:
The author did not provide code to carry out. Besides, it is regrettable that the difference in performance between QDO and multi-objects on various data sets has not been explored in depth. 

review_summary: Overall, this paper is a comprehensive one, which clearly expresses the ideas to be conveyed and the experimental results are sufficient. While the experiment section has not delved into the details, this does not affect the excellence of this paper.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: - The paper discusses neural architecture search (NAS) as a quality diversity optimization problem. 
- Instead of treating it as a multi-objective optimization problem, a primary objective function (in the context of NAS usually the model performance, e.g., accuracy) is optimized s.t. it is optimal within one niche, e.g., such that a certain memory requirement is fulfilled.
- The authors present a new acquisition function for Bayesian Optimization, the expected joint improvement of elites, which extends expected improvement to the given problem setting.
- The authors present three algorithms, BOP-Elites*, qdHB, and BOP-ElitesHB, and show their performance on several NAS-Bench benchmarks. 
overall_review: *** Strong points ***

- For NAS, the approach is reasonable and shows good empirical performance. 
- The motivation is clear.
- The experiments are well-described and seems to be complemented by detailed supplementary material.

*** Weak points ***

- In the experiments, there is a penality of 100 added to the validation error if a niche is empty. There is no argument for why this particular number was chosen. This choice should be motivated.
- The experiments only consider one secondary objective (number of trainable parameters), however, the general setup allows for an arbitrary number of secondary objectives. It would have been interesting to see the impact of more than one secondary objective. 
- For the final experiment, training MobileNetV2, a comparison to other methods is missing. 
review_summary: The paper presents an alternative way of incorporating secondary objectives in NAS by using quality-diversity optimization. They present a common use case, in particular the deployment of models on devices with different memory resources.  The paper is overall clear and well-motivated. The results show strong performance of their method. A comparison to other methods is missing on the final benchmark, also the choice of the secondary objective for NAS-Bench-201 was unclear to me. 

Overall, I think some aspects need to be clarified but I overall recommend to accept the paper.
review_title: null

Meta-review: metareview: The paper proposes to consider solving a quality diversity optimization (QDO) problem instead of solving a traditional multi-objective optimization problem when performing hardware-aware NAS. The motivation is that in practice, one is typically more interested about finding the best configurations for different hardware (constraints) represented as different niches. Reviewers noted the pertinence of the problem and highlighted the quality of the results obtained. One of the main critic was to not consider actual hardware devices but this concern was addressed by the authors with extra experiments on MobileNet and two reviewers acknowledge this improvement and raise their score (one reviewer did not acknowledge this addition whereas it was one of his main critic).

Given the relevance of the problem studied and the good performance of the method proposed, I recommend accepting the paper.

A small comment, the following paper could be mentioned given the similarity with some aspect of the paper, in particular using multi-fidelity and a multi-objective approach for hardware-aware NAS: 
A multi-objective perspective on jointly tuning hardware and hyperparameters (https://arxiv.org/abs/2106.05680).


confidence: 3: The area chair is somewhat confident
Decision: Accept