Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an hardware-aware NAS on a superset that preserves the Pareto ranking (accuracy and latency). It demonstrates better accuracy-latency tradeoff when compared to SOTA approaches.
strength_and_weaknesses: Strength
* Good trade-off between latency and accuracy is achieved using the proposed approach comparing to the SOTA.

Weakness
* It is unclear how the alpha parameter is derived and how does the correctness of accuracy / latency modelling affect the quality of results.
clarity,_quality,_novelty_and_reproducibility: Clarity
* This paper is generally well structured. Some details are missing and need to be clarified (see below).

Quality
* The approach is generally well described and is developed based on existing proven method. Some clarifications are needed to justify the correctness (see below).

Novelty
* This work appears to be an extension of superset training to include the proposed Pareto-rank. 

Reproducibility
* Cannot be evaluated based on the existing materials.
summary_of_the_review: * This paper presents an hardware-aware NAS on a superset that consider preservation of Pareto-ranking.
* How is the operation score parameter calculated? Is this the same as the architecture parameter in DARTS? Is this score a good representation of the importance of operation?
* Similarly, the line “Adjust alpha” in Algorithm 1 is not clearly explained.
* The notations in Section 3.2 are slightly confusing, operation score (alpha), Pareto-rank score (alpha), Pareto score (P_s).
* The latency model is based on the summation of layer-wise latency stored in a lookup table. Is this assuming sequential execution of layers in the target hardware? Is this assumption true for the target hardware?
* Are the latency numbers reported in Table 2 and 3 based on real measurement or the latency model?
* Can you provide more comparison to SOTA HW-aware NAS, e.g. BRP-NAS [1] and HELP [2] similar to Table 2 and 3?

Ref:
[1] https://arxiv.org/abs/2007.08668
[2] https://arxiv.org/abs/2106.08630
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors introduce a methodology for pareto front sub-network identification from a supernetwork. This is done in a two step process, by first conducting fair subsampling of subnetworks for Nf epochs, followed by Pareto-Rank  training for Np epochs. Sub-networks that are furthest from the pareto front are also dropped to accelerate training. The proposed method achieves state of the art pareto front approximation (97%) in 2 GPU days. A new parameter, $\alpha$ is introduced to measure which operation is critical, and pareto-ranking loss is utilized to adjust the $\alpha$ for different paths. The sum of the selected operations alpha values is utilized as the pareto score and Cross Entropy Loss is taken between the ranking scores and pareto ranks (ground truth) to update $\alpha$. 
strength_and_weaknesses: Strengths: 

- The idea is straightforward to implement on existing supernetwork based training schemes, and seems to be effective from the results.
- The methodology is validated on NASBench201, DARTS and ImageNet. 

Weakness:

- It is not clear how the second phase of training (argmax(a)) enforces fairness. Justification and implication of unfair sub-network sampling in this stage may be a useful discussion. 
- A table showing the pareto approximation effectiveness for each of the tested search spaces would be very useful.
- The current results highlight the fact that PRP-NAS-BL finds a model with low latency, and PRP-NAS-BA finds a model with high accuracy. (As expected.) If possible, comparing with the ground truth optimal architectures in each case may add more value/context to the result. 

Questions:

- If argmax(a) selection scheme does not enforce fairness, it is natural to assume that the correlation between argmax(a) and kendall tau rank correlation would increase simply due to the nature of the CELoss minimization, with no bearing on the ground truth. How is the 'Truth' in the CELoss(Score, Truth) formulation calculated?
- If Truth is a list of intermediate accuracies of the sub-networks, then naturally the kendall tau rank correlation will keep increasing, simply due to sampling bias. Comments on this issue would be appreciated. 

Minor comments:

- Table 1 (Training hyperparameters) can be moved to the Appendix.
- Fix 3.2 : 'Algorithm 1 and Figure 1 summarizes summarizes..' 
clarity,_quality,_novelty_and_reproducibility: see above
summary_of_the_review: The paper is interesting but there are some concerns (see above).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method for multi-objective Neural Architecture Search (NAS), i.e., finding the best NAs based both on task prediction and hardware-based metrics (e.g., latency).  The proposed method (HW-NAS, HW for "hardware") builds on single-stage supernetwork techniques (the union of possible architectural choices for the neural final architecture) by co-training the supernetwork's parameters and a "Pareto" parameter to find (and rank) the sub-models.   They propose to use Pareto-ranks as the primary search objective and NAS evaluation criteria.  They implemented their algorithm and compare the found rank of the models on the Pareto frontier with Pareto rank of the individually trained sub-models for multiple benchmarks.   In addition the trained Pareto parameter allows them to decrease supernetwork training time by pruning sub-networks not be on the Pareto front.    By doing so, they achieve a 97% Pareto front approximation (vs 87%) for the resulting sub-models. 
strength_and_weaknesses: Positive Points:
+ Multi-objective NAS is an important area, and the proposed method (Pareto-based search training) appears novel
+ Evaluation used 3 benchmarks/search spaces (DARTs, NAS-Bench-201, ProxylessNAS) and additionally compares to FairNAS and a recent one-shot GCN-based technique. 
+ Results are promising: optimal sub-networks often meet or improve accuracy/latency while also taking less time to train.  

Negative Points
- The paper is written as if optimizing for multiple objectives was a relatively new idea.  But searching for models on the Pareto front has been previously explored in the NAS area.  Related work doesn't describe much of the work in that multi-objective space.   
- The description of the algorithm is light on details, specifically the Pareto loss function, the Pareto parameter updates, and how NAs are found.  This makes reproducibility difficult. 
- Given the above limited description of the operation of the Pareto-based training, it isn't clear how easy it is to extend to multiple objectives. 
- There is little to no evaluation of the Pareto parameter training in isolation to understand its behavior / performance.
clarity,_quality,_novelty_and_reproducibility: Overall, this paper appears to propose a relatively novel take on discovering sub-networks on the Pareto frontier.   However, a lack of related work and detail around the algorithm muddy the presentation, ultimately making it difficult to assess novelty, reproducibility, and applicability. 

The end of Sec 1 states that it achieves a "97% near Pareto front approximation" -- it might be straightforward but be very clear if it's a critical metric.  

Perhaps the largest omission concerns Algorithm 1.   Section 3.2's "steps" do not seem to accurately reflect the algorithm.  In step 2 "training to solve the task", the bullet only states taking a mini-batch and choosing a subnetwork.  It does not describe training that subnetwork nor give intuition behind that action.   The next section "Pareto-rank training" starts with the sentence "After completing the iteration" but step 2 is part of the iteration?    At least the appendix could spell out the "list wise CE loss" function (and Algorithm 1 could refer to it as opposed to just "Loss_{pr}".    Is \alpha just a single parameter (value) or something else (vector)?   What does "Adjust \alpha" mean? 

It's not clear whether you're training the sampled subnetworks from scratch.  I would assume this was the case, since the Pareto training compares the found Pareto ranks (so far) to "ground truth."   However, part of the point of the work was to avoid expensive retraining of the sub-network.   It looks like the benchmarks provided trained networks and in one section you mentioned being able to use those for estimating the Pareto Kendall measure (sec 4.3).   So, does training time (GPU Hours) include that time? 

Another gap is that the paper does not investigate the performance / convergence of the Pareto training.   How does the search proceed with N_p?  How many sub-networks should be sampled (n)?   It isn't clear what "Hyper Volume" is or why the reader should care.   It seems to have something to do with completeness of the frontier.   How often will this approach return only rank 1 models?  How often will it return the complete frontier?   How can we give the reader an intuition or proof of that behavior?   I.e., before we compare HW-NAS' final models, it wasn't clear how well the Pareto-training performed. 

I'm curious if there is a discussion to be had around the differences of Pareto ranks and performance ranks.   This mechanism will almost always return all models as rank 1 (sec 4.4), but performance ranks will likely have many fewer identical ranks.   Would it be possible to return a set of sub-networks scored rank of 1, but actually they all belong to the second frontier?  When you compute the "ground truth" rank, was it only Pareto ranking the sampled set?  In other words, if the ground truth set was scored in isolation, they would look like rank 1, and the correlation would be artificially high.  

The end of 3.1 says we'll only use Pareto ranks, but it would still be nice to repeat in Fig 5 caption.   

Related work should take into consideration other pieces of work that have looked at Pareto-based multi-objective formulations, such as 
Elsken, Thomas, Metzen, Jan Hendrik, and Hutter, Frank. Efficient multi-objective neural architecture search via lamarckian evolution, ICLR 2019, and Efficient Forward Architecture Search, Hanzhang Hu, John Langford, Rich Caruana, Saurajit Mukherjee Eric Horvitz, Debadeepta Dey, Neurips 2019.    BTW, I got these by looking at prior openreview at ICLR.  
summary_of_the_review: This looks like an interesting and fruitful direction of research - cotraining the supernetwork with a Pareto-based parameter to guide the search.  However, there are presentation, clarity, and background work issues that make it difficult to place the work into context, interpret the results, and understand the algorithm enough for reproducibility. 

[Post Author Response] The authors have addressed significant issues regarding the lack of details, including more detailed training steps, specifying the Pareto parameter's loss function, and including new results.   Many of these details should have been present in the original paper. but I've bumped the recommendation to marginal below in light of them.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a supernetwork training strategy for preserving the Pareto ranking between different subnetworks for handware-aware architecture search. To maintain a higher ranking correlation supernetwork, the authors propose to prune the operations based on the optimal Pareto front during the training of the supernetwork. Moreover, the authors define a new metric for better evaluating the architecture in a multi-objective context. Experimental results on three benchmarks demonstrate the effectiveness of the algorithm in some cases. However, the proposed method is not clearly described and the experiments can be further improved. My detailed comments are as follows.
strength_and_weaknesses: ### Positive points:

- The authors introduce the Pareto ranking as a novel metric for multi-objective architecture search. 

- The authors propose a supernetwork training strategy for preserving the Pareto ranking between different subnetworks.

 - The authors propose to prune the operations based on the optimal Pareto front while training the supernetwork.

### Negative points:

 - In the Pareto rank-preserving training section, the details of the proposed method are missing. In other word, after reading this paper, I can not re-implement the proposed method. It would better for the authors to make it more clear and detailed.
   - a)	The description of the listwise ranking loss is not clear. More explanations are needed.
   - b)	Detailed descriptions for obtaining the Pareto ranks (ground truth) for different subnetworks are needed. 
   - c)	The detail of the hypervolume improvement should be provided since it’s an important metric for pruning the operations. Besides, the motivation for choosing HVI is unclear. Moreover, the number of sample networks n seems important for the proposed method. How to choose a good n?
 - Estimating architecture latency using a lookup table has been proposed and investigated in FBNet. In my opinion, this is not a good way to evaluate the latency of candidate architectures. In practice, the inference framework (e.g., TensorRT) would perform layer/operation fusion. For instance, TensorRT will fuse conv and batch norm layer into a single layer. In this case, the overall latency of the architecture is not equal to the sum of that of each layers. The reality is often more complicated than the above case. Thus, I think it would be difficult to accurately estimate the latency based on the lookup table.
 - In Eq. 1, the authors state $F_{k’}$ is a set of sub-networks ranked k’. I have no idea what is $F_{k’}$ even aftering carefully reading it. More expalinations should be provide.
 - The ablation studies on “the number of sampled sub-network” and “the number of epochs for Pareto-training” are missing. It would be better for the authors to provide more ablation studies.
 - Some other multi-objective search algorithms [1][2] should also be compared in Tables 2 and 3.
 - The details of the experimental setting in Figures 3 and 5 are unclear. More explanations are needed.
 - The ablation study on the proposed pruning strategy is missing. More experiments are expected.

### Minor issues:

 - In Section 3.2, “Algorithm 1 and figure 1 summarizes summarizes the training procedure” should be “Algorithm 1 and figure 1 summarizes the training procedure”.


### Reference

[1] Nsganetv2: Evolutionary multi-objective surrogate-assisted neural architecture search ECCV 2020.

[2] Ponas: Progressive one-shot neural architecture search for very efficient deployment.


clarity,_quality,_novelty_and_reproducibility: Lots of important details of the proposed methods are missing, I can not reproduce the proposed method based on the given manuscript.
summary_of_the_review: I vote for reject since the proposed method is not clearly described and several key ablations are missing in the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a supernetwork-based approach to hardware-aware neural architecture search (HW-NAS), which consists of subnetworks that form the Pareto front in terms of accuracy-latency tradeoff across a set of hardware platforms. This is done by first training the supernetwork with strict fairness across operations (using FairNAS), and pruning the operations to obtain subnetworks that have higher Pareto-rankings. The experimental validation of the method on multiple hardware platforms against an extensive set of HW-NAS, or NAS baselines show that the proposed method can obtain architectures that have better Pareto-rankings compared to baselines (better accuracy for the same latency, or better latency for the same accuracy).

The reviewers in general agree that the proposed supernet-based approach to HW-NAS is well-motivated, as it is addressing an important challenge, and find the method as promising as it obtains architectures with good trade-offs between the accuracy and latency, against an extensive set of baselines. However, they were also concerned about lack of novelty since existing works on HW-NAS also consider searching for Pareto-optimal architectures. Also, more importantly, they were concerned with very unclear descriptions of the methods, which makes the work irreproducible, which was pointed by all reviewers and were the main reason they unanimously recommended rejection. 

The authors provide short responses to the reviewers during the author-reviewer discussion period, which mostly focus on clarifications, and partly revised the paper. However, clarity on the method and the experimental evaluation part still subpar, making it extremely difficult to reproduce the work, given that there is no code or the supernetworks provided. I suggest the authors to completely rewrite the paper with clear descriptions of both the methods and the experimental evaluation part. Another, minor concern on my side is the lack of experimental validation on an exhaustive set of hardware devices, since the method seems to have been validated on only few devices. Since the hardware devices may exhibit largely different latency-generating behaviors depending on the platforms, there should be a more exhaustive experimental validation of the method over a diverse set of devices. 
justification_for_why_not_higher_score: The clarity of the paper does not meet the standard of a research paper, as it leaves out tons of details to reproduce the method and the experimental results.
Decision: Reject