Pareto Rank-Preserving Supernetwork for HW-NAS
Authors: Hadjer Benmeziane
Hamza Ouarnoughi
Smail Niar
Kaoutar El Maghraoui
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural Architecture Search
Supernetwork
Computer Vision

Abstract: In neural architecture search (NAS), training every sampled architecture is very time-consuming and should be avoided. 
Weight-sharing is a promising solution to speed up the evaluation process. 
However, a sampled subnetwork is not guaranteed to be estimated precisely unless a complete individual training process is done. 
Additionally, practical deep learning engineering processes require incorporating realistic hardware-performance metrics into the NAS evaluation process, also known as hardware-aware NAS (HW-NAS). 
HW-NAS results a Pareto front, a set of all architectures that optimize conflicting objectives, i.e. task-specific performance and hardware efficiency. 
This paper proposes a supernetwork training methodology that preserves the Pareto ranking between its different subnetworks resulting in more efficient and accurate neural networks for a variety of hardware platforms. The results show a 97% near Pareto front approximation in less than 2 GPU days of search, which provides x2 speed up compared to state-of-the-art methods. We validate our methodology on NAS-Bench-201, DARTS and ImageNet. Our optimal model achieves 77.2% accuracy (+1.7% compared to baseline) with an inference time of 3.68ms on Edge GPU for ImageNet.

Under review as a conference paper at ICLR 2023
PARETO RANK-PRESERVING SUPERNETWORK FOR HW-NAS
Anonymous authors Paper under double-blind review
ABSTRACT
In neural architecture search (NAS), training every sampled architecture is very cone-conuuming and should be avoided. Weight-sharing is promising solution to speed up the evaluation process. However, sampled subnetwork is not guar- anteed to estimated precisely unless complete individual training process done. Additionally, practical deep learning engineering processes require incor- porating realistic hardware- performance metries into the NAS evaluation process, also known hardware-aware NAS (HW-NAS). HW-NAS results in Pareto front, a set of all architectures that optimize conflicting objectives, i.e. task- specific performance and hardware efficiency. This paper proposes a supernet- work training methodology that preserves the Pareto ranking between its different subnetworks resulting in
hurdware-performance
efficient and accurate neural networks for variety of hardware platforms. The results show a 97% near Pareto front approximation in less than 2 GPU days of search, which provides x2 speed up compared to state- af-t methods. We validate our methodology on NAS-Bench-22 Bench-201, DARTS and ImageNet. Our optimal model achieves 77.2% accuracy (+1.7% compared baseline) with inference time of 3,68ms Edge GPU for ImageNet.
INTRODUCTION
A key element in solving real-world deep learning (DL) problems is the optimal selection of the se- quence of operations and their hyperparameters, called DL architecture. Neural architecture search (NAS) (Santra et al. 202T)) automates the design of DL. architectures by searching for the best ar- chitecture within a set of possible architectures, called search space. When considering hardware constraints, hardware- aware neural architecture search (Benmeziane et al. (202T); I); [Sekanina] (202T) (HW-NAS) simultancously optimizes the task-specific performance, such as accuracy, and the hard- ware efficiency computed by the latency, energy consumption, memory occupancy, and chip area. HW-NAS works (Cai et al. (2019); Lin et al. (202I): Wang et al. (2022) showed the usefulness and discovered sate-of-the-an art architectures for Image Classification (Lin et al. (202T)), Object detec- tion (Chen et al. (201)). ), and Keyword spotting (Busia et al. (2022)).
HW-NAS is cast multi-objective optimization problem. Techniques for HW-NAS span evolu- tionary search, Bayesian optimization, reinforcement learning and gradient-based methods. These require evaluating each sampled architecture on the targeted task and hardware platform. How- ever, the evaluation is extremely time-consuming, especially for task-specific performance, which requires training in the architecture. Many estimation strategies (White al. (202T) are used to alleviate this problem, such as neural predictor methods (Benmeziane et al. (2022a); Ning et al. (2020), ) zero-cost learning (Lopes al. (202T): Abdelfattah al. (2021)), and weight sharing (Chu et al. (202T); Chen et al. (2021). These strategies are evaluated on how well they respect the ground truth ranking between architectures in the search space.
Weight sharing is an estimation strategy that formulates the search space into supernetwork. A supernetwork is an over-parameterized architecture where each path can be sampled. At the end of this sampling, sub-network of the supernetwork is obtained. In each layer, all possible operations are trained. With this definition, we can classify weight-sharing NAS in two categories: (I)a two- stage NAS in which we first train the supernetwork the targeted task. Then, using the pre- traited supernetwork, each sampled sub-network's 's performance can be estimated using a search strategy, such evolutionary algorithm. (2) one-stage NAS in which we simultaneously search and

---

Under review as a conference paper at ICLR 2023
train the supernetwork. Additional parameters are assigned each possible operation per layer. These parameters are trained to select which operation is appropriate for each layer.
Both Weight-sharing categories assume that the rank between different sub-networks is preserved. Two architectures with the same rank imply that they have the same accuracy. State-of-the-as works (Zhang et al. (2020); Peng et al. ) Zhhoo et al. (202T)) have highlighted the training inefficiency in this approach by computing the ranking correlation between the architectures actual rankings and the estimated rankings. Some solutions have been proposed to train the supernetwork with strict constraints on fairness to preserve the ranking for accuracy, such as FairNAS (Chu et al. (202T). Others train a graph convolutional network in parallel fit the performance of sampled sub-networks Chen et al. (2021). However, current solutions have two main drawbacks:
1. In the multi multi-objective context of HW-NAS, different objectives such as accuracy and la- tency have be estimated. The result is Pareto front, set of architectures that better respects the trade-off between the conflicting objectives. The ranking following one ob- jective no longer a good metric for the estimator. In this setting, we need to take into account the dominance concept in the ranking. Both estimations hinder the final Pareto front approximation and affect the search exploration when considering the accuracy and latency as objectives.
2. Many works (Chen et al. (2021); Zhao et al. (2021); Guo et al. (2020)) attempt to fix the supernetwokk sampling after its training. We believe that this strategy inefficient due to the pre-training of supernetwork. Its accuracy-
ranking correlation is bad. In Dong & Yang (2020), a reduced Kendall's tau-b rank correlation coefficient of 0.47 has been obtained on NAS-Bench-201 when using this approach. The accuracy estimation is thus non-conclusive and will mislead any NAS search strategy.
To overcome the aforementioned issues, we propose a new training methodology for supernetworks to preserve the Pareto ranking of sub-networks HW-NAS and avoid additional ranking correction steps. The contributions of this paper are summarized follows:
We define the Pareto ranking as novel metric to compare HW-NAS evaluator in the multi-objective context. Our study shows that optimizing this metric while training the supemetwork increases the Kendall rank correlation coefficient from 0.47 to 0.97 for Vanilla Weight-sharing NAS.
We introduce novel onc-stage weight-sharing supernetwork training methodology. The training optimizes the task-specific loss function (e.g. cross-entropy loss) and Pareto ranking listwise loss function to select the adequate operation per layer accurately.
During training. we prune the operations that are the least likely be in the architec- ture of the optimal Pareto front. The pruning is done by overlapping the worst Pareto- ranked sub-networks and removing the operations that are only used in these sub-networks.
We demonstrate that using our methodology three different search spaces, namely NAS-Bench 201 (Dong & Yang (2020)) DARTS (Liu et al.] (2019)) and ProxylessNAS search space (Cai et al. (2119)), we achieve higher Pareto front approximation compared to current state-of-the-art meth- ods. For example, we obtained 97% Pareto front approximation when One-Shot- NAS-GCN (Chen
al. 202T)) depicts only 87% on
BACKGROUND & RELATED WORK
This section summarizes the state-of-the-arr in accelerating multi-objective optimization HW-NAS.
2.1 ACCELERATING HARDWARE-AWAR NAS
Given a target hardware platform and DL task, Hardware-aware Neural Architecture Search (HW-NAS) Benmeziane et al. (202T) automates the design of efficient DL architectures. HW- NAS is oulti-obeective optimization problem where different and contradictory objectives, such as accuracy, latency, energy consumption, memory occupancy, and chip area, have to be optimized. HW-NAS has three main components: (1) the search space ((2) the evaluation method and (3) the
2

---

Under review as a conference paper at ICLR 2023
Input
op1 op2 op3 op4
op1 op2 op3 op4
op1 op2 op3 op4
layer
layer
layer
Initialization
Pareto Rank-Preeerrinng Phase
Output
Step 1: Train with faimess
Step Training
Step 3: Pruning
Each op is sampled the same amount of time as the others
parameter Alpha is trained and defines which operation is sampled
The operations that are the furthest pruned
Figure 1: Our Pareto Rank-Preserving Training methodology for Supernetwork. The strongest shades illustrate the most important operations for each layer each iteration. a corresponds to the parameter alpha associated with layer I, operation
search.strategs strategy The main cone-connuming component in HW-NAS is the evaluation method. Several state-of-thed art works (White et al. (2021) have been proposed to alleviate this problem.
Predictor-based methods (Nig et. 2020 Lomurno et al. (202T) are the most popular strategies
where machine learning models used to predict the accuracy or latency from the architecture features (e.g. number convolutions, widening factor, etc.) or its representation using Graph
Neural Networks (GNN) (Ning al. (2220)) and Recurrent Neural Networks (RNN) (Lomurno et al. 22021). However, these methods are not flexible to different search spaces as they require training a sampled dataset and then training the predictor.
Weight-sharing approaches (Chu et al. (2021); Chen et al. 20221 Zhao et al. (002)); Guo et al. (2020)) on the other hand, define the search space supemetwork. In each layer, the supernet- work combines the results of possible operations. A sequence of operations from the input to the output is called sub-network and constitutes a possible architecture. Training the supernetwork consists of training several paths at once. The input is forwarded through series of parallel op- erations whose outputs are summed after each layer. There are two main issues when training a supernetwork:
1. The order of the sampled sub-networks matters: Assume we have two sub-networks and B. Both and B start with the same operation op1 in layer 1. During the first training iteration, A is sampled and орı weights are adjusted. The second iteration samples and adjusts opı weights again. we want to evaluate A, we would use the new adjusted weights of opı which degrades the estimation.
2. Unfair Bias: Sub-networks with an initial better task-specific performance are more likely to be sampled next and maintain a higher coefficient in one-stage supernetwork. Fair- nas (Chu et al. (202T) defines strict fairness constraints that ensure that each operation's weights are updated the same amount of times at each stage.
2.2 MULTI-OBIECTIVE OPTIMIZATION HW-NAS
Optimizing conflicting objectives simultaneously requires the definition of decision metric. In multi-ohjective optimization Batista al. (20TT), this metrie the dominance criteria. two- objectives minimization problem, dominance defined as: If 81 and s2 denote two solutions, si dominates 32 ($1 > 82) if and only if Vi f((s)) S f((22) AND 3j f5()) < f3()). fi and ff are conflicting objective functions such latency and accuracy.
Using the dominance, there is no single solution that dominates all the others. We instead build the Pareto front; the set of all dominant solutions. The Pareto front approximation is evaluated using the hypervolume metric. The hypervolume measures the area dominated by a Pareto front approximation P and reference point. The reference point is defined as architecture with high
3

---

Under review as a conference paper at ICLR 2023
latency and low accuracy (furthest from the optimal points). The maximization of hypervolume leads to a high-qualified and diverse Pareto front approximation set.
In HW-NAS, computing the hardware efficiency is expensive due to the time-consuming deployment
and measurements on the hardware. Using multiple performance estimators thus popular Hu
al. 2019); Elsken et al. 2019 eu al. 2020); Huang & Chu (002T) Current multi-objective
HW-NAS approaches focus on optimizing the search algorithm at the expense of poor performance estimators. However, using performance estimator per objective
optimal [Beneeziane al. 2022b). In this paper, we present an original weight-sharing technique that directly predicts a multi-obeective metric, called Pareto ranking.
METHODS
The core motivation for novel training methodology is to achieve an efficient sub-networks evalu- ation for HW-NAS. The proposed training methodology must preserve the Pareto ranking between different sub-networks while reducing the overall search time.
Task-specific weights
Input
W
W
Conv 3x3
Conv 1x1
Avg Pool 3x3
Latency (mm)
Q23
Layer
Operation
scores
Output
Accuracy (%)
Figure 2: Illustration of the Pareto rank and
Figure 3: Supernetwork definition when cou- pling task-specific weights W and operation's score parameters Conv 3x3 is the operation with the highest selection score.
F.
3.1 PARETO RANKING
In this section, we define the Pareto ranking metric used to train and evaluate the supernetwork.
Pareto Ranking Solving the multi-objective optimization problem on a of sub-networks results in a Pareto front. This set of architectures in this front is denoted as Fi. i.c., all the architectures have a rank of We achieve the lower ranks by successfully solving the problem on the set of sub- networks pruned from the previous solutions. The lowest rank is assigned to the sub-networks that do dominate any sub-network. We formally define the Pareto ranking in equation] [ where is the entire supernetwor, Fe. is set of sub-networks ranked k' and is the dominance operation.
Using this ranking, multiple architectures may have the same rank. This happens when none of them can dominate the others.
a ranked > vā E
U àraa
&.FFWARck
Pareto Ranking Correlation. We evaluate the quality of estimator using ranking correlations such as Kendall's tau-b Correlation or Spearman Correlation. Kendall's tau-b determines whether there is monotonic relationship between two variables and is suitable when variables contain many
4

---

Under review as a conference paper at ICLR 2023
tied ranks Benmeziane et al. (2021) which is case. In the rest of the paper, we compute Kendall's Tau-b correlation between the ground truth ranks (i.e. the Pareto ranks obtained from independently training the sub netwwrkwwrss and the Pareto ranks obtained by evaluating each architecture with the supernetwork shared weights.
3.2 PARETO LANK-PRESERVING TRAINING
Our training methodology aims preserving the Pareto ranking obtained by the weight-sharing evaluation.
Figure shows representation of the supernetwork definition and the different parameters we aim to learn. A sub-network is path from the input to the output. All extracted sub-networks are of the same depth. We train the supernetwork with two goals: 1) enhance the task-specific loss function by adjusting W, the task-specific weights of the original model associated with the neural network operations such as the kernels in convolution, and 2) improve the Pareto ranking loss between its different paths by adjusting the weights associated with the operation selection. measures which operation is critical and which one selected.
Alg Algorithm and figure summarize the training procedure.
Step 1: Train with Striet Fairness We train our supernetwork using FairNAS (Chu et al. (202T) strict fairness constraint. This step adjusts the weights of all the sub-networks W and gives a good starting point for the Pareto ranking training. Additionally, the accuracy estimation on the task-specific loss this point is well estimated. We use these estimations to compute the true Pareto ranks in case no accuracy was provided by the benchmark.
Step 2: Pareto ranking training For each iteration, we apply:
Training to solve the task: A mini-batch is sampled from the training set, and sub- network is chosen according to each operation's ' highest a. The operation's 's weights are updated using the task-specific loss, e.g., cross-entropy loss for image classification.
Pareto rank training: In this phase, we purposefully bias the training towards better ranked architectures using the parameters. a parameters are trained using the loss function provided in equation 2 During the forward pass, we Pareto rank the sampled sub-networks. We compute the number of times an operation op; appears in layer 1j on top-ranked sub-networks, denoted as g(op;; 12). N is a hyperparameter defined before training. We denote by g(op;, (z), the ground truth. Equation 2 computes the hinge loss over all layers the sampled sub-networks and compares the number of times the operation with the highest appears in the predicted Pareto front and the ground truth one.
L
Σ
max 10, m g(rg max (a) g(ops,))) (2)
j=1 15) istar max(())
We adjust each operation's a parameters and compute each sampled sub-network's latency using lookup table. We define the predicted Pareto score according to P, = Σopsa Cops i.e., the sum of selected operations' alpha values. Next, we compute the listwise ranking loss defined by the cross entropy between the ranking scores and the Pareto ranks (ground truth).
Step 3: Pruning by Pareto Ranking Sub-networks We drop sub-networks furthest from the optimal Pareto front to accelerate the training. First, select the sub-networks be- longing to the two first Pareto ranks. Then, based on the hypervolume improvement (HVI) (Emmerich et al. (20IT), we select n sub-networks. The operations never used by any sub-network in this selection are removed for each layer. Equation | illustrates how the hypervolume improvement is computed in this context. Oij denotes operation in layer j. HV denotes the hypervolume function and {So.,) } denotes the set of sampled sub-networks using operation in layer
HVI P)= HV ((S HV(P -(Sag)
(3)
5

---

Under review as a conference paper at ICLR 2023
Finally, going over all the layers select the operations with the highest a would suffice to find the most efficient DNN within the search
With alpha parameters
alpha parameters
FairNAS only
space.
Ke-dals Taub care coneeation
Figure 4 shows the training results. We com-
0.8
pare our methodology to FairNAS (Chu et al.
(202I)) strict fairness training. During training, the Pareto ranking correlation increases with the quality of the estimations. When using our training methodology without considering the alpha parameters, the ranking correlation sat- urates at 0.7. FairNAS achieves the same be- haviour with reduced variance among the dif- ferent training runs. However, if we consider the alpha parameters, the selection is more ef- ficient and the architectures' rankings well represented with 0.94.
0.4
50
75
100 opoch
125
150
175
200
Figure 4: Training performance computed with the Kendall's Tau Correlation between the inde- pendently trained Pareto ranks and the estimated Pareto ranks obtained by training the supernet- work.
Algorithm Supernetwork Training Algorithm
Input: Search space S, number of epochs for fairness training Ny, number of epochs for Pareto training Np. Supernetwork parameters (W, a:), training dataloader D, task-specific loss Loss, Pareto raking loss Losspn, number of sampled sub-netwond
procedure TRAIN
Initialize and a for each operation in Supernetwork Striet fairness training for N₃ epochs for i=1 to Np do
for data, labels in D do
Build model with (rgmaa(()) following step
Reset gradients to zero for W parameters
Calculate gradients based on Loss, data, labels and update W by gradients
end for
Sample sub-networks, models
Compute: Pareto rank models, LOSSPR between scores and Pareto rank.
Update a by gradients
end for end procedure
4 EXPERIMENTS
In this section, we evaluate our training methodology three search spaces: NAS-Bench- 201 (Dong Yang (2020), DARTS (Liu et al. (2019) and ProxylessNAS Search space (Cai et al. 2019)).
4.1 SETUP
Search Spaces: Several search spaces have been used to evaluate our method's performance. NAS- Bench-201 (Dong & Yangl (2020) is tabular benchmark that contains 15k convolutional neural networks, Each architecture is trained on CIFAR-10, CIFAR-100 and ImageNet- 16- (Chrabaszez al. (017 We use the latency values obtained from HW-NAS-Bend' (Li et al. (2021) DARTS et al. (2019) is a supernetwork benchmark that contains 10 18 architectures. Each archi- tecture is trained CIFAR-10 and is transferable ImageNet. We also validate methodology on ImageNet using ProxylessNAS search space Cai et al. (2019) whose size goes to 619 All training hyperparameters listed in Table in Appendix

---

Under review as a conference paper at ICLR 2023
Edge GPU
Mobile Phone: : Pixel
HW
GPU
Architecture
Top-1
Params (M)
Latency (ms)
Top-1
Params
Latency
Aware
Days
Test Acc.
Test Acc.
(M)
(ms)
DARTS
Liu et. al.
68.3 + 0.08
3,4
5.36
68.3 0.08
3.4
11.4
No
4
(2019)
ENAS
Pham
53.89 + 0.16
4.6
6.32
53.89 0.16
4.6
19.8
No
0.16
et al. (2018)
GDAS
Dong &
90.89 + 0.08
3.4
5.21
90.89 = 0.08
3.4
10.36
No
0.21
Yang
20199)
FaI
Chu
93.23+0.18
3.2
4.68
92.4 + 0.15
3.6
8.65
Yes
10
al. (2021)
PRP-NAS-BL
92.34 + 0.05
3.0
2.3
89.54 + 0.07
2.8
3.6
Yes
2
(Ours)
PRP-NAS-BA
94.37 + 0.02
4.5
4.35
94.2 0.03
4.3
5.6
Yes
2
(Ours)
PRP-NAS-O
93.65 + 0.01
4.3
3.64
93.74 0.00
3.4
4.61
Yes
2
(Ours)
Table I: Comparison on NAS-Bench-201 CIFAR-10 on Edge GPU (Jetson Nano) and Mobile phone
(Pixel 3).
4.2 SEARCH RESULTS
In these experiments, we consider two objectives: accuracy and latency (inference time). The latency either given by HW-NAS-Bench (Li al. (202T)), or computed using a lookup table as explained in section
12
Edge GPU Latency (ms)
Other Architsctures
Edge GPU Latency (mm)
100
True Pareto front
10
Pareto front
Ours
8
FairNAS
FairNAS
DARTS
The-Shot-NAAS-CE
One-Shot-HAS-cE
2
30
40
50
60
70
80
90
5
25 ImageNet Accuracy
10
15
20
30
35
40
CIFAR-10 Accuracy
Figure 5: Pareto front approximation comparison on CIFAR-10 and ImageNet.
Figure 5 shows the Pareto front approximations obtained using different methods on NAS-Bench 201 for CIFAR-10 10 and ProxylessNAS Search space for ImageNet. We obtain a 10% hypervolume increase NAS-Bench-200 and 43% hypervolume increase ImageNet compared to the best baselines: One-Shot-NAS-GCN and FairNAS, respectively.
4.2.1 SEARCH ON HAS-BENCH-200
TableI the results of our methodology on NAS-Bench-200 compared to statc-of-the- offtthh-att meth- ods. PRP-NAS-BL PRP-NAS-BA and PRP-NAS-O are three sampled architectures from our final Pareto front. BL stands for "Best Latency". BA stands for "Best Accuracy", and stands for "Op- timal". Notably, our architecture obtains highly competitive results. The optimal architecture, PRP- NAS-O, outperforms current state-of-the -ott--ffthherrr art methods accuracy and latency. Including hardware awareness during the search allows to obtain flexible results according to the targeted hardware platform. Besides, multiple training runs show the stability of our method compared to other base- lines. The acceleration in the search cost is mainly due to applying the pruning while training. This cost can vary according to the used GPU. We used GPU V100 to train the supernetwork. Results on other targeted platforms, can be found in Appendix B

---

Under review as a conference paper at ICLR 2023
4.2.2 SEARCH ON IMAGENET
Similar conclusions can be extracted when searching on ImageNet. Table summarizes the results. Our optimal model surpasses FairNAS-A (+1.9%) and Ome-Shot-NAS-GCN (+1.7%) while running faster. Training on Imagenet time-consuming due the difference image resolution, which explains the increase in the search cost. We still surpass most of the methods in terms of search time. We compare two ProxylessNAS architectures; ProxylessNAS-R specific Mobile inference.
When using data augmentation and architecture tricks, namely Squeere-and-excitaie and Au- toAugment, in the optimal architecture, we achieve 78.6% accuracy on Imagenet. However, this may affect the latency badly. On FPGA ZCU102, the latency increases from 4.63ms to 7.9ms.
4.3 RANKING QUALITY
The ranking preservation measures the quality of the evaluation component in NAS. In HW-NAS, argue that this measure should consider the Pareto ranking instead of the independent ranks of each objective. We compare different estimators used in HW-NAS using Kendall's Tau Correlation between the predicted Pareto ranks and the Pareto ranks obtained from independently training the architectures. These latter are extracted from NAS-Bench-201. Figure 6 shows the correlation results. In general, it is more complex train supernetwork to respect the Pareto ranks because of the impact of the sub-networks on each other, i.e., the outputs of each layer are summed together. The increase in Kendall's 's tau correlation of previous weight-sharin methodology due to the improvement in the accuracy estimation provided by the supernetwork.
Predictor-based evaluators use the learning- torrank theory and train their predictors only predict the ranking. Methods such as GATES (Ning et al. (2020) or BRP-NAS (Dudziak et al. (2020) train many independent predictors, one for each objective. HW-PR-NAS (Benmeziane al. (2022a) trains single predictor to fit the Pareto ranks. However, their methodology is not flexible for supernetwork training.
Kendall's Tau Correlation
0.95
0.84
400
0.78
0.76
0.79
0.73
0.68
300
Hypervolume
0.6
0.47
200
0.4
Kendal Correlation
100
Hypervolume
GATES
BRP-NAS
HELP HW-PR-NAS DARTS ProxylesSNAS FairNAS One-Shot- SharpSep
Ours
NAS-GCN ConVDARTS
Predictor-based HW-NAS
Weight-sharing HW-NAS
Figure 6: Kendall's Tau-b correlation and hypervolume comparison using different estimators
DARTS.
4.3.1 ANALYSIS OF PARAMETER
Figure illustrates the evolution of alpha parameters for each operation in layer and 2 during the training. It clearly shows how alpha favors one operation over the others during training. At the end of the training, we take the operations with the highest alpha that represents the operations constructing architectures in the final Pareto front. If one layer has a clear candidate such as layer 1, with conv3x3 that exceeds 60%, this operation then chosen. layer contains multiple operations with similar alpha values, we constructs all the path of that layer.
4.4 BATTERY USAGE PRESERVATION
The amount of energy consumed by each model can be different. It mainly attributed to the number of multi-adds computed. We take supernetwork usage another level by adequately scheduling the run of different sub-networks according to the system's 's battery life. In this experiment, the training done with two objectives: accuracy and energy consumption. Once the training is done, only the
8

---

Under review as a conference paper at ICLR 2023
Alpha values training for Layer
Alpha values training for Layer
0.5
conv3x3
conv3x3
0.6
pool
pool
alpha values
0.5
identity
convix1
0.4
identity
convix1
0.4
alpha
0.3
0.2
0.2
0.1
0.1
40
40
Epoch
Epoch
Figure 7: Analysis of trained alpha values for layer and
Pareto front solutions are kept in the supernetwork, thanks to the pruning. We further select, from the final Pareto front, s architectures. In this experiment 5. The total size of the supernetwork is then reduced to 20.5MB, comparable to MobileNet-V3 Large with 21.11MB. We deploy the model on smartphone application that is always on. The application repeats the inference classification of one image. The application initially uses the sub-network with the highest accuracy. We switch to lower accurate model every five hours for better energy preserving. Figure&Jshowe shows the results of the system's battery life while running the application for 24 hours. We use three scenarios:
1. Worst Battery Usage: From the Pareto front, we select the most accurate architecture.
This the only architecture the application runs and the only one loaded in memory.
2. Best Battery Usage: Similar to the worst battery usage, select the most energy-efficient.
3. Adequate Battery Usage: We load the complete supernetwork and switch the sub-network every hours.
Using this strategy helps save up to 34% of the battery life while using highly accurate models of the time. The average accuracy of the five selected sub-networks is 75.2%.
80
Battery Capacity
5 CONCLUSION
40
34.19%
This work analyzes Hardware-aware weight- sharing NAS where the multi-objective con- text requires the estimator to preserve the Pareto rankings between sub-networks accu- rately. Contrary to standard baselines that inde- pendently estimate each objective, we propose supernetwork training methodology able to preserve the Pareto rankings during the search. Using methodology, achieve 97% near Pareto front approximation on NAS-Bench- 201, DARTS, and ProxylessNAS Search Spaces. We find 77.2% accuracy model on ImageNet while only training the supernetwork for 3.8 days. Using the supernetwork capabilities, we saved up to 34% of the battery capacity with an average accuracy of 75.2%.
Worse Capacity Usage
Best Capacity Usage
Adequate Capacity Usage
Hour
Figure 8: Battery life management.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mohamed S. Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas Donald Lane. Zero- cost proxies for lightweight NAS. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://
openreview.nete
Lucas S. Batista, Felipe Campelo, Frederico G. Guimarães, and Jaime A. Ramírez. A comparison of dominance criteria in many-objective optimization problems. In Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2011, New Orleans, LA, USA, 5-8 June, 2011, pp- 2359 2366. IEEE, 2011. doi: 10.1109/CEC.2011.5553 20..EEC9.22115999999 URL htttps://doi..oo 1169/CBC 2011. 5949909
Hadjer Benmeziane, Kaoutar El Maghraoui, Hamza Ouarnoughi, Smail Niar, Martin Wistuba, and Naigang Wang. A comprehensive survey on hardware-aware neural architecture search. CoRR, abs/2101.09336, 2021.
Hadjer Benmeziane, Smail Niar, Hamza Ouarnoughi, and Kaoutar El Maghraoui. Pareto rank surrogate model for hardware-awe aware neural architecture search. In International IEEE Sym- posium on Performance Analysis of Systems and Software, ISPASS 2022, Singapore, May 22- 24, 2022, pp. 267-276. IEEE, 2022a. doi: 10. 10.1009/SPSSS551009.... 0002 URL https: 7/doi (rg.i..og//ii 10. 111.1110AS/S1SSS11 2022 2022. C0040
Hadjer Benmeziane, Smail Niar, Hamza Ouarnoughi, and Kaoutar El Maghraoui. Pareto rank surrogate model for hardware-aware neural architecture search. In International IEEE Sym- posium Performance Analysis of Systems and Software, ISPASS 2022, Singapore, May 22- 24, 2022, pp. 267-276. IEEE, 2022b. doi: 10. (009////AAAS55119222222.... 7/doi arooi.oog/100. 11w.::111119/9SSSSSS1111 20211119.2220.....2
URL https:
2022. C0040
Paola Busia, Gianfranco Deriu, Luca Rinelli, Cristina Chesta, Luigi Raffo, and Paolo Meloni. Target-aware neural architecture search and deployment for keyword spotting. IEEE Access, 10: 40687-40700, 2022. doi: 10. WWW.CCESSS.22....0. 66939. URL htttps:://oi......
I1G9/ACCESS 2022. 3669939
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA. May 6-9, 2019. OpenReview.net, 2019. URL httpp::/
met//oount ww..ddyylBBBBB...
Xin Chen, Lingxi Xie, Jun Wu, Longhui Wei, Yuhui Xu, and Qi Tian. Fitting the search space of weight-sharing NAS with graph convolutional networks. In Thirty-Fifth AAAI Conference on Artificial Intelligence, 2021, pp. 7064-7072. AAAI Press, 2021.
Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Xinyu Xiao, and Jian Sun. Det- nas: Backbone search for object detection. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp- 6638 (088-664, 2019. URL
neure ips. httpppppp...pppeeeeppeeppppppppp ccc/paperr/219/haatt 2255555777322267777228bbbaaaa Abstraatt html
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the CIFAR datasets. CoRR, abs/1707.088199 2017. URL httlaattt/a/xii//
abs/1707. 08819
Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Fairnas: Rethinking evaluation faimesss of weight sharing neural architecture search. In International Conference on Computer Vision, ICCV 2021, pp. 12219-12228. IEEE, 2021.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four GPU hours. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 1761-1770. Computer Vision Foundation IEFF, 2019.
10

---

Under review as a conference paper at ICLR 2023
doi: 10.1009/CVPR2201..00188 URL http://openaccess. thw.aacccccsss..c.....cee com wtt.:/cccc...tntne CVPR 2019/ktmm/Dongg_eeeaeaaaa SeessSeearrrnngeeee for_ a Robust_Neural Architecture. in Four GPU Hours CVPR 2019 aaperrhtml
Xuanyi Dong and Yi Yang. Nas-hench-200 Extending the scope of reproducible neural architecture search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:/ /tstps://peeneview. nett
Forum?ic Horum?id-HDxy2kkKK HJxYZKBKDE
Lukasz Dudziak, Thomas Chau, Mohamed S. Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D. Lane. BRP-NAS: prediction-based NAS using gens. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference
on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL (alee:8824aasS0tbbeeeeeeeeee
neuri ps. cc wttpc//ppppa.ee.. pec//appr/202// paaper/2000hhahi. Abstraatt....................aa. html
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:
//openzeview. htt. fort//fommm d= B-
Michael T. M. Emmerich, André H. Deutz, and Jan Willem Klinkenberg. Hypervolume-based expected improvement: Monotonicity properties and exact computation. In Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2011. New Orleans, LA, USA, 5-8 URL https:
June, 2011, pp. 2147-2154. IEEE, 2011. doi: 10.1109/CEC22011.5944999 //doi .rt. O. 11 1109/CEC..... 2011 5949880
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Techun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI, vol- ume 12361 of Lecture Notes in Computer Science, pp. 544 560. Springer, 2020. doi: 10. 10.1007/ URL https:: /ttp:://doi..o /ttp:::/ddi.org/100... 10t. 100
32
Hanzhang Hu, John Langford, Rich Caruana, Saurajit Mukherjee, Eric Horvitz, and Debadeepta Dey. Efficient forward architecture search. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alcéé-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 10122- 2019. URL htttps:/
neurips. pccppapp///119//////e latt..hhhm
Sian-Ya Huang and Wei-Ta Chu. PONAS: progressive one-shot neural architecture search for very efficient deployment. In International Joins Conference Neural Networks, IJCNN 2021, Shen- zhen, China, July 18-22, 2021, pp. 1-9. IEEE, 2021. doi: 10900//CNN553387.2222111 .9533700. 2021. 2022..0332100
URL https: doi. org/ 100 wtw.11100//IN25580
Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, Cong Hao, and Yingyan Lin. Hw-nas-bench: Hardware-aware neural architecture search benchmark. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event Austria, May 3-7, 2021. OpenReview.net, 2021. URL htllhttss///ppeeeviiie net/ -aammmddd00kaa33vVV...
Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot NAS for high-performance image recognition. In 2021 IEEE/CVF International Conference on Computer Vision. ICCV 2021, Montreal, QC, Canada. October 10- 1.00400 URL https:
17, 2021, pp. 337-346. IEEE, 2021. doi: 10. 10010099999992........ 7/doi. org 1109/ W 2021 00040

---

Under review as a conference paper at ICLR 2023
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans. LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL Dettps://ppenevvaii .ew. htw.://wwwwte/oouuuemmmam forum?id
S #1
Eugenio Lomurno, Stefano Samele, Matteo Matteucci, and Danilo Ardagna. Pareto-optimal pro- gressive neural architecture search. In Krzysztof Krawiec (ed.). GECCO '21: Genetic and Evolu-
fionary Computation Conference, Companion Volume, Lille, France, July 10- 14, 2021. pp. 1726 1734. ACM, 2021. doi: 00.11457344777633463146 3449726, 3463146
URL https: /cts::///cceee org
Vasco Lopes, Saeid Alirezazadeh, and Luís A. Alexandre. EPE-NAS: efficient performance timation without training for neural architecture search. In Igor Farkas, Paolo Masulli, Se- bastian Otte, and Stefan Wermter (eds.), Artificial Neural Networks and Machine Learning ICANN 2021 30th International Conference on Artificial Neural Networks, Bratislava, Slo- vakia, September 14-17, 2021, Proceedings, Part V, volume 12895 of Lecture Notes in Com- puter Science, pp. 552 563. Springer, 2021. doi: 10. 1001/9777 3-0-0-0000--0000 86383-8) URL https://doi http:/oi...ggg/// 1ttp:/10000000111131110 1--17/998---000--33----------3-- 44
Zhichao Lu, Kalyanmoy Deb, Erik D. Goodman, Wolfgang Banzhaf, and Vishmu Naresh Boddeti. 2sganetv2: Evolutionary multi-objective surrogate- assisted neural architecture search. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part vol- ume 12346 of Lecture Notes in Computer Science, pp. 35-51. Springer, 2020. doi: 10. 1007/ 100---030058455---
URL puttps://dos org 1 0 1007 978 3- 030 58452 8_3
Xuefei Ning, Yin Zheng, Tianchen Zhao, Yu Wang, and Huazhong Yang. A generic graph-based neural architecture encoding scheme for predictor-based NAS. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.). Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28 2020, Proceedings. Part XIII, volume 12358 of Lecture
Notes in Computer Science pp. 204.004.6600 Springer, 2020. doi: URL httpp:://doi.or/// 10t.://1.111000010 1007/ 3-1-9788-----------
Jiefeng Peng, Jiqi Zhang, Changlin Li, Guangrun Wang, Xiaodan Liang, and Liang Lin. Pi-nas: Improving neural architecture search by reducing supernet training consistency shift. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pp. 12334-12222 12344. IEEE, 2021. doi: 10. 001109//CCV449922222000111
URL https:ttt///t...t. /doi wtw.doi..rgr///oo 10. 10w.1111/9////92...... 291144888222222..00 202..012221111
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssaan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 4092- 4101. PMLR, 2018. URL / htttp::/proceedinggs. mir press / / phamphaammmmmmmmm 8a html
Santanu Santra, Jun Wei Hsieh, and Chi-Fang Lin. Gradient descent effects differential neural architecture search: A survey, IEEE Access, 9:89602-89618 2021. doi: 10. ACCIESSS...
3090918. URL https:://oois org /ttp::::://oii.r//110100.111.1.0 1109 ACCESS 2021 3090918
Lukas Sekanina. Neural architecture search and hardware accelerator co-search: A survey. IEEE Access, 9:151337-151362, 2021. doi: 10. (09/ACCEES.2021.311
Linnan Wang, Chenhan Yu, Satish Salian, Slawomir Kierat, Szymon Migacz, and Alex Fit-Florea. Gpunet: Searching the deployable convolution neural networks for gpus. CoRR, abs/2205.0088 000441 2022. doi: 10. 10.485550/rXivv22255.00 7205.00841
URL :tt::://ooooooo /
Colin White, Arber Zela, Robin Ru, Yang Liu, and Frank Hutter. How powerful are performance predictors in neural architecture search? In Marc' Aurelio Ranzato, Alina
12

---

Under review as a conference paper at ICLR 2023
Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems 34: Annual Conference on Neural Infor- mation Processing Systems 2021. NeurIPS 2021, December 6-14, 2021, virtual, pp. 28454- cc/ wtt.cc/ppppp.... pttp:/pperr221111// / 28469, 2021. URL 050.0005555000222000770722-0-0.0
Yuge Zhang, Zejun Lin, Junyang Jiang, Quanlu Zhang, Yujing Wang, Hui Xue, Chen Zhang, and Yaming Yang. Deeper insights into weight sharing in neural architecture search. CoRR, abs/2001.01431. 2020.
Yiyang Zhao, Linnan Wang, Yuandong Tian, Rodrigo Fonseca, and Tian Guo. Few-shot neural architecture search. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 12707-12718. PMLR, 2021. URL http:// . mlr. gress/v1339/haa2ldd /r html
13

---

Under review as a conference paper at ICLR 2023
A
RESULTS ON IMAGENET
Edge GPU
Mobile Phone: Pixel
HW Aware
GPU
Architecture
Top-1
Params
Latency (ms)
Top-1
Params
Latency (ms)
Days
Test Acc.
(M)
Test Acc.
(M)
DARTS
L Liu et al.
73.3 + 0.03
4.7
5.36
73.3 # 0.03
4.7
11.5
No
4
(2019)
Proxylesse
Ca
75.1 + 0.00
7.1
5.1
Yes
8.3
et al. (2019)
ProxylessNAS
74.6
4.0
6.8
Yes
8.3
R Cai et al.
(2019)
FairNAS-A (Chu
0.05
4.6
5.32
75.1 = 0.06
4.3
6.84
Yes
12
et al. (2021)
One- One-Shot-NAS-
4.4
5.23
75.5 0.09
4.4
9.4
No
4.7
GCN Chen et al.
(202T)
PRS-NAS-BL
68.95 + 0.01
3.7
3.2
67.56 0.05
3.1
4.8
Yes
3.8
(Ours)
PRP-NAS-BA
77.5 0.02
4.8
4.68
76.94
0.02
4.4
6.13
Yes
3.8
(Ours)
PRP-NAS-O
0.03
4.6
3.68
75.6
0.02
3.8
5.68
Yes
3.8
(Ours)
Table 2: Comparison on ImageNet on Edge GPU (Jetson Nano) and Mobile Phone (Pixel 3).
B ADDITIONAL RESULTS
Table 3 shows the results of our training methodology on FPGA ZCU 102 and Raspberry Pi3. Our methodology consistently outperforms state-of-the-art methods different hardware platforms.
FPGA ZCU102
Raspberry Pi3
HW Aware
GPU
Architecture
Top-1
Params (M)
Latency (ms)
Top-1
Params (M)
Latency
Days
Test Acc.
Test Acc.
(ms)
DARTS
68.3 # 0.08
3.4
7.32
68.3 + 0.08
3.4
45.36
No
ENAS
53.89 + 0.16
4.6
8.91
53.89 0.16
4.6
35.8
No
0.16
GDAS
90.89 0.08
3.4
4.98
90.89
0.08
3.4
41.8
No
0.21
FairNAS
92.920.23
3.4
5.12
92.31 0.9
3.3
34.15
Yes
10
PRP-NAS-BLE (Ours)
91.35 + 0.04
3.2
3.6
88.7 +
2.4
7.6
Yes
PRP-NAS-BA (Ours)
94.37 0.01
4.9
6.8
93.68
0.05
4.68
40.7
Yes
PRP-NAS-O (Ours)
93.55 # 0.04
4.2
4.23
92.54 + 0.02
3.6
18.5
Yes
2
Table 3: Comparison to baselines on CIFAR-10 on FPGA ZCU-102 and Raspberry Pi3
C NUMBER OF SAMPLED SUB-NETWORKS
Figure shows the effect of increasing the number of sampled sub-networks on the search results. Generally, increasing the number samples, increases the hypervolume. The hypervolume used to evaluate Pareto front approximations. It computes the area contained by the Pareto front points found by the search and a reference point. Our reference point is set as a pre-sampled architecture from the supernetwork, with low accuracy and high latency. When the number of sampled sub- networks is too high, each layer's output the sum of multiple operations that can or cannot be within the final Pareto front which induces bias when adjusting the alpha parameters.
14

---

Under review as a conference paper at ICLR 2023
400
Hypervolume
350
300
250
20
40
100
Increasing number of sub-networks
Figure 9: Hypervolume analysis with increasing number of sampled sub-networks for the final Pareto front throughout the search (higher is better) on NAS-Bench-201.
D PRUNING ALGORITHM
We validate the results of our pruning algorithm by comparing the results of our algorithm with and without it in table Obviously without the pruning, the search time exponentially increases from 3.8 GPU days to 8.1. However, the hypervolume improves slightly. The final most accurate architecture is in both Pareto front obtained with and without pruning. The optimal architecture using pruning is better in terms of accuracy and latency. The latency computed on Jetson Nano Edge GPU.
Model
Test Acc (%)
Latency (ms) Search Hypervolume
GPU days
PRP-NAS-O
93.65
3.64
423,45
3.8
PRP-NAS-O-no-prute pruning
92.1
3.26
433.09
15.1
Table 4: Ablation results of Pruning of Pareto ranking,
E LATENCY ESTIMATION
In this section, we compare different latency estimators to validate the use of LUT during the search. We randomly extract 1000 architectures from NAS-Bench-201 and 1000 from DARTS. We measure the exact latency on Jetson Nano for each architecture. We train two predictor-based models, namely XGBoost and MI. with 3 layers. The training dataset contains 700 architectures and 300 were used for testing. On the most accurate in terms latency ranking the architectures. On DARTS, XGBoost prediction was the most suitable methods. But, LUT was not far with 0.915 against 0.942. Computing the LUT in our algorithm is simple. Using a hook during the forward function on a PyTorch model is sufficient and much more direct than calling surrogate model. We thus this strategy estimate the latency in our method.
the architectures have sequential execution which made LUT
15

---

Under review as a conference paper at ICLR 2023
(a) Results on 1000 sampled networks from NAS-Bench-201
(b) Results on 1000 sampled
networks from DARTS
1.0
Kendall's Tau Correlation
Correlation
0.8
0.8
0.6
0.6
0.4
Kendall's
0.2
0.2
0.0
0.0
LUT
XGBoost
MLP
LUT
XGBoost
MLP
Estimation Methods
Estimation Methods
Figure 10: Comparison of latency estimators on Jetson Nano.
Benchmark
Hyperparameter
Value
Nf
20
Np
50
F TRAINING HYPERPARAMETERS
50
NAS-Bench-201
batch_size
128
The training hyperparameters listed in Table 5 takes 2, 3.8, 3.8 GPU-days for NAS-Bendh 20I, DARTS and ProxylessNAS search space to train each supernetwork to fullness. Our training is 5x faster than previous works due the pruning strat- egy. To be consistent with previous works, we do employ data augmentation tricks such as cutout or mixup. We also do not employ any special op- erations such as squeeze-and-exx anxeeet-and-exiitatiio... All these methods can further improve the scores the test set.
Ir
0.01
optim
SGD
momentum
0.9
Nr
30
Np
150
100
DARTS
batch_size
256
Ir
0.025
optim
SGD
momentum
0.9
Nf
30
Np
150
ProxylessNAS Search Space
n
100
batch_size
256
Ir optim
0.025
SGD
momentum
0.9
Table 5; Training Hyperparameters
16


Reviews:
[{"review_id": "sFwezOd3BS", "reviewer": "Reviewer_Z8Vf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an hardware-aware NAS on a superset that preserves the Pareto ranking (accuracy and latency). It demonstrates better accuracy-latency tradeoff when compared to SOTA approaches.", "summary_of_the_review": "* This paper presents an hardware-aware NAS on a superset that consider preservation of Pareto-ranking.\n* How is the operation score parameter calculated? Is this the same as the architecture parameter in DARTS? Is this score a good representation of the importance of operation?\n* Similarly, the line “Adjust alpha” in Algorithm 1 is not clearly explained.\n* The notations in Section 3.2 are slightly confusing, operation score (alpha), Pareto-rank score (alpha), Pareto score (P_s).\n* The latency model is based on the summation of layer-wise latency stored in a lookup table. Is this assuming sequential execution of layers in the target hardware? Is this assumption true for the target hardware?\n* Are the latency numbers reported in Table 2 and 3 based on real measurement or the latency model?\n* Can you provide more comparison to SOTA HW-aware NAS, e.g. BRP-NAS [1] and HELP [2] similar to Table 2 and 3?\n\nRef:\n[1] https://arxiv.org/abs/2007.08668\n[2] https://arxiv.org/abs/2106.08630", "strength_and_weaknesses": "Strength\n* Good trade-off between latency and accuracy is achieved using the proposed approach comparing to the SOTA.\n\nWeakness\n* It is unclear how the alpha parameter is derived and how does the correctness of accuracy / latency modelling affect the quality of results.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WTvesyiyXXA", "reviewer": "Reviewer_UeSf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method for multi-objective Neural Architecture Search (NAS), i.e., finding the best NAs based both on task prediction and hardware-based metrics (e.g., latency).  The proposed method (HW-NAS, HW for \"hardware\") builds on single-stage supernetwork techniques (the union of possible architectural choices for the neural final architecture) by co-training the supernetwork's parameters and a \"Pareto\" parameter to find (and rank) the sub-models.   They propose to use Pareto-ranks as the primary search objective and NAS evaluation criteria.  They implemented their algorithm and compare the found rank of the models on the Pareto frontier with Pareto rank of the individually trained sub-models for multiple benchmarks.   In addition the trained Pareto parameter allows them to decrease supernetwork training time by pruning sub-networks not be on the Pareto front.    By doing so, they achieve a 97% Pareto front approximation (vs 87%) for the resulting sub-models. ", "summary_of_the_review": "This looks like an interesting and fruitful direction of research - cotraining the supernetwork with a Pareto-based parameter to guide the search.  However, there are presentation, clarity, and background work issues that make it difficult to place the work into context, interpret the results, and understand the algorithm enough for reproducibility. \n\n[Post Author Response] The authors have addressed significant issues regarding the lack of details, including more detailed training steps, specifying the Pareto parameter's loss function, and including new results.   Many of these details should have been present in the original paper. but I've bumped the recommendation to marginal below in light of them.    ", "strength_and_weaknesses": "Positive Points:\n+ Multi-objective NAS is an important area, and the proposed method (Pareto-based search training) appears novel\n+ Evaluation used 3 benchmarks/search spaces (DARTs, NAS-Bench-201, ProxylessNAS) and additionally compares to FairNAS and a recent one-shot GCN-based technique. \n+ Results are promising: optimal sub-networks often meet or improve accuracy/latency while also taking less time to train.  \n\nNegative Points\n- The paper is written as if optimizing for multiple objectives was a relatively new idea.  But searching for models on the Pareto front has been previously explored in the NAS area.  Related work doesn't describe much of the work in that multi-objective space.   \n- The description of the algorithm is light on details, specifically the Pareto loss function, the Pareto parameter updates, and how NAs are found.  This makes reproducibility difficult. \n- Given the above limited description of the operation of the Pareto-based training, it isn't clear how easy it is to extend to multiple objectives. \n- There is little to no evaluation of the Pareto parameter training in isolation to understand its behavior / performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HZ9ksgiq0R", "reviewer": "Reviewer_4mMw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a supernetwork training strategy for preserving the Pareto ranking between different subnetworks for handware-aware architecture search. To maintain a higher ranking correlation supernetwork, the authors propose to prune the operations based on the optimal Pareto front during the training of the supernetwork. Moreover, the authors define a new metric for better evaluating the architecture in a multi-objective context. Experimental results on three benchmarks demonstrate the effectiveness of the algorithm in some cases. However, the proposed method is not clearly described and the experiments can be further improved. My detailed comments are as follows.", "summary_of_the_review": "I vote for reject since the proposed method is not clearly described and several key ablations are missing in the paper.", "strength_and_weaknesses": "### Positive points:\n\n- The authors introduce the Pareto ranking as a novel metric for multi-objective architecture search. \n\n- The authors propose a supernetwork training strategy for preserving the Pareto ranking between different subnetworks.\n\n - The authors propose to prune the operations based on the optimal Pareto front while training the supernetwork.\n\n### Negative points:\n\n - In the Pareto rank-preserving training section, the details of the proposed method are missing. In other word, after reading this paper, I can not re-implement the proposed method. It would better for the authors to make it more clear and detailed.\n   - a)\tThe description of the listwise ranking loss is not clear. More explanations are needed.\n   - b)\tDetailed descriptions for obtaining the Pareto ranks (ground truth) for different subnetworks are needed. \n   - c)\tThe detail of the hypervolume improvement should be provided since it’s an important metric for pruning the operations. Besides, the motivation for choosing HVI is unclear. Moreover, the number of sample networks n seems important for the proposed method. How to choose a good n?\n - Estimating architecture latency using a lookup table has been proposed and investigated in FBNet. In my opinion, this is not a good way to evaluate the latency of candidate architectures. In practice, the inference framework (e.g., TensorRT) would perform layer/operation fusion. For instance, TensorRT will fuse conv and batch norm layer into a single layer. In this case, the overall latency of the architecture is not equal to the sum of that of each layers. The reality is often more complicated than the above case. Thus, I think it would be difficult to accurately estimate the latency based on the lookup table.\n - In Eq. 1, the authors state $F_{k’}$ is a set of sub-networks ranked k’. I have no idea what is $F_{k’}$ even aftering carefully reading it. More expalinations should be provide.\n - The ablation studies on “the number of sampled sub-network” and “the number of epochs for Pareto-training” are missing. It would be better for the authors to provide more ablation studies.\n - Some other multi-objective search algorithms [1][2] should also be compared in Tables 2 and 3.\n - The details of the experimental setting in Figures 3 and 5 are unclear. More explanations are needed.\n - The ablation study on the proposed pruning strategy is missing. More experiments are expected.\n\n### Minor issues:\n\n - In Section 3.2, “Algorithm 1 and figure 1 summarizes summarizes the training procedure” should be “Algorithm 1 and figure 1 summarizes the training procedure”.\n\n\n### Reference\n\n[1] Nsganetv2: Evolutionary multi-objective surrogate-assisted neural architecture search ECCV 2020.\n\n[2] Ponas: Progressive one-shot neural architecture search for very efficient deployment.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2Kb5omOeR3", "reviewer": "Reviewer_qrrY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors introduce a methodology for pareto front sub-network identification from a supernetwork. This is done in a two step process, by first conducting fair subsampling of subnetworks for Nf epochs, followed by Pareto-Rank  training for Np epochs. Sub-networks that are furthest from the pareto front are also dropped to accelerate training. The proposed method achieves state of the art pareto front approximation (97%) in 2 GPU days. A new parameter, $\\alpha$ is introduced to measure which operation is critical, and pareto-ranking loss is utilized to adjust the $\\alpha$ for different paths. The sum of the selected operations alpha values is utilized as the pareto score and Cross Entropy Loss is taken between the ranking scores and pareto ranks (ground truth) to update $\\alpha$. ", "summary_of_the_review": "The paper is interesting but there are some concerns (see above).", "strength_and_weaknesses": "Strengths: \n\n- The idea is straightforward to implement on existing supernetwork based training schemes, and seems to be effective from the results.\n- The methodology is validated on NASBench201, DARTS and ImageNet. \n\nWeakness:\n\n- It is not clear how the second phase of training (argmax(a)) enforces fairness. Justification and implication of unfair sub-network sampling in this stage may be a useful discussion. \n- A table showing the pareto approximation effectiveness for each of the tested search spaces would be very useful.\n- The current results highlight the fact that PRP-NAS-BL finds a model with low latency, and PRP-NAS-BA finds a model with high accuracy. (As expected.) If possible, comparing with the ground truth optimal architectures in each case may add more value/context to the result. \n\nQuestions:\n\n- If argmax(a) selection scheme does not enforce fairness, it is natural to assume that the correlation between argmax(a) and kendall tau rank correlation would increase simply due to the nature of the CELoss minimization, with no bearing on the ground truth. How is the 'Truth' in the CELoss(Score, Truth) formulation calculated?\n- If Truth is a list of intermediate accuracies of the sub-networks, then naturally the kendall tau rank correlation will keep increasing, simply due to sampling bias. Comments on this issue would be appreciated. \n\nMinor comments:\n\n- Table 1 (Training hyperparameters) can be moved to the Appendix.\n- Fix 3.2 : 'Algorithm 1 and Figure 1 summarizes summarizes..' ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject