Zero-Cost Operation Scoring in Differentiable Architecture Search
Authors: Lichuan Xiang
Łukasz Dudziak
Mohamed S Abdelfattah
Thomas Chun Pong Chau
Nicholas Donald Lane
Hongkai Wen
Venue: ICLR 2022 Submitted (2022)

Abstract: Differentiable neural architecture search (NAS) has attracted significant attention in recent years due to its ability to quickly discover promising architectures of deep neural networks even in very large search spaces. Despite its success, many differentiable NAS methods lack robustness and may degenerate to trivial architectures with excessive parameter-free operations such as skip connections thus leading to inferior performance. In fact, selecting operations based on the magnitude of architectural parameters was recently proven to be fundamentally wrong, showcasing the need to rethink how operation scoring and selection occurs in differentiable NAS. To this end, we formalize and analyze a fundamental component of differentiable NAS: local "operation scoring" that occurs at each choice of operation.
When comparing existing operation scoring functions, we find that existing methods can be viewed as inexact proxies for accuracy.
We also find that existing methods perform poorly when analyzed empirically on NAS benchmarks. From this perspective, we introduce new training-free proxies to the context of differentiable NAS, and show that we can significantly speed up the search process while improving accuracy on multiple search spaces. We take inspiration from zero-cost proxies that were recently studied in the context of sample-based NAS but shown to degrade significantly for larger search spaces like DARTS. Our novel "perturbation-based zero-cost operation scoring" (Zero-Cost-PT) improves searching time and accuracy compared to the best available differentiable architecture search for many search space sizes, including very large ones. Specifically, we are able improve accuracy compared to the best current method (DARTS-PT) on the DARTS CNN search space while being over 40x faster (total searching time 25 minutes on a single GPU). Our code is available at: https://github.com/avail-upon-acceptance.

Under review as a conference paper at ICLR 2022
ZERO-COST OPERATION SCORING IN DIFFERENTIABLE ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Differentiable neural architecture search (NAS) has attracted significant atten- tion in recent years due to its ability quickly discover promising architectures of deep neural networks even in very large search spaces. Despite its success, many differentiable NAS methods lack robustness and may degenerate to trivial architectures with excessive parameter-free operations such as skip connections thus leading to inferior performance. In fact, selecting operations based on the magnitude of architectural parameters was recently proven be fundamentally wrong, showcasing the need to rethink how operation scoring and selection oc- curs in differentiable NAS. To this end, we formalize and analyze a fundamental component of differentiable NAS: local "operation scoring" that occurs at each choice of operation. When comparing existing operation scoring functions, we find that existing methods can be viewed inexact proxies for accuracy. We also find that existing methods perform poorly when analyzed empirically on NAS benchmarks. From this perspective, we introduce new training-free proxies to the context of differentiable NAS, and show that we can significantly speed up the search process while improving accuracy on multiple search spaces. We take inspiration from zero-cost proxies that were recently studied in the context of cample-based NAS but shown to degrade significantly for larger search spaces like DARTS. Our novel "perturbation-base" Cost-PT) improves searching time and accuracy compared to the best available differentiable architecture search for many search space sizes, including very large ones. Specifically, we able improve accuracy compared to the best current method (DARTS-PT) PTT on the DARTS CNN search space while being over 40 faster (total searching time 25 minutes on a single GPU). Our code is available at: https://gi com avai wervail-uuaen upon acceptance
zero-cost operation scoring" (Zero-
INTRODUCTION
Since the recent dawn of deep learning, researchers have designed new architectures of neural networks on an unprecedented scale, with more efficient and accurate networks being proposed each year (landola et al. 2016 Howard et al.) 2017: Tan & Le 2019 2021). However, the manual design of better DNN architectures has proven quite challenging, requiring extensive domain knowledge and persistent trial-and-ent -nd-rorr in search of the optimal hyperparameters (Sandler al. [0018] Tan &
Le 20 Recently this process has been successfully aided through automated methods, especially neural architecture search (NAS) which can be found behind many of the ofatc-of-the-art deep neural networks (Real et al. 2019 Wu et al. 2019 Cai et al., 2020 Moons et al., 2020). However, one of the biggest problems in NAS is the computational cost even training single deep network can require enormous computational resources and many NAS methods need to train tens, if not hundreds,
of networks in order to converge to a good architecture (Real et al. 2019 Luo et al. 2018 Dudziak al. 2020). A related problem concerns the search space size larger search space would typically contain better architectures, but requires longer searching time (Real et al.] 2019).
Differentiable architecture search (DARTS) was first proposed to tackle those challenges, showcasing promising results when searching for network set of over 10 possible variations (Liu et al. 2019). Unfortunately, DARTS has significant robustness issues as demonstrated through many recent works (Zela al. 2020a Shu et al.) 2020 Yu al., 2020). It also requires careful selection of hyper- parameters which makes it somewhat difficult to adapt to a new task. Recently, Wang et al. (2021) showed that operation selection in DARTS based on the magnitude of architectural parameters (a) is fundamentally wrong, and will always simply select skip connections over other more meaningful operations. They proposed an alternative operation selection method based on perturbation, where the importance of an operation is determined by the decrease of the supernet's validation accuracy

---

Under review as a conference paper at ICLR 2022
when it is removed. Then the most important operations selected by exhaustively comparing them with other alternatives on each single edge of the supernet, until the final architecture is found.
In a parallel line of work that aims to speed up NAS, proxies are often used instead of training accuracy to obtain indication of performance quickly without performing expensive full training for each searched model. Conventional proxies typically consist of reduced form of training with fewer epochs, less data or smaller DNN architecture (Zhou et al.) 2020). Most recently, zero-cost proxies, which are extreme types of NAS proxies that do not require any training, have gained interest and are shown to empirically outperform conventional training-based proxies and deliver outstanding
results on common NAS benchmarks (Abdelfattah et al. 2021 Mellor al. (021). However, their efficient usage on a large search spaces, typical for differentiable NAS, has been shown to be more challenging and thus remains an open problem (Mellor et al.) 2021).
The objective of our paper is shed some light onto the implicit proxies that are used for operation scoring in differentiable NAS, and to propose new proxies in this setting that have the potential of improving both search speed and quality. We decompose differentiable NAS into its two constituent parts: (1) supernet training and (2) operation scoring. We focus on the second component and we formalize the concept of "operation scoring" that happens during local operation selection at each edge in supernet. Through this lens, we are able empirically compare the efficacy of existing differentiable NAS operation scoring functions. We find that existing methods act as a proxy to accuracy and perform quite poorly operation scoring functions based on zero-cost proxies that outperform existing methods in both search speed and accuracy. Our main contributions are:
common NAS benchmarks, consequently, we propose new
1. Formalize "operation scoring" in differentiable NAS and perform a first-of-its-king analysis of the implicit proxies that are present in existing differentiable NAS methods.
2. Propose, evaluate and compare perturbation based zero-cost operation scoring (Zero-Cost-Pa) for differentiable NAS building upon recent work on training-free NAS proxies.
3. Perform a thorough empirical evaluation of Zero-Cost-PT in six search spaces and datasets including DARTS, DARTS subspaces S1-S4 and
2 RELATED WORK
Classic NAS and Proxies. Zoph & Lee were among the first to propose an automated method to search neural network architectures, using a reinforcement learning agent to maximize rewards coming from training different models (Zoph & Le| 2017). Since then, a number of alternative approaches have been proposed in order to reduce the significant cost introduced by training each proposed model. In general, reduced training can be found in many NAS works (Pham al., 2018 Zhou et al. 2020). and different proxies have been proposed. c.g. searching for a model on a smaller dataset and then transfer the architecture to the larger target dataset (Real et al. [019] Mehrotra et al.
202T) or incorporating a predictor into the search process (Wei et al. 2020 Dudziak et al. 2020 Wu et al.
2021 Wen et al. (2019).
Zero-cost Proxies. Very recently, zero-cost proxies (Mellor et al. 2021 Abdelfattah et al. 202 for NAS emerged from aaninng-t-initiallin
literature (Tanaka et al.) 2020) Wang et al., 2020 Lee et al. 2019 Turner et al. 2020). Such proxies can be formulated as architecture scoring functions 5(A) that evaluate the potential or "saliency" of given architecture A in achieving accuracy at initialization, without the expensive training process. In this paper, we adopt the recently proposed zero-cost proxies (Abdeliattah et al. 2021 Mellor et al. (20TT) namely grad__oo _o snip, grasp, synflow, fisher and nwot. Those metries either aggregate the saliency of model parameters to compute the score of an architecture (Abdelfattah et al. (022). or use the overlapping of activations between different samples within a minibatch of data In a similar vein, Chen et al. (2021) proposed the use of training- free scoring for operations based on the neural tangent kernel (Jacot et al. (02I)) and number of linear regions in a DNN; the operations with the lowest score are pruned from the supernet iteratively until a subnetwork is found.
performance indicator (Mellor et al.) 2021).
Differentiable NAS and Operation Perturbation. Liu et al. first proposed to search for neural network's architecture by parameterizing it with continuous values (called architectural parameters a) in differentiable way. Their method constructs a supernet, i.c., a superposition of all networks in the search space, and optimizes the architectural parameters (a) together with supernet weights (w). The final architecture extracted from the supernet by preserving operations with the largest a. Despite the significant reduction in searching time, the stability and generalizability of DARTS have been challenged, e.g., may produce trivial models dominated by skip connections (Zela et al.
2

---

Under review as a conference paper at ICLR 2022
n
d ,00, o
e"=[o" "o"" on
d
perturb of )
discrelize o²
At (6 ,00)
n!3
n²
eP , e e?!n(o"M,oP?
e
n
Figure 1: Visualization of perturbation and discretization of an edge in a supernet. Middle: a supernet
is composed of three edges {e 1,2,3 each consisting of three possible operations [ 1,2,3
1,2,3
which are applied in parallel to the same input. Left: edge e(2) e is perturbed by removing (11) from
the set of candidate operations assigned to this edge. Right: the same edge e(2) is discretized with operation o!4) by removing all other candidate operations leaving o(2) as the only choice left.
2020a). SDARTS (Chen Hsieh, 2020) proposed overcome such issues by smoothing the loss landscape. while SGAS (L et al. (0200 considered a greedy algorithm to select and prune operations sequentially. The recent DARTS- (Wang et al. 2021) proposed a perturbation- -based operation selection strategy, showing promising results on DARTS space. In DARTS-PT operations are no longer selected by optimizing architectural parameters (a), but via scoring function evaluating the impact on supernet's validation accuracy when the operation is removed.
RETHINKING OPERATION SCORING IN DIFFERENTIABLE NAS
In the context of differentiable NAS, a supernet would contain multiple candidate operations on each edge as shown in Figure Operation scoring functions assign a score to rank operations and select the best one. In this section, we empirically quantify the effectiveness of existing operation scoring methods in differentiable NAS, with a specific focus on DARTS (Liu et al. 2019) and the recently-proposed DARTS-PT (Wang et al.] (221)) l). Concretely, we view these scoring functions as proxies for final subnetwork accuracies and we evaluate them on that basis to quantify how well these functions perform. We challenge many assumptions made in previous work and show that we can outperform existing methods with lightweight alternatives.
3.1 OPERATION SCORING PRELIMINARIES
For supernet A we want to be able to start discretizing edges in order to derive subnetwork. When discretizing we replace an edge composed of multiple candidate operations and their respective (optional) architectural parameters with only one operation selected from the candidates. We will denote the process of discretization of an edge with operation o, given a model A, as: A + (c, o). Analogously, the perturbation of a supernet A by removing an operation o from an edge e will be denoted as A (c, o). Figure [ illustrates discretization and perturbation. Furthermore, use A. & and O to refer to the set of all possible network architectures, edges the supernet and candidate operations, respectively extra details about notation can be found in the Appendix I
will
NAS then be performed by iterative discretization of edges in the supernet, yielding in the process a sequence of partially discretized architectures: Ao, AI..., A Aig,, where Ao is the original supernet, Alel is the final duste-diseee subnetwork (result of NAS), and A, is At-1 after discretizing next edge, i.e., A, A,-1 + (et, ox) where is an iteration counter. The problem of finding the sequence of (e,, o.) that maximizes performance of the resulting network Ag has an optimal substructure and can be reduced to the problem of finding the optimal policy π A È + O that is used to decide on an operation to assign to edge at each iteration, given current model (state). This policy function is defined by means of an analogous scoring function f A x x O - IR, that assigns scores to the possible values of the policy function, and the policy is then defined as arg max or arg min over f. depending on the type of scores produced by f-
We begin by defining the optimal scoring function that we will later use to assess quality of different, practical approaches. For given partially-discretizee discretized model A,, let denote the set of all possible fully-discretized networks that can be obtained from Ar after a next edge e is discretized with an operation o as Arc.o. Our optimal scoring function can then be defined as:
Toest-ace ( (Ar, e) arg max max
V (A(g))
(1)
oEO. Ale
'Since scoring function clearly defines relevant policy function, we will sometimes talk about scoring function even though the context might be directly related to policy function in those cases it should be understood as the policy function that follows from the relevant scoring function (and vice versa).
3

---

Under review as a conference paper at ICLR 2022
where V is validation accuracy of a network after full training (we will use V to denote validation accuracy without training). It is easy to see that this policy meets Bellman's principle of optimal- ity (Bellman, 1957)- in fact its definition follows directly from it and therefore is the optimal solution to our problem. However, it might be more practical to consider the expected achievable accuracy when an operation is selected, instead of the best. Therefore we also define the function Tagg
Tavg-ace (A,, e)
arg max E (A(gg)
(2)
oEO, Ale,EAr.c.o
In practice, we are unable to use either Thest-we or Tavg-ace since we would need to have the final validation accuracy V of all the networks in the search space. There have been many attempts at finding approximate operation scoring functions, in the following we consider the following practical alternatives from DARTS (Liu et al. 2019) and DARTS-P7 (Wang et al. 202T]:
arg max Oieoo OEO, disce-ace (Ac,e) arg max (* + (c, Tdarts pe (At,e) arg min V(A, - (e) oEO,
Taarrs (At,e)
(3)
(4)
oEO,
where ae.o is the architectural parameter assigned to operation on edge e presented in DARTS (Liu al. 2019). Tdise-ace uses accuracy of a supernet after an operation is assigned to an edge e this is referred to as "discretization accuracy" in DARTS-PT and is assumed to be a good operation scoring function (Wang et al. 2021), most intuitively, it could approximate favg-ace- Tdutts-i is the perturbation- approach used by DARTS-PT is presented as a practical and lightweight alternative to Tdise-ace (Wang et al. 2021).
Zero-Cost Operation Scoring. We argue that the scoring functions (3) and 4 are merely proxies for the best achievable accuracy (Equation[T] As such, we see an opportunity to use a new class of training-fire proxies that are very fast to compute and have been shown to work well within NAS, albeit not in differentiable NAS, nor within large search spaces. We present the following scoring functions that use a zero-cost proxy S instead of validation accuracy when discretizing an edge or perturbing an operation. Note that the supernet is randomly-initializen and untrained in this case.
Te (Ar, E) = arg max S(Ar + (e, o)), oEO,
Tze-pe(Ae, e) = arg min S(A: - (e,o))
(5)
oEO.
Note that TE-NAS (Chen et al.) (02T)) also uses training-fres scoring of operations, however, they use different scoring metries prume operations from supernet opposed to discretizing or perturbing
operations as we show above. We include a comparison to TE-NAS throughout our paper.
3.2 EMPIRICAL EVALUATION OF OPERATION SCORING METHODS
In this subsection we investigate the performance of different operation scoring methods. Because we want to compare with the optimal fbest-ace and Jawg-ace- we conduct experiments on NAS-Bench-201 which contains the validation accuracy for all 15,625 subnetworks in the search space (Dong & Yang 2020). We conduct our investigation in two settings, initial and progressive. The first setting compares operation scoring functions while making their first decision (iteration 0) during NAS. The second (progressive) setting takes into account retraining that occurs between iterations for some of the differentiable NAS algorithms that we consider like darts-pt (Wang et al. 2021). Throughout this
section, whenever we discuss a method based on zero-cost scoring, we use naswot metric Meelorr
et al. 2021).
3.2. 1 INITIAL OPERATION SCORING
For the supernet An we compute the operation scores for all operations all edges, at the first iteration (iteration 0) of NAS, that is, (Au, e, o) A e E E, e O.. In our first experiment, we collect the scores produced by different scoring methods, per operation, per edge, then compute the Spearman rank correlation for operations on each edge, and finally average the rank correlation coefficient over all edges (details of our experiments and illustrative examples are provided in Appendix A.3). The resulting averaged rank correlation is indicative of how well an operation scoring method would do when making the first discretization decision, relative a perfect "oracle" search. We plot the rank correlation coefficients in Figure 2a showing many surprising findings. First, disc-ace is inversely correlated to best-acc. This refutes the claim in the DARTS-PT paper that disc-ace is a reasonable operation score (Wang et al. [202]) - these findings are aligned with prior work that has already shown that the supernet accuracy is unrelated to the final subnetwork accuracy (Li et al. (020). Second, the darts-pt score does not track disc-acc, in fact, it mnversely-oorrlane to it as well. meaning that the darts-pt score is not a good approximation of disc-acc. However, darts-pt is ceakly-correlated to the "oracle" best-acc and avg-ace scores which supports (empirically) why it works well. Third, our ze-pt
4

---

Under review as a conference paper at ICLR 2022
disc-acc aeg-acc best-acc
0.8
0.42
0.77
0.66
0.50
0.25
0.67
0.033
0.32
-0.2
0.00
0.0
doft-
-0.22
0:12
-0.32
0.35
-0.16
-0.2
-0.50
* x--p
dicoc
-0.4
-0.75
disc-acc
avg-acc disc-acc
darts.pt
disc-ac
zo-pe
darts
Iteration 0
(a)
(b)
Figure 2: (a) Spearman's 's rank correlation coefficient of different operation scoring metrics with each other at the first iteration of NAS. (b) Rank correlation coefficient of different operation scoring functions vs. best-acc when invoked iteratively for each edge. In iteration i, only edge is discretized then all scores for all operations on the remaining edges is computed and correlated against best-acc.
Table 1: Model selected based on maximizing each operation strength independently.
best-acc
avg-acc
disc-ace
darts-pt
ze-pt
disc-zc
darts
tenas
Avg. Error| [%]
5.63
6.24
13.55
19.43
5.81
22.96
45.7
7.19
Rank in NAS-Bench-201
166
12,744
13,770
14
14,274
15,231
1,817
Computed the average of all available seeds for the selected model in NAS-Beaet-2001 CIFAR- dataset.
with both the best-acc and avg-acc metries, indicating that there could be huge promise when using this scoring function within NAS. Note that disc-zc, like disc-ace is inversely correlated with the oracle score suggesting that perturbation is generally a better scoring paradigm than discretization. Fourth, tenas (Chen et al. 202I), which also utilizes training-free operation scoring, performs fairly well, with Spearman-p = 0..44 but still falls short of the performance of ze-pt (p -0.77. Finally, the original darts further supporting arguments in prior work that this is not an effective operation scoring method.
strongly-corred
is weakly and inversely correlated with the oracle scores,
In Table we show the discovered NAS- MAS-Bench-201 architecture when applying the seven scoring functions (Eq. [I] (5)) for operation selection on all edges. As expected, best-ace chooses the best subnetwork, while av avg-acc selects a very good model but not the best one, likely due to the large variance of accuracies in
zc-pt selected one of the top models in as expected from the strong correlation with the oracle best-ace function; while tenas selected good model, in the top 15% of the NAS-Bench- dataset, commensurate with the average correlation shown in Figure 2a The remaining operation scoring functions failed to produce a good model in this experiment, suggesting that these metrics do not make good initial choice of operations at iteration 0 of differentiable NAS. While this signals a major weakness of those differentiable NAS proxies, it's worthwhile to further analyze these methods in the progressive setting which would show what happens in later iterations of NAS. To further investigate the initial behaviour of different scoring methods, as well as their effects on NAS performance, we run similar experiments on NAS-Bench- Ishot search space, the results can be found in Appendix A.8
3.2.2 PROGRESSIVE OPERATION SCORING
Until now, we have only investigated the performance of operation scoring functions in the first iteration of NAS. This approach is relevant for methods like DARTS, where operation scoring function ff does not depend on A₁ in any way (only Ag), but is not truly representative of other methods that work iteratively. Because of that, extend our analysis to investigate what happens in later iterations of NAS. In order to do that, we calculate the correlation
scoring functions in the pregressive setting by performing the following steps: (1) score operations on all undiscretized edges, (2) discretize edge (3) retrain for 5 epochs (darts-pt and disc-ace only), (4) increment i and repeat from step I until all edges are discretized. At each iteration i, we calculate the scores for the operations on all remaining undiscretized edges and compute their Spearman rank correlation coefficients (Spearman-p) with respect to best-ace. This plotted Figure 2bb averaged over four seeds.
Our results confirm many of our initial (iteration-O) analysis. ze-pt continues to be the best operation scoring function, and darts-pt is the second-best, improving in correlation from 0.4 0.6 between the first and last iterations, indeed showing that retraining and/or progressive discretization helps. However, disc-acc continues to be unrepresentative of operation strength even when used in the iterative setting. This is not what we expected, especially the very last iteration when disc-ace is supposed to match subnetwork exactly. As Figure 2 shows, the variance in the last iteration is quite large- believe this happens because do not train convergence every time discretize
5

---

Under review as a conference paper at ICLR 2022
an edge, and instead we only train for 5 epochs. Our progressive analysis provided further empirical evidence that supernet discretization accuracy should not used as proxy for subnetwork accuracy, contradicting WWnn et al.] (2021). However, we have confirmed that darts-pt pt does in fact improve when retraining performed between NAS iterations, but could still be improved upon with zc-pt- it performed exceptionally well as proxy for accuracy on NAS-Bense 2ench-201, and has the potential to make differentiable NAS both much faster and of higher accuracy.
ZERO-COST-PT NEURAL ARCHITECTURE SEARCH
In this section, we propose a NAS algorithm called Zero-Cost-PT based on zero-cost perturbation, and perform ablation studies to find the best set of heuristics for our search methodology.
4.1 ARCHITECTURE SEARCH WITH ZERO-COST PROXIES
Our algorithm contains two stages: architecture proposal and validation. It begins with an untrained supernet Ao which contains set of edges& E, the number of proposal iterations N. and the number of validation iterations V. In each proposal iteration i, we discretize the supernet Ag based on our proposed pero-cost-based perturbation function fzc-pe that achieved promising results in the previous section. After all edges have been discretized, the final architecture is added to the set of candidates and we begin the process again for + starting with the original Ag. After N candidate architectures have been constructed, the validation stage begins. We score the candidate architectures again using selected zero-cost metric (the same which is used in pi), but this time computing their end-to-end score rather than using the perturbation paradigm. We calculate the zero-cost metric for each subnetwork using V different minibatches of data. The final architecture is the one that achieves the best total score during the validation stage. The full algorithm is outlined as Algorithm [ in Appendix A22 Our algorithm contains four main hyperparameters: N, V, ordering of edges to follow when discretizing, and the zero-cost metric to use (S). In the following we present detailed ablations to decide on the best possible configuration of these.
4.2 ABLATION STUDY ON
We conduct ablations of the proposed Zero-Cost-PT approach on NAS-
Table 2: Comparison in test error (%) with the state-of-the-ar perturbation-based and zero-cost NAS on NAS-Bench-201
Bench-201 (Dong & Yang, 2020). NAS-Bench-201 constructed a uni- fied cell-based search space. where each architecture has been trained on three different datasets, CIFAR- 10. CIFAR-100 and ImageNet-166 1 In our experiments, we take randomly initialised supernet for this search space and apply our Zero- Cost-PT algorithm to search for ar- chitectures without any training. We
(Best in red, 2nd best in blue. Same for all following tables).
Method
CIFAR-10 CIFAR-100
ImageNet-16
Zero-Cost-PT with different proxies (Section 4.2)
fisher
10.64+1.27
38.48
82.85
grad_norm
10.55
38.4322.10
80.71
grasp
9.81
36.5216.33
64.27:6.62
snip
8.32+2.02
34,00+4.03
65.35
synflow'
6.24
28.8944..0m
58.56+0.00
nwot
+0.7
27.4740.22
53.82+00.7
Baselines and SOTA approaches (Section
5.1)
run the search with four different ran- Random
13.29
39.
19.155
66.8719.66
DARTS
45.70
84.3910.00
83.68
dom seeds (0, 1, 2, 3) and report the average and standard deviation of the test errors of the obtained architec- tures. All searches are performed on CIFAR- 10, and obtained architec- tures are then additionally evaluated on the other two datasets.
DARTS-PT
11.89
45.72.6..6
69.60
PARTS-PT (fix a)
6.20
34.03
61.36.11.91
NASWOT(synflow)
6.54
29.53
58.2224.188
NASWOT(nwot))
7.04
29.97
55.57+2.07
TE-NAS
6.1040.47
28.7610.56
57.62220.46
Only I model was selected aeross all seeds.
Different Zero-com Metrics. Since
Resalts on CIFAR-IO taken from [Wang et al. 2021 Results on other datasets com- puted using official code [Wag et (021) across
sccds.
our focus is to understand how the ex-
Ussing N=1000 for both proxies and averaged over 500 as Mellor 2021
isting zero-cost metrics can be successfully applied to large-space NAS, we begin our investigation by analysis how different metrics behave when used in the proposed combination with perturba- tion-based search. In particular, we consider the following metrics that have been proposed in recent zero-cost NAS literature (Abdelfattah al. 2021 Mellor al., 202T): : grad_no __norormm____mmmmmmmmm (Abdelfattal ct al. 202I), snip (Lee et al.) 2019). grasp (Wang et al. 2020), syn synflow (Tanaka ct al. (022)) fisher (Thiss et 2018)) 8) and nwot (Mellor et al.] 2021). Table 2 compares the average test errors of architectures selected by different proxies on
We see that nwot and synf fow perform considerably better across the three datasets than the others, where nwot offers
We use the three random seeds available NAS. A.S-Bench-201: 777, 888, 999.

---

Under review as a conference paper at ICLR 2022
Table 3: Test error (%) of ZerooCost-PP when using different search orders NAS-Bench-201.
Search Order'
# of Perturbations
CIFAR-10
CIFAR-100
ImageNet-16
fixed
O|| [ +
5.98
27.60=1.66
54.23 03
global-op-iter
5.69
26.80 -0.61
53.64
global-op-Once
2|
10
6.30
28.96 1.68
55.04
global-cdge-itez
+1
6.24
28. 28.42=0.cm
54.39
global-edge
2
O0 6.30
28. 28.96 1.66
55.04
random
E
5.97
27.47
53.82
methods metrie, N=10 architecture proposal Iterations V=100 validation iteration.
Number of perturbations per search
global-op-tte
random
random
N-10
N-100
N-1000
(N-10)
(N-1000)
random (N-10)
B4.
93.0
Architecture Score
V-1 V-10
V-100
V+1
V-10
V-100
V-1 V-10 V-100
Figure 3: (a) Accuracy vs. score of architectures discovered on CIFAR- 10 by Zero-Cost-Pa with different N. (b) Accuracy distribution of discovered architectures with different N and V.
around 0.27% improvement over synflow. However, even the worst performing fisher and naive grad _rrd_ norm outperform the state-of-the-art DARTS-PT on this benchmark (see Table This confirms that the zero-cost metrics, when combined with the perturbation-based NAS paradigms as in Kero-Cost-PP could become promising proxies to the actual trained accuracy. We also observed that the ranking of those metrics are quite stable on the three datasets (descending order terms of error as in Table 2) indicating that architectures discovered by our Zero-Cost-PP have good transferability datasets. nwot consistently performs best, reducing test errors all three datasets by considerable margin.
Edge Discretization Order. We also study how different edge discretization order may impact the performance of our Zero-Cost-PT approach. We consider the following edge discretization orders:
f i xed: discretizes the edges in fixed order, where in our experiments we discretize from the input towards the output of the cell structure;
random: discretizes the edges in a random order;
iteratively evaluates S(A ()) for all operations all edges in E, selects the edge e containing the operation o" with globally best score. Discretizes with o", then repeats to decide on the next edge (re-evaluating scores) until all edges have been discretized; only evaluates S(A o)) for all operations once obtain ranking order of the operations and decide the edge order upfront based on it, then starts following the algorithm usual, calculating scores of operations at each edge iteratively;
global-op-iter
global-op-oncee
g global -edge-i ter: similar to global -op-otttteee ieer but iteratively selects edge e from based on the average score of all operations on each edge;
global-edge-one once: similar to aloballlp--me but uses the average score of operations on edges to obtain the edge discretization order.
In this experiments for all variants, using naswot metric. Table. 33 shows the performance the approaches. We see that the global-ooooo leooaloop-ii teropppitee iierrierrr consistently performs best across all three datasets, since it iteratively explores the search space of remaining operations, while greedily selecting the current best. However, it comes with a higher cost than iixed or random, since we need to perform + [|]|| (|&)
N=10 architecture proposal iterations and V=100 validation iterations
perturbations in total, while the latter require |O|[E].O |On the other hand, we see that the performance -p-once once inferior since it determines the order of perturbation by assessing the importance of operations once for all at the beginning, which may not be appropriate as discretization continues. Note that when discretizing still need perturb the [O) operations on each remaining edge. We observe similar behaviour in global- werl--edge-a it et and operations edges to decide search order, leading to suboptimal performance. It is also worth
of global- -opp-looppooo
edge according to the obtained order, gl oba 1-op-
-attall-dge--neee both of which use the average importance of
7

---

Under review as a conference paper at ICLR 2022
pointing out that fixed performs relatively well comparing to the other variants, offering com- parable performance with random. This shows that Zero-Cost-PT is generally robust to the edge discretization order. For simplicity, in the following experiments we use random order with a moderate setting in architecture proposal iterations (N=10) to balance exploration and exploitation during search, while maintaining the efficiency of Zero-Cost-PT.
Proposal vs. Validation, We study the impact of different architecture proposal iterations N and validation iterations V when Zero-Cost-PT uses random as the search order and nwot metric. Intuitively, larger leads to more architecture candidates being found, while V indicates the amount of data used to rank the search candidates. As shown in Fig- ure Bal we larger N does lead to more architectures discovered, but not proportional to the value of N on NAS-Bench-201 space. For N=100 we discover 27.8 distinet architec- tures on average, but when increased to N=1000 the number only roughly doubles. We also see that even with N=10, Zero- Prro-Cost-PT,, random can already discover top models in the space, demonstrating desirable balance between search quality and efficiency. On the other hand, as shown Figure [3b] larger V tends to reduce the performance variance, especially for smaller N. This is also expected as more validation iterations could stabilise the ranking of selected architecture candidates, helping Zero-Cost-PT to retain the most promising ones with a manageable overhead of V minibatches. We provide additional ablations regarding the effects of N and V conducted on the DARTS CNN space in Appendix A.7 We also revisit our comparison between zc-pt and c di sc from Section 3.2 by comparing our Zero-Cost-PT when run with either scoring method on NAS-Bench-201 in Appendix A.6.3
Table 4: Comparison with the state-of-the-art differentiable
NAS methods on the DARTS CNN search space (CIFAR-10)
Test Error (%)
Method
Params (M)
Cost
Avg.
Best
DARTS SDARTS-RS SGAS
3.00
3.3
0,4
2.67
3.4
0.4
2.66:0.24
3.7
0.25
DARTS-PT
2.61 10.08
2.48
3.0
0.8
DARTS-PT +none
2.73
2.67
3.2
0.8
TE-NAS
2.634
3.8
0.05
Bero-Cost-Be randon
2.64
2.43
4.7
0.018
Zero- Pero-Cost-PT. globallop--ite
2.62
2.49
4.6
0.17
Results obtained by re-cuabling none operation DARTS-PT (Wang
2021)
In GPU days. Cost of existing appeoaches taken from (Wang al. 202 Zero Cost-PT measured on a single 2080T: GPU.
Cost of
5 RESULTS
In this section we perform extensive empirical comparisons of Zeno-Cost-PT with the state-of-the-art differentiable and zero-cost NAS algorithms number of search spaces, including NAS-Bencke 201 (Dong& Yang 2020). DARTS' CNN space (Liu et al., 2019) and the four DARTS subspaces
S1-S4 (Zela et al. 2020a). Detailed experimental settings are in Appendix A.5
5.1 COMPARISON WITH SOTA ON
Table 2 shows the average test er- ror (%) of the competing approaches and our Zero-Cost-PT on the three Here
Table 5: Comparison with the state-of-the-art differentiable
NAS methods on the DARTS CNN search space (ImageNet).
datasets in NAS-Bench-201 we include the naive random search and original DARTS as baselines, and compare our approach with the recent zero-cost NAS algorithm NASWOT (Mellor et al. (Q22))) TE- NAS (Chen et al. 2021), as well as the perturbation- based NAS ap- proaches DARTS-PP PT and DARTS- PT (fix a) (Wang et al. 2021). As in all competing approaches, we per- Zero-Cost-PTT Cost-PT seeds) 24.640.19
Error [%]
Params
Cost
Method
Top-1
Top-5
[M]
[GPU-days]
DARTS
26.7
8.7
4.7
0.4
SDARTS-RS
25.6
8.2
0.4
DARTS- PT
25.5
8.0
4.6
0.8
PC-DARTS
25.1
7.8
5.3
0.1
SGAS
24.1
7.3
5,4
0.25
TE-NAS
26.2
8.3
6.3
0.05
Zero-Cost- (ose--PT(bbst))
24.4
7.5
6.3
0.018
7.6
6.3
0.018
form search on CIFAR- 10 and evalu- ate the final model on all three datasets. consistently offers superior performance, especially on CIFAR- and ImageNet-16 16. On the other hand, the best existing perturbation-basee producing suboptimal results with small improvements compared random search, suggesting that architectures discovered by DARTS-PT might not transfer well to other datasets. TE-NAS is second best on CIFAR but we show in $ 5.2 performance deteriorates larger datasets like ImageNet.
use the same training pipeline from DARTS Lin et al. 2019
We see that all datasets our Zero-Cost-PTT (with nwot)
algorithm, DARTS-PT (fix a),, fails on those two datasets,
8

---

Under review as a conference paper at ICLR 2022
5.2 DARTS CNN SEARCH SPACE
We now move to the much larger DARTS CNN search space. We use the same settings as in DARTS-PT (Wang et al.; 202T). but instead of pre-training the supernet and fine-tuning after each perturbation, we take an untrained supernet and directly perform Zero-Coste Ccro-Cost-PT algorithm as in Section 4.1 As in previous experiments, we run with N=10 architecture proposal iterations and V=100 validation iteration, using the same random seeds as in DARTS-PI.. We then train the selected four architectures under different initializations (seeds 0-3) for 600 epochs, and report both the best and average test errors on both CIFAR- 10 and ImageNet. Experimental details, discovered architectures and additional basclines can be found the in Appendix A.3 A0 and A.6
Results on CIFAR-10. 10. As shown in Table 4 the proposed Zero-Cost-PP approaches can achieve much better average test error then the original DARTS and comparable to its newer variants SDARTS- RS (Chen & Hsieh (020) and SGAS (Li et al. (2200 at much lower searching cost (especially when using random edge ordering). There a significant search cost reduction compared DARTS-PT. While DARTS-PT needs to perform retraining between iterations, Zero-Cost-PT only evaluates the score of the perturbed supernet with zero-cost proxies (Sawet). requiring more than minibatch of data. Note that here the cost of Zero-Cost-PP reported in Table is for N=10 architecture proposal iterations ( (ranom edge discretization order), and thus a single proposal iteration only takes about a few minutes to run. The other variant Zero- Cero-Cost-PT. Cost-PTggobbll-- globel-oppitte offers better performance with lower variance compared to random but incurs slightly heavier computation.
Results on ImageNet. 3 the ImageNet classification accuracy for architectures searched randem
on -I. Our Zero-Cost-PTT random algorithm is able to find architectures with comparable accuracy much faster than previous work, further reinforcing its efficacy in this setting. While TE-NAS results on CIFAR-10 10 were very close to 7err-CCost-PT. a much larger difference is observed on ImageNet with an accuracy drop of 1.8 pp and a runtime that ~2.5 x slower than Zero-Cost-PT.
5.3 ROBUSTNESS ANALYSIS It is well known that DARTS could generate trivial architectures with degenerative perfor- mance in certain cases. Zela et al. (2020a) have designed various special search spaces for DARTS to investigate its failure cases on them. As in DARTS-PT, we consider spaces $1-$4 to validate the robustness of Zero-Cost-PTT in a controlled environment (detailed specifications can be found in Ap- pendix A.4.. As shown in Table 6 our ap- proach consistently outperforms the original DARTS, the state-of-the-ar DARTS-PT and DARTS-PT(fxx a:) across $1 to S3 on both datasets CIFAR-10 and CIFAR- -000, while on SVHN it offers competitive performance comparing the competing algorithms (best in S1, second best in space S2/S3 with 08%.02% gap). This confirms that our Zero-Cost-PT is robust in finding good performing archi- tectures in spaces where DARTS typically fails, e.g. has been shown (Wang al.)
Table 6: Comparison in test error (%) with state-of-thy art perturbation-base NAS on DARTS spaces S1-S4.
DARTS
DARTS-PT
Zero-Cost-PIT
Space
Best
Best
Best (fix a)
Avg.
Best
CIFAR-10
S1
3.84
3.5
2.86
2.75
2.55
S2
4.85
2.79
2.59
2.49:20.0
2.45
S3
3.34
2.49
2.52
2.47 2047=20.09
2.40
S4
7.20
2.64
2.58
5.23
4.69
CIFAR-100
S1
29.64
24.48
24.4
22.05 ...29
21.84
S2
26.05
23.16
23.3
20.97
20.61
S3
28.9
22.03
21.94
21.02=
20.61
22.85
20.80
20.66
25.70=a.01
25.69
SVHN
$1
4.58
2.62
2.39
2.37 10.06
2.33
S2
3.53
2.53
2.32
2.40
2.36
3.41
2.42
2.32
2.34
2.30
S4
3.05
2.42
2.39
2.83
2.79
Results taken from Wang et al.
2021].
Results obtained using random seeds D and 2.
2021) that in S2 DARTS tends to produce trivial architectures saturated with skip connections. On the other hand, we observe that Paro-Cost-Poe
doesn't perform well in search space S4, struggling with operation noise, which simply outputs a random Gaussian noise (0, 1) regardless of the input. This is expected as score .() can be completely random if o = noise. However, since noise operation not useful in NAS, we are satisfied with the robustness of Tero-Cost-PT on S1-S3.
CONCLUSION
In this paper, we formalized the implicit operation scoring proxies that are present within differentiable NAS algorithms to both analyze existing methods and propose ones. We showed that lightweight operation scoring methods based on zero-cost proxies empirically outperform existing operation PORTS-PTTTR scoring functions such DARTS Liu et al. (2019) DARTS PT Wang et al. (202I) and TE-NAS (Chen) lt al. 2021). We also found that perturbation is more effective than discretization when scoring an operation, leading to our lightweight NAS algorithm, Zero-Cost-PT. Our approach outperforms the best available differentiable architecture search in terms of searching time and accuracy even in very large search spaces- something that was previously impossible with zero-cost proxies.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
In our paper, we adhere to the NAS best practice checklist (see below) (Lindauer & Hutter 2020) to ensure reproducibility. Our code (both NAS and training pipeline) is available in the supplementary material. We report all the details of our experimental setup, and describe all hyperparameters for both NAS and the final evaluation pipeline, as well as random seeds used in our main paper, appendix and code. We perform multiple runs of our experiments with the reported random seeds and provide average and standard deviation of the results. We report the wall-clock time for the search cost of our approach, and the training time for the discovered models are discussed in appendix. We also provide the detailed information on confounding factors, such as GPU hardware, versions of DL libraries, different runtimes of our experiments, in the appendix and the code (README in supplementary material).
When applicable we also compare to random search and other naive baselines (e.g. max-param models as detailed in appendix), and also use tabular benchmarks (NasBench-201 for in-depth evaluations and ablation study of our approach. For all existing NAS methods compared in this paper, we use exactly the same NAS benchmark as described in the literature and associated public repositories, including the same dataset (with the same training-tes test split), search space and code for training the architectures, and hyperparameters for that code. We use the same evaluation protocol for the methods being compared, as provided in their code repositories.
7.1 NAS BEST PRACTICE CHECKLIST
1. Best Practices for Releasing Code
For all experiments you report:
(a) Did you release code for the training pipeline used to evaluate the final architectures?
[Yes]
(b) Did you release code for the search space [Yes]
(c) Did you release the hyperparameters used for the final evaluation pipeline, as well as random seeds? [Yes]
(d) Did you release code for your NAS method? [Yes]
(e) Did you release hyperparameters for your NAS method, well as random seeds? [Yes]
2. Best practices for comparing NAS methods
(a) For all NAS methods you compare, did you use exactly the same NAS benchmark, including the same dataset (with the same training-test split), search space and code for training the architectures and hyperparameters for that code? [Yes]
(b) Did you control for confounding factors (different hardware, versions of DL libraries, different runtimes for the different methods)? [Yes]
(c) Did you run ablation studies? [Yes]
(d) Did you use the same evaluation protocol for the methods being compared? [Yes]
(e) Did you compare performance over time? [N/A]
(f) Did you compare to random search? [Yes]
(g) Did you perform multiple runs of your experiments and report seeds? [Yes]
(h) Did you use tabular or surrogate benchmarks for in-depth evaluations? [Yes]
3. Best practices for reporting important details
(a) Did you report how you tuned hyperparameters, and what time and resources this required? [Yes]
(b) Did you report the time for the entire end-to-end NAS method (rather than, e.g., only for the search phase)? [N/A]
(c) Did you report all the details of your experimental setup? [Yes]
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Mohamed S Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas Donald Lane. Zero-cost proxies for lightweight NAS. In International Conference on Learning Representations (ICLR), 2021.
ew net. leeeet/ffrrmmmma==cmmmmaama
Richard Bellman. Dynamic Programming. Dover Publications, 1957. ISBN 9780486428093.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations (ICLR), 2019. URL
hnttp:://arxiv: org/ 1121.11110 00332. pdf
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network
and specialize it for efficient deployment. International Conference on Learning Representations (ICLR), 2020. URL https: http//axiivii. Bttpggppe//1990 09791.pdE
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: theoretically inspired perspective. In International Conference on Learning Representations, 2021.
Xiangning Chen and Cho-Jui Hsich. Stabilizing differentiable architecture search via perturbation- based regularization. In International Conference on Machine Learning (ICMI), pp. 1564-1565. PMLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: Large-Scall Hierarchical Image Database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattemn Recognition (CVPR), 2009.
Xuanyi Dong and Yi Yang. NAS-Bench-201: Extending the Scope of Reproducible Neural Architec- ture Search. International Conference on Learning Representations (ICLR), 2020.
Fukasz Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D. Lane. BRP-NAS: Prediction- NAS using GCNs. In Neural Information Processing Systems (NeurlPS), 2020.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv preprint 170.:007744 2017
Forrest N. landola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <0.5mb model size. arXiv: 1602.07360 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks (invited paper). In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2021, pp. 6, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978 1450380539, doi: 10.114573406325.3465333
URL URL http:/httt://tt //tp:p::://d...o. .org/oogggooo 10. 1..11 145114/344 0455534033555 1...5.465555 3465355
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations (ICLR),
2019.
Guohao Li, Guocheng Qian, Itzel C Delgadillo, Matthias Muller, Ali Thabet, and Bernard Ghanem. Sgas: Sequential greedy architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1620-1630, 2020.
Marius Lindauer and Frank Hutter. Best practices for scientific research on neural architecture search. Journal of Machine Learning Research, 21 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations (ICLR), 2019.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 7827-7838, Red Hook, NY, USA, 2018. Curran Associates Inc.
11

---

Under review as a conference paper at ICLR 2022
Abhinav Mehrotra, Alberto Gil Ramos, Sourav Bhattacharya, Lukasz Dudziak, Ravichander Vipperla, Thomas Chau, Mohamed S. Abdelfattah, Samin Ishtiaq, and Nicholas D. Lane. NAS-Bench-ASR: Reproducible Neural Architecture Search for Speech Recognition. In International Conference on Learning Representations (ICLR), 2021.
Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J. Crowley. Neural architecture search without training. In International Conference on Machine Learning (ICML), 2021.
Bert Moons, Parham Noorzad, Andrii Skliar, Giovanni Mariani, Dushyant Mehta, Chris Lott, and Tijmen Blankevoort. Distilling optimal neural networks: Rapid search in diverse spaces. vrXiv:20122.08599 2020.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning (ICML), pp. 4095-4104 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized Evolution for Image Classifier Architecture Search. In AÁAI Conference on Artificial Intelligence (AAAI), 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVE Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 4510 4520, 2018. doi:
10.1109/CVRR.2018.000 204.00444.
Yao Shu, Wei Wang, and Shaofeng Cai. Understanding architectures learnt by cell-based neural architecture search. In International Conference on Learning Representations, 2020. URL
htttp::/cppereviiee
net/
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning (ICML), 2019.
Mingxing Tan and Quoc V. Le. Efficientnetv2: Smaller models and faster training. arXiv:1104.00298. 2021.
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Neural Information Processing Systems (NeurIPS), 2020.
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferene Huszár. Faster gaze prediction with dense networks and fisher pruning. arXiv:1801.05799 2018.
Jack Turner, Elliot J. Crowley, Michael O' O'Byyle, Amos Storkey, and Gavin Gray. Blockswap: Fisher-guided block substitution for network compression budget. In International Conference on Learning Representations (ICLR), 2020.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by
preserving gradient flow. In International Conference on Learning Representations (ICIR), 2020.
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architecture selection in differentiable NAS. In International Conference on Learning Representa- tions (ICLR), 2021.
d= KewaeeKuuuaaaaaa net wwt. forrtt/fo/ummmm// Peerrmmmdd=PPuPe
Chen Wei, Chuang Niu, Yiping Tang, and Ji min Liang. Npenas: Neural predictor guided evolution for neural architecture search. arXiv:2003. 2020.
Wei Wen, Hanxiao Liu, Hai Li, Yiran Chen, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. arXiv:1912.0000 00848. 2019.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-awwe aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10726 10734, 2019. doi: 10. 10.1109/CVPR.201 9.01099.
Junru Wu, Xiyang Dai, Dongdong Chen, Yinpeng Chen, Mengchen Liu, Ye Yu, Zhangyang Wang. Zicheng Liu, Mei Chen, and Lu Yuan. Weak NAS predictors are all you need, arXiv:2102.10490, 2021.
12

---

Under review as a conference paper at ICLR 2022
Kaicheng Yu, Christian Sciuto, Martin Jaggi. and Mathieu Salzmann Claudiu Musat. Evaluat- ing the search phase of neural architecture search. In International Conference on Learning Representations, 2020. /
net/ Lo
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. In International Conference on Learning Representations (ICLR), volume 3, pp. 7, 2020a.
Arber Zela, Julien Siems, and Frank Hutter. Nas-bench- bes-bennc--sshol:: Benchmarking and dissecting one-shot neural architecture search. arXiv preprint arXiv:2001. 10422, 2020b.
Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. Econas: Finding proxies for economical neural architecture search. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In International Conference Learning Representations (ICLR), 2017.
13

---

Under review as a conference paper at ICLR 2022
A
Ao
A1 Ao + (e .022)
A
Figure A1: Visualisation of the searching tree related to a hypothetical supernetwork with 3 nodes (represented by different colors) and 2 candidate operations (represented by different numbers). The root node of the tree represents a full supernetwork. Each descendant node represents a non-unique network architecture obtained by discretizing an edge identified by the color of the node, with an operations encoded by number in the node, given an architecture represented by a parent node. A is the set of all possible architectures, including the supernet Ag. networks. The set of all fully-discretized networks is additionally denoted as Alg). and the set of all fully-discretized networks that are achievable after operation o(2) is assigned
networks like A
and fully-discredd
to edge e(t), given Ag. is labeled as A0,e(1).o(2)) The arrow from Ao to Ai represents possible first step during the discretization process which closes all grayed-out configurations. While performing NAS, at each tree level we need to decide which edge to follow; this is achieved by assigning scores to all/some of the edges that we are willing to consider in each iteration using different scoring functions which are the main focus of our analysis in Section 3
A APPENDIX
A.1 EXTRA DETSILS ABOUT NOTATION USED IN THE PAPER
All sets are denoted with stylised capital letters using latex's 's matcal font. Letters denoting elements of different sets use the same letters as the sets, e.g. e E E, A E A, etc. For any element, we use subscript for indexing iterations of the discretization process - e.g., Ao is a network architecture the beginning of iteration 0, e2 is an edge that is being investigated in iteration 2, etc. To identify different elements in any other context we use superseript, e.g. e(!) might denote the first edge in a supernet, which might be different from é which denotes the edge that is first going to be dicretized following relevant discretization order.
To better understand meaning of each used symbol, consider a hypothetical supernet with 3 edges -e(1), e(3) - repented by different colors (green, blue and red, respectively) and 2 candidate operations o(1), o(2) represented by different numbers (I and 2). Figure Al visualises the entire space related to the decision process that is happening in order to perform NAS in this setting, including the first discretization step A₁ = Au + (Co, og), where eu = e(1), Og = o(2), and the related set of all achievable fully- -illy-discretized models Ao. concept central to our definition of the optimal scoring function.
A.2 DETAILED TERO-COST-RR ALGORITHM
Algorithm| I presents the proposed Zero-Cost-PT algorithm introduced in Section 4.1 It has two stages: searching and validation, where we first iteratively discretize the supernet Ag based on gcro-cost-based perturbation function fzo-pt (line 1- 16), and then in the second stage we use the zero-cost metric (the same which is used in fzc.p) score the candidate architectures (line 17 20), and select the one with the highest end-to-end score. In particular for DARTS CNN search space, our Zero-Cost-PT algorithm has an additional topology selection step (line 8 14), where for each node in architecture we only retain the top two incoming edges based on the zc-pt score this similar to the vanilla DARTS algorithm (Liu et al. 2019). For TAAS-Bench-225 space our algorithm skips this topology selection step.
14

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Zero-Cost certurbttionbbase Architecture Search (Zero-Cost-P))
In untraimed supernetwork As with of edges & and of nodes N, of architecture proposal iterations of validation itcrations
Result: selected architecture Alg
// Stage 1:
propose architesture candidates
C=
for :::N:..
for t |
Select next edge e, using the chosen discretization ordering
= (Ar-((-
A. A-1 (e))
end
while
IV do // prune the edges of the obtained architecture A Randomly select node A forall Input edge to node do Evaluate the ze-pt score of the architocture Aie when removed
12
end
Retain only edges en" (2)+ with the 1st and 2nd best ze-pt score, and remove from V end
13
14
15
Add Aigl to the set candidate architectures
16 end
Stage 2: val i date the arghi tecture candi dates
17 for j = 1: do
Calculate 5th) (A) for each A E C using a random mini-batch data;
18
19 end
20 Select the hest architecture Alel arg max ACC Ej=1y 5th (A);
A.3
MORE ON OPERATION SCORING
This section provides more experimental details and examples of our analysis of the operation scoring functions introduced in Section
A.3.1 DETAILED SCORING METHODOLOGY
analysis the on the initial operation scoring aims to investigate how well an operation scoring method can perform when making the first discrietization decision, with respect to the perfect search (the best-acc approach). Here "the first discrietization decision" made the first iteration (iteration 0) of progressive operation selection algorithm, and in our experiment we compute the score for all operations on an edge and later average across all edges to account for random selection of the first edge. Concretely, all edges, then compute the Spearman rank correlation for operations on each edge. After that, we average the rank correlation coefficient over all edges.
As discussed in Section 3.2.1
compute the score per operation across
Consider the example shown in Figure A2 Suppose we have supernet with just two edges in Figure A2 In this case, an operation scoring function should pick just one operation per edge. For given operation scoring function, we compute the scores for each operation on each edge. Then for each edge, we compute the rank correlation of the oracle scores (best-acc or avg-acc acc) against the scores from the other operation scoring function (e.g. ZC-pt or darts-pt). We then average their correlation coefficient across all edges in supernet to get average correlation for each operation scoring function. The resulting average rank correlation indicative of how well a given operation scoring function (starting with a random edge) would do when making the first discretization decision, relative to the oracle search.
A.3.2 EXPERIMENTAL DETAILS
Here, provide some additional experimental details for the data presented in Section 3 The following list describes how we compute each operation score.
best-ace: To get the score for an operation o on specific edge e, we find the maximum test accuracy of all Bennhhen architectures with (o,e).
15

---

Under review as a conference paper at ICLR 2022
opt
052
cp3
commatiio
opt
op2
op3
Edge
2e:pe
0.5
0.9
3a
1.00
Avarage corrlatiion Over edges
ong. comelation
clarts
0.75
op
052
9p3
cormilation
opt
op2
op3
Edge
best-aco
darts-g gt
1.00
node
Figure A2:
An example showing how correlations of different proxies are computed in our analysis on initial operation scoring in Section 3.2.1
avg-acc: Same as best-acc but we average all instead of finding the maximum.
architecture test accuracies
disc-ace: We discretize one edge by selecting an operation then we train for 5 epochs and record the supernet accuracy this is used the for (o,e).
darts-pt: We perturb one edge with one operation A (e,,, o) and record the validation accuracy. For perturbation-basel scoring functions, multiply the score by -1 before computing correlations.
dise-ze:
We discretize one edge by selecting an operation and then compute the zero-cost
metric.
ze-pt: We perturb edge with operation A- (e, and compute the zero-cost metric. For perturbation-base scoring functions, we multiply the score by -1 before computing correlations.
We record the value of the architecture parameters a after 60 epochs of training the supernet.
darts:
We perturb one edge with one operation A (e, o) and compute the KN and the Ry (number of linear regions). Then we rank K ascendingly and descendingly rank the Ry. At last, we add those two ranks together to get the final ranks of the operations. We multiply the score by- -1 before computing correlations.
tenas:
A.3.3 DETAILED OPERATION SCORES
Table A8 (at the end of this Appendix) shows all operation scores at iteration O. This data was used to compute Spearman-p in Figure 2a Note that compute Spearman-p per edge and average all edges this summarizes how well each score tracks our "oracle" best-ace score.
A.4 DESCRIPTION DARTS SUBSPACES (S1-S4)
RobustDARTS introduced four different DARTS subspaces to evaluate the robustness of the original DARTS algorithm (Zela al. 2020a). In our work, validate the robustness of Zero-Cost-PT against some of the more recent algorithms using the same subspaces originally proposed in the RobustDARTS paper (Section 5.3). The search spaces are defined follows:
DARTS PT detines discretization accuracy the accuracy after convergence. We elected only train for epochs make experiments feasible but results.
investigating whether longer training will affect
16

---

Under review as a conference paper at ICLR 2022
in S1 each edge of a supernet consists only of the two candidate operations having the highest magnitude of in the vanilla DARTS (these operations can different for different edges);
S2 only considers two operations: skip_connect and sep_conv_33__
similarly, S3 consists of three operations: none, skip_connect and sep_conv_3x3
finally, S4 again considers just two operation: noise and __p__coo__33____________________ __v3,,3333_ where nois operation generates random Gaussian noise (0, 1) in every forwards pass that is independent from the input.
A.5 EXPERIMENTAL DETAILS
All searches were run multiple times with different searching seeds (usually O, 1, 2 and 3). Addition- ally, each found architecture
trained multiple times using different training seeds for DARTS the same set of seeds was used for training and searching, for NAS-Bench-201 (NB201) training seeds were taken from the dataset (777, 888 and 999, based on their availability in the dataset). Therefore, for each experiment we got total of searching_seeeeeeeeeeessssssssse seeds tra traininggssett _eeds accuracy values. Whenever average performance is reported, it is averaged across all obtained results. Similarly, best values were selected by taking the best single result from the searching and training seeds.
A.5.1 EXPERIMENTAL DETAILS- - NAS-BENCH- BENCH-201
Searching was performed using different seeds (0, 2, and 3) to initialise a supernet. Whenever had to perform training of supernet during the searching phase (Section 3), used the same hyperparameters as the original DARTS-PT code used. When searching using our Zero-Cost-PTl we used batch size of 256, N=10, V=100 and S=nwot, unless mentioned otherwise (e.g.. during ablation studies). Inputs for calculating zero-cost scores came from the training dataloader(s), as defined for relevant datasets in the original DARTS-PT code (including augmentation). For zero-cost proxies that require loss function, standard cross-entropy an algorithm had identified final subnetwork, we extracted the final architecture and queried the NB201 dataset to obtain test accuracy one value for each training seed available in the dataset.
used. For any searching method, after
All experiments concerning operation scoring (Sections and A.3 used averaged accuracy of models from NB201 for simplicity.
We did not search for architectures targeting CIFAR-100 ImageNet-16 16 directly whenever we report results for these datasets, we used the same architecture found using CIFAR-10.
A.5.2 EXPERIMENTAL DETAILS - DARTS
DARTS experiments follow a similar methodology to NB201. Each algorithm was run with 4 different initialisation seeds for a supernet (0, 2 and 3). When running Zero-Cost-PT, used the following hyperparameters: batch size of 64, N=10, V=100 and S=nwot. Inputs and loss function for zero-cost metrics were defined analogically to NB201. We did not run any baseline method search space (all results were taken from the literature), so we did not have to perform any train a supernet. After an algorithm had identified a final subnetwork, we then trained it 4 times using different initialisation seeds again (0, 1, 2 and 3). When training subnetworks, used a setting aligned with the previous work (Liu et al. 2019 Wang et al. 2021).
the DARTS
Unlike NB201, whenever different datasets were considered, (Section 5.3) architectures were searched on each relevant dataset directly.
For CIFAR- 10 experiments, we trained models using heavy configuration with init_cchnne]] ls 36 and layers 20. Models found on CIFAR-100 and SVHN were trained using mobile setting with init_chaaree
16 and layers = 8. Both choices follow the previous work (Zela et al.
2020a
Wang et al. 2021
A.6
MORE BASELINES
A.
MAXIMUM BASELINE
As shown in Table and Table 5 in Section 5.2, our approach tends to select architectures with comparable or slightly more parameters than the sfatc-of-the-art, However, this does not mean any larger models would lead to superior performance, i.e. simply maximizing FLOPs/Params is not an
17

---

Under review as a conference paper at ICLR 2022
appropriate searching methodology in general. Below are the results of training 4 random models with separable convolution 5x5 (the most expensive operation in the DARTS search space) selected everywhere and random connections between layers. The evaluation methodology follows all other experiments, i.e. we have searched models trained 4 times with different training seeds, and report the average and minimum error.
Overall, the test error (%) of this method is 2.93 (min), vs. ours 2.64-0.16 (avg.) and 2.43 (min). This confirms that sim- ply selecting models with maximu FLOPs/Params is not an appropriate searching methodology as evidenced by the above results the DARTS search spaces. We also show that the results translate to ImageNet in later sections. On the other hand, on simple search spaces like NAS-Bench-201 (NB201), this maximum-ppran baseline may per- form relatively better, e.g. can find the model with test accuracy of 33.76% (165th position in the ranking), but our method can still do better, discovering the 33rd best model in the search space (test accuracy 94.03%).
10.23 (avg.) and 2.78
Table Al: Randomly selected architectures with only opera- tion sep_conv_ 5x5 on DARTS CNN space.
Test Error(
seed
general,
Training seed
Avg.
Best
0
1
2
3
0
3.07
2.93
2.89
2.85
2.94
2.85
1
2.92
2.93
3.17
3.05
3.02_0.20
2.92
2
2.98
2.97
2.93
2.90
2.95
2.95
3 2.87
2.83
3.02
2.78
2.88 u.11
2.78
Random for searching architectures.
Ramdom for training the architectures.
In summary, on both NB201 and DARTS space, the proposed combination of the perturbation paradigm with zero-shot proxies does better than the naive usage of the proxies presented (Ab delfattah et al., [021] Mellor et al. (2021). as shown in Tables 2 4 and A2 Note that we are better than the plain perturbation-basee 2021) on NB201 (Table 2 and comparable/better on DARTS and derived subspaces (Table and
baseline (Wang et al., [202]) and recent zero-cost NAS (Chen et al.
while being much cheaper to run.
The structures of models are provided as Genotype objects below for reproducibility.
random max Genotype ("rsoraaal(sspp..o__e "sse__cooy 5x5", 0], ["sep_conv conv _as____oo___________________ 11, dise__cov___e _x_5", 0), aree_coov_stt "_____x",,_ 11, ''se__cony" _tp___ooo__'''_______ 0], ["
4]], normal_concat-range (2, 6), zeduce= ("riiceelssspp_e___e sissep_covv_5x333 01, ["
rep_conv_5x5" 3], "sse__cov__5xx
1), [*scp_cony_5x5
sep_conv_5x5", 11, ["sep_conv_5x5", 0], ["sep_conv_xx5", 11, [" sep_conv_5x5", 0], ["sep_conv_5x5", 11, ["sep_conv_xx5", 0], [" www._ccocnttttcce2..22 6))
sep _sp_conv__x5", 111, reduce__conatttee
random_max_1 [eenotrooeenmmallaeeeeeeoooooo_
01, ["sep_conv_5p 5x5*, 11, ["scp_conv_5x5", 0), ""scp_conv_5x5", 2], ("scp_conv_5x5", 0), ["
sep_conv_5x5", 2], ["sep_conv_5x5" 1], "=se__conv_5x5", 4]], normal_concat-<rrgee
6), reducee
0], [" 01, conv__cov 5x5", 11, |"
sep_coav_bze 5x5", 11, sep_conv. 5x5", 01, [ssep copy_oov___s_ 11, "sep conv_oovvv_ooo 0], ["
sep_conv_5x5", 1]1, roduce__concat-rangee (2, 6))
random_
(normall
__sppon___________ 5x5", 0], ["sep_cone conv__xx",, 11,
("sep_conv_5x5", 0], ""se__conv_5x5", 2], "sse__covv55x5, 0], ["
oep_ccav_5x5", 3], ["sep_conv_5x5", 11, ["scp_cony_xx5] 4]], nozmal_concat=rarn
6), reduce=l "sep__oo________ 01, 1"
sep_conv_5x5", 1], ["sep_conv_5x5", 0], ["sep_conv_xx3" 11, ["
sep_conv_5x5", 0], ["sep_conv_5x5", 11, i"sep_conv_5x5", 01, ["
sep_ sep_conv_5x5" 1]], reduce_ redice__conctt=rangg111 (2, 6))
vandom_max__ Ge (Moraaa 5x5* 0], ["sep_conv 5x5", 11.
["scp_conv_5x5" 0), ( ["scp_conv_5x5"" 11, ("scp_conv_5t
0), ["
sep_conv_5x5" 2], ["sep_conts ["sep_conv_5x5", 1], "'sep_conv_5xx" 4]],
01, [" conv__xx", 11, "sep_ conv 5x5", 01, ["sep_com conv_5x3", 11, 1" sep_ sep_conv_st sep_conv_5x5", 0], ["sep_com cocvcovvss'.. 11, sapp_conv_xx'"" 01, [" reduce_concat--range (2, 6)
normal_concat--rreee (2, 6), (edice-((see_ccov__sk
sep_conv_5x5", 1]],
A.6.2 RANDOM-SAMPLING BASELINE
18

---

Under review as a conference paper at ICLR 2022
we compared our method to sampling-based zero-cost NAS in Table NAS- WOT lines). Our results are empirically better on all three datasets. Additionally, our method com- putes the operation score per edge in a supernet whereas the sampling- based approach computes the
In Section 5.1
Table A2: Comparison with randomly sampled
networks in DARTS CNN space (CIFAR- 10).
Test Error (%)
Sample Size
Cost (GPU-Days)
Avg.
Best
end-to-end network score. The relationship between the number of subnetworks and the number of op- erations is exponential. Therefore, we anticipate having to sample exponentially many networks in sample-based NASWOT (Mellor et al. 2021) com- pared to our proposed Kero-Cost-P1
2500
2.
2.66
0.018
20000
2.73
2.58
0.083
50000
2.70+00.u
2.52
0.208
Zero-cosi-PT 2.64
2.43
0.018
In order to extend the comparison between zero-cost NAS (NASWOT) and our zero-cost PT to the DARTS CNN search space, we have conducted further experiments, in which we allow NASWOT to sample and score random models from the DARTS search space for for a specified amount of samples (2500, 20000, 50000, corresponding to roughly 25min, 2h, and 5h on single 2080Ti GPU) and the best model, according to the nwot metric, is selected as the result of search. For each of the three sampling budgets, we run the entire process 4 times using different searching seeds (0-3), thus resulting in x44= = 12 final architectures (9 unique
are selected). Each of the final architectures was then trained times using different training seed (0-3). The results are presented in details in Table A2
As expected, both average and best performance of the sample-based zero-cost search increases with more samples. However, the results are visibly behind our proposed method, even for the most expensive searches (5 hours). In addition, hours does not result in a proportional gains in accuracy, compared to 25min, suggesting diminishing returns. We hypothesize it is related to the mentioned fact that the number of architectures grows exponentially, so we'd d need to sample significantly more networks before the probability of hitting a good one increases noticeably. In fact, we find that when increasing searching budget from 20000 to 50000 samples in experiments, the baseline only get better results in out of 4 cases (searching seeds), and in one it actually made the results worse. This further suggests diminishing returns. Finally, for a similar time budget to ours (25 min), the average performance of the baseline is actually much closer to the random search (3.29=0.15) (Liu al. 2019) than to our method, with significant variance.
see that increasing searching budget from 2 hours to 5
A.6.3 ZERO-COST-DISC BASELINE
In Section. 3 we propose two zero-cost operation scoring function Tdise-ze and xee-pe, and study their cor- relation with an oracle metric best- acc on NAS-Bench-201 (Dong & Yang 2020). We find that discretiza- tion is generally weaker scoring paradigm than perturbation, as shown by their correlations with respect to the oracle score best-ace. To further evaluate the end-to-end NAS perfor-
Table A3: Comparison in test error (%) between zero-cost perturbation-ban based and discretization-based NAS on NAS- Bench-201.
Method
CIFAR-10
CIFAR-100
ImageNet-16
Zero-Cost-DISC
6.22
28.1842.01
14.14
Zero-Cost-PT
5.97 u
27.47 +0.25
53.82
We use the same hyperparameter settings reported in the main paper: Nww 10, V=100, awot zero-cost and random edge discretization order.
mance of discretization vs. perturbation with zero-cost metries, we consider baseline named Zero-Cost-DISC, which discretizes the supernet based on Tiisc instead of Tzz-pt- Details on how disc-zc computes the operation scores can be found in Appendix A.3.2 We compare the performance of Zero-CosttDDSC and our proposed Zero-Cost-PT on MEAS--Bench-20 (Dong & Yang, 2020). as shown in Table A3 We see that discretization (Zero-Cost-DISC) results in inferior performance compared to the proposed perturbation based approach (ero-CCstt-PT) on all datasets, confirming our previous analysis on their correlations with the oracle metric.
A.7 ADDITIONAL ABLATION STUDY ON DARTS CNN SPACE
We conducted ablation study of our Zero-Cost-PP algorithm on NAS-Bench-200 (Dong & Yang 2020) in Section. (42 aiming to decide the best possible configuration of the main hyperparameters of algorithm:
architecture proposal iterations N, validation iterations ordering of edges to follow
19

---

Under review as a conference paper at ICLR 2022
when discretizing, and the zero-cost metric to use. In the following we present additional ablations on the much larger DARTS CNN space, in particular to study the impact different architecture proposal iterations N and validation iterations V when Zero-Cost-PT uses random as the search order and nwot metric.
As detailed in Algorithm ! Zero- Cost-PTrandon firstly proposes candidate architectures using the proposed zero-cost perturbation paradigm, which then evaluated lightweight manner during the validation phase to come up with single outcome of a search. extends the existing zero-cost NAS approach such NASWOT (Mel- lor et al.) 202T) by including more sophisticated architecture selection phase based on the combination of zero-cost metrics and perturbations.
Table A4: Detailed performance of Zero-Cost-PY random with N=1, V=0, nwot metric on DARTS CNN space.
Test Error (%)
S. seed
Training seed 2
Avg.
Best
0
1
2
3
0
2.72
2.55
2.83
2.71
1
3.25
3.26
3.28
3.20
2..2
2.81
2.43
2
2.59
2.84
2.59
2.79
3
2.43
2.77
2.52
2.66
Random seeds for searching architoctures.
Random for training the architectures.
We first consider an treme case, setting archi- tecture proposal iteration N=1, where Zero-Cost- PT only proposes one ar- chitecture candidate (with random edge discritiza- tion order), and with no validation stage per- formed. The detailed re- sults shown in Ta- ble A4] As can seen, the average performance affected quite signif- icantly. However, the best model still happens to be on-par with our main results. This sug- gests what has already been mentioned in our main paper, that search- ing phase alone tends to "find" many different ar- chitectures depending on, broadly speaking, random seed, and this randomness especially visible in the case of random edge ordering (Figure 3al While the high variance might seem undesired at first, empirically observe that the higher exploration resulting from beneficial for finding some very good models, e.g., globallop-iae iter e discretization order tends to be less sensitive to random seed as it takes away one degree of randomness (edge order), producing more stable results average, but at the same time limiting its ability to maximize performance maximize performance of our method, we balance exploration (higher N random edge order) and exploitation (higher v) the searching and validation phase respectively.
Table A5: Detailed performance of Zero-Cost-PT. Cost-PTrandon with N=10, V=(1, 10, 100), nwot metric on DARTS CNN space.
Test Error (%)
S. seed
Training seed
Avg.
Best
2
3
0
3.08
3.16
3.06
2.96
2.74
2.91
2.92
2.92
2.11
2.65
2
2.96
3.10
3.06
2.90
3
2.86
2.85
2.85
2.65
0
3.08
3.16
3.06
2.96
2.74
2.91
2.92
2.87
10
2.88
2.65
2.77
2.71
2.65
2.76
2.83
3.00
2.82
2.87
0
2.86
2.97
2.77
2.82
2.51
2.47
2.56
2,43
100
2.64
2.43
2.74
2.73
2.54
2.62
2.43
2.77
2.52
2.64
Random seeds for searching the architectures. Random seeds for training the selected architectures.
the best model found (Table 4). order to
Admittedly, the interplay between those two phases is crucial for our method. To further showcase how the validation phase complements the searching phase, DARTS CNN space with N=10 and V= V,/1,10.1 1010).00), the results are shown in Table The results are consistent with what is shown in the main paper: higher V produces better results on average but does not affect the best case that much (the best model is still upper-bounder bounded by what was found with N=1).
additional ablations the
20

---

Under review as a conference paper at ICLR 2022
Search Space
1
Search Space
Search Space
darts svg. vel best vel avg. s.teet best Liest
0.00
9.
2.03
04
0.73
0:73
0.22
Q.1
6.38
0.6
0.6
0:75
0.
0.12
0.0
08
0:75
0:22
0.7
-04
9.42
0.
avg. test best_ val avg_val derts
test best_vel
darts
20-pt
wegttest best_val avg.val darts
Figure A3: Spearman's 's rank correlation coefficient of different operation scoring metrics with each other at the first iteration NAS evaluated on NAS-Bench- 1Shot benchmark.
A.8 ADDITIONAL EXPERIMENTS ON NAS BLN--BNHHHHI 1SHCT
In addition to experiments on NAS-Bench-22 201 (Dong Yang 2020). DARTS CNN space (Liu
et al. (2019) and DARTS subspaces $1-S4 (Wang et al. (021), we perform additional experiments NAS-Bench- Ishotl (Zela al. 2020b) further evaluate the performance of our Zero-Cost-PT algorithm. We first extend our correlation analysis from Section (in particular, the initial operation scoring in Section 1.2.1) We compare to DARTS as it is already available in the NB shotl codebase ! As in Figure A3 the results show that DARTS is surprisingly well-correlated both best and avg accuracy, in some cases even better than our proposed ZC-PT. We believe that this is because the search space does not contain skip connections and overall is rather unusual compared to others used with differentiable NAS, so it is possible that constitutes an edge-case where DARTS performs relatively well.
Table A6 shows the NAS results. We see that DARTS retains its high performance. On the other hand, our ZC-PT achieves worse aver-
Table A6: Performance of DARTS (Liu et al. 2019) and our Zero- CoTt-CC on NAS-Bench- IShotl ZZel al. 2020b).
age performance, with noticeable variance (espe- cially on the largest Space 3), but is also able to find better models in the best-case scenario. We would like to notice that unlike DARTS or NAS- Bench-200 201c tains architectural parameters associated with entire connections between cells (a and in the paper (Zela et al. (020b)), additionally to the standard ones associated with candidate opera- tions in single layer (3 in the paper). What it means for our method is that for those param- eters we no longer perturb a single edge of supernet but rather the entire path, making the setting for our algorithm noticeably different.
Test Error (%)
Method
Space
1snch- supernet
Avg.
Best
DARTS
6.67
6.44
1
Zero-Cost-PT
6.76
5.45
DARTS
6.68
6.24
2
Zero-Cost-PT
6.8420.40
6.14
DARIS
6.64
6.5
3
Zero-Cost-PT
7.45
6.03
Further extending method from operation scoring to path scoring is very relevant goal for future work.
A.9 ADDITIONAL EXPERIMENTS ON MOBILENET-LIKE SEARCH SPACE
In addition to the DARTS-like search spaces (including MAS-Bench-2011 DARTS CNN Space, DARTS subspaces SI S4) studied in our paper, for completeness we also conduct additional experiments on the MobileNet-like search space, to verify the robustness and generalization capability of the proposed Zero Prro-CCst-PPT approach.
It is well known that most of the existing NAS algorithms designed for MobileNet-like space constrain the model #FLOPS/Params during the search process. However, our method is not designed for such constrained NAS context (like the original DARTS and DARTS-PT), To the best of our knowledge,
httpp::/ http://ggtttuuuoc hww.sommmm//aaecnhhhaa 1shot
21

---

Under review as a conference paper at ICLR 2022
Table A7: Error and search cost of Zero-Cost-PT MobileNet-like search space (ImageNet)
Architecture
Top-1 Error (%)
Top-5 Error (%)
Params. (M)
Cost (GPU Days)
ProxylessNAS (GPU)
24.9
7.5
7.1
8.3
Pero-Cost-PP(sses
24.0
0.041
Paro-Cost-PP(seet
23.6
0.041
Prno-Cost-PT(seeee 2)
23.9
7.0
8.3
0.041
so far there is no clear solution on how one could constrain #FLOPS/Params of the resulting final architecture during the process of discritizing operations on edges of the supernet. Essentially, to do edge of the supernet need consider both their scores
that, when selecting the operations and the potential contributions to the sum of FLOPS/Params of the final model, which is potentially a NP hard problem. Therefore, in our experiments MobileNet-like search space, we do not enforce constraints on #FLOPS/Parass during search, as it is less relevant the proposed approach.
A.9.1 EXPERIMENT SETTING
We adopt the same settings as in (Cai et al. 2019) and construct supernet with 21 choice blocks, and each block has the following alternative operations:
x22 6 MobileNet blocks, with 3 different kernel sizes [3,5, 7) and expansion ratio (3, 6)).
skip connection.
We follow the previous work (Cai et al. 2019) on this space and search directly on ImageNet dataset (Deng et al.] 2009). with input size 224. We use different random seeds (0-2) to perform architectures search, and train discovered final models (3 models searched and trained in total). We use bat ch-size- 1024 and training was performed on 8 NVIDIA V100 GPUs for 300 epochs, with initial learning rate set 0.5. We colour-jitters, random horizontal flip and random crop for data augmentation, set label heabel_smmotheeeee = 0.1. All other training setting is identical to ProxylessNAS (Cai et al. 2019).
A.9.2 RESULTS
Table A7 shows the performance (error %) of the architectures discovered by the proposed Zero-Cost- PT algorithm ImageNet. We see that comparing to ProxylesNNAS, with comparable or better accuracy, with much less search cost. This further confirms the robustness and transferability of the Cero-Cost-PTT approach, on a different search space in addition to the DARTS-like spaces studied in our paper. Noticeably the models found by our approach have higher #Params comparing to ProxylessNAS, but as we discussed above, constraining PPPS//aassss is less relevant in context, and the aim of this experiment is to show Zero-Cost-Pi can also discover good performing architectures on MobileNet-like space with low search cost.
approach can find models
A.10
DISCOVERED ARCHITECTURES
Figures
A4 and A5 present cells found by our Zero-Cost-PTI on the DARTS CNN search space and random discretization orders, respectively (see 2 for the definition of the two discretization orders). Figures A7 through A18 show cells
(Section
5.2 when using gl oba abbal-op-ite
Section
show discovered the four DARTS subspaces and the three relevant datasets (Sections 5.3 and A.4).
Figure A19 shows architectures found by our Pero-Cost-PT approach on MobileNet-like search space.
48.mer_3d
*p ctex
www. 3r
Xy. con.
up mex 'S7
(a) Normal cell
(b) Reduction cell
Figure A4: Cells found by Zero-Cost-PT sesllobal-ppittee discretization order) on the DARTS search space using CIFAR-10.
22

---

Under review as a conference paper at ICLR 2022
cose fa5
Ceg cnee
Mg.ieee J03
Normal cell
(b) Reduction cell
Figure A5: Cells found by Zero-Cost-PT (random discretization order) the DARTS search space using CIFAR-10.
30
*P. wtr_enec.o0 703
Mg.ssess 365 cias Sa5
(a) Normal cell
(b) Reduction cell
Figure A6: Cells found by Zero-Cost-PT (random discretization order) on the DARTS search space using ImageNet.
41 2L. Sed
(a) Normal cell
(b) Reduction cell
Figure A7: Cells found by Zero-Cost-Pi (random discretization order) the DARTS-SI space using 1I.
tep coes 765
16-4
Ar:: cnex
(a) Normal cell
(b) Reduction cell
Figure A8: Cells found by Zero-Cost-PT (random discretization order) on the DARTS-SS S2 space using CIFAR-10.
coee 363
Kg_sces 3u3
Mg_uces 3u3
(a) Normal cell
(b) Reduction cell
Figure A9: Cells found by Zero-Cost-PT (random discretization order) the DARTS-S S3 space using CIFAR-10.
14
11-11
My. soes 305 363
(a) Normal cell
(b) Reduction cell
Figure A10: Cells found by Zero-Cost-PT (random discretization order) on the DARTS-S4 space using CIFAR-10.
23

---

Under review as a conference paper at ICLR 2022
diccore
dow 3+3
tx3 363
up mp_rree_300
(a) Normal cell
(b) Reduction cell
Figure All: Cells found by Zero-Cost-PT (random discretization order) the DARTS-S SI space using CIFAR-100.
iep_stes__00 ogg etay
3u3
akp commedt
apurree 200
(a) Normal cell
(b) Reduction cell
Figure A12:
Cells found by Zero-Cost-PT (random discretization order) on the DARTS-S2 space using CIFAR-100.
rineect
201 po0
SBU DAY 303 1BL MAW
363
rep
HULSIn 20
Ho 2001
(a) Normal cell
(b) Reduction cell
Figure A13: Cells found by Zero-Cost-PT (random discretization order) on the DARTS-S! space using CIFAR-100.
sep cae
coee: 365
mww. 200
500 ORW 363
sril cnex 203
Sep
(a) Normal cell
(b) Reduction cell
Figure A14: Cells found by Zero-Cost-PT (random discretization order) on the DARTS SS4 space using CIFAR-100.
pooi
cony 5A5
JAN Jool 3x3
comv 5a5
co0Y 515
sum. Jn3
(a) Normal cell
(b) Reduction cell
Figure A15: Cells found by Zero-Cost-Pa (random discretization order) on the DARTS-S S1 space using SVHN.
sk0 comnet
coes lu3
eg_rreex_o3
My. cues
Xy coss
(a) Normal cell
(b) Reduction cell
Figure A16: Cells found by Zero-Cost-PT (random discretization order) on the DARTS-S? space using SVHN.
24

---

Under review as a conference paper at ICLR 2022
SELSEES MELAREN 305
Mp: cree
sy: clss 505 okzzommod
(a) Normal cell
(b) Reduction cell
Figure A 11: Cells found by Zero-Cost-PT (random discretization order) on the DARTS-S3 space using SVHN.
Sp. came
CaRS 3x3
X0 100-0ee.00
men 203
(a) Normal cell
(b) Reduction cell
Figure A18: Cells found by Zero-Cost-Pi (random discretization order) on the DARTS-SE -S4 space using SVHN.
c-56
C-256
G-432
-
Seed
Veel
Figure A19: Discovered Architectures in MobileNet-like Search Space
25

---

Under review as a conference paper at ICLR 2022
Table A8: Raw values of operation scoring functions at iteration 0 to reproduce Figure 2a
edge'
none
skip_connee _op_conncct
nor_conv _x1
nor_conv__3
avg_ wwg_pool_33x _33
94.15
94.18
94.44
94.68
93,86
94.24
94.16
94.49
94.68
94.09
best-acs acc
94.25
94.43
94.49
94.68
94.19
94.16
94.68
94.03
94.04
93.85
94.29
94.18
94.56
94.68
94.23
94.05
94.16
94.68
94.56
94.1
77.36
81.02
83.81
86.38
87.32
80.03
83.11
85.23
85.99
81.52
arg-ace
82.9
82.44
84.05
84.49
81,98
74.02
85.17
87.3
88.28
81.38
80.14
83.05
85.09
85.7
81,89
77.61
83.43
86.18
86.95
81,74
83.27
82.24
65.0
71.76
54.31
disc- dec
84.94
83.23
73.23
76.77
83,45
83.87
83.73
77.33
76.83
83.25
65.77
84.44
75.82
78.68
62.7
83.57
82.03
75.02
76.09
82.56
83.95
82,45
66.69
71,36
80.31
-85.43
-17.02
-78.13
-5..09
-8.34
-85.22
-36.1
844399
-80.95
-85.49
disc-2e darts-pt'
-85.51
-80.29
-81.86
-77.68
-85.32
-85.49
-9.66
-81.79
-59.18
-85.48
-85.45
-51.15
-7.84
-64.64
-85.14
-85.54
-32.43
-81.04
-72.75
-85.51
3331.01
3445.49
3366.88
3437.55
3423.18
3429.07
3435.75
3407.87
3434.58
3421.44
3428.8
3423.36
3440.93
3437.29
3416.89
3408.99
3464.05
3359.89
3382.18
3431.81
3433.99
3435.57
3424.47
3431.14
3423.15
3434.42
3437.66
3418.57
3397.52
3424.17
3355.23
33499.9
3449.54
-341.82
3341.18
15
3448.7
3441.81
3340.65
3453.74
72-pt
3446.52
3447.61
3335.46
3436.4
3449.28
-353.881
-3435.99
3344.04
-3445.6
3347.07
3451.06
3349.8
3442.63
-3411.13
3453.31
3450.97
3448.21
3440.8
3443.24
3452.99
0.14
0.48
0.13
0.18
0.07
0.12
0.55
0.11
0.12
0.09
darts
0.24
0.33
0.15
0.17
0.11
0.06
0.08
0.13
0.07
0.12
0.48
0.13
0.17
0.1
0.16
0.49
0.12
0.14
0.09
-38.5
-3.00
-6.0
-37.5
-7.0
-55.0
-10.0
-15.0
-39.0
tenas
-31.5
-10.0
-3..0
-16.5
36.55
-34.0
-44.0
-53.5
-23.0
-3000
-32.0
-32.5
-36.5
-32.0
-52.0
-38.5
-16.0
20.00
-17.0
-27.5
"Lower better add negative "-pt
26


Reviews:
[{"review_id": "ahf0brbPsC", "reviewer": "Reviewer_podk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- This is a great combination of two popular areas: zero-cost proxies, and differentiable NAS via perturbation-based operation scoring. It is a natural idea to combine these two things, but also not trivial to do it well.\n- These ideas can have a high impact.\n- It seems that they give new insights for the celebrated “Rethinking Architecture Selection in Differentiable NAS” paper, including more proof of why DARTS-PT does well, and a refutation of the claim that disc-acc does well.\n- The appendix is pretty good and has a lot of information for reproducibility (but for code, see weaknesses).\n- Sections 3.1 and 3.2 are nicely done and make sense, and Fig 2 in particular is very interesting.\n\n## Weaknesses\n- Over-reliance on NAS-Bench-201. Almost all of Sections 3 and 4 focus on NAS-Bench-201. But this is the smallest search space, and in particular, it is known that zero-cost proxies do well on NAS-Bench-201 and bad on larger search spaces (the authors mention this as well). Although the authors also test their full algorithm on the DARTS search space in Section 5, it would strengthen the paper to do the analysis of Sections 3 and 4 on at least one other search space. NAS-Bench-1shot1 is the best option because it is size 360k and can run one-shot algorithms. TransNAS-Bench-101 can also run one-shot, and it is smaller but would add diversity. Finally, it might be possible to run some of the experiments using the 60k trained architectures from NAS-Bench-301).\n- Better ablation between the search and the validation stage. Sections 3 and 4 motivate the search part of the algorithm, but adding in the validation stage makes it less clear what leads to strong performance, especially on DARTS where the search step was never run in isolation. I think it makes it less apples-to-apples comparisons, e.g. because DARTS, DARTS-PT etc do not have the final validation stage. The authors did have an ablation, but it is only on NAS-Bench-201 and does not cover search only (\"validation only\" is basically NASWOT). In order to have a better comparison between ZC-PT and DARTS-PT, the authors could run ZC-PT once with no validation stage. Or, run ZC-PT and DARTS-PT both V times and do the validation stage for both of them. For Tables 2, 4, 5. As a result, I think the claim in the intro that \"Our novel Zero-Cost-PT improves searching time and accuracy compared to the best available differentiable architecture search for many search space sizes, including very large ones.\" is misleading. \n- The authors claim in Section 7 that they released their code in the supplementary material, but unfortunately there was no supplementary material submitted. If the authors provide the code during the rebuttal period, I will feel more positive about the paper (for example, anonymous github).\n", "summary_of_the_paper": "Differentiable neural architecture search (NAS), and more recently, perturbation-based operation selection in differentiable NAS, is a popular recent area of study in NAS. In parallel, zero-cost proxies are also gaining in popularity in NAS. In this paper, the authors combine the insights from perturbation-based operation selection and zero-cost proxies, to create a new method that sees substantial time speedups. Specifically, they formalize \"operation scoring\", and use zero-cost proxies to score operations with the perturbation paradigm. This leads them to a new NAS method (Zero-Cost-PT) that outperforms existing methods. They evaluate their proposed algorithm on the DARTS search space (and its subsets) and NAS-Bench-201, showing that it can achieve up to 40x speedups.", "summary_of_the_review": "This paper combines two popular techniques in a nice way. I think the ideas in this paper can be pretty impactful. I do have three weaknesses about reliance on NAS-Bench-201, a fairer experimental setup, and code release, so I will give a weak accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L-z75e1BmW", "reviewer": "Reviewer_q4ge", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper is well-written and easy to follow. However, analyze part is slightly too long however the paper is in general in good shape.\n2. The experiment results look promising and interesting. It could achieve better accuracy across several benchmarks and some spaces, which could be useful to the community.\n\nCons:\n1. The paper's novelty is quite limited. It simply combines the differentiable neural architecture search with the one-shot one. Although it has been shown and addressed across the paper the zero-cost proxies are quite useful, however, it lacks insight on how it is correlated with the final performance ranking. \n2. At the same time, all the zero-cost proxies are proposed by other works and there is limited understanding of why some specific proxies are clearly better than others in the differentiable setting but not in the one-shot setting. \n3. Although the paper formalizes the problem into perturbation and discretion, I find only discretization is used in the final experiments, where the formalized framework seems not that important.", "summary_of_the_paper": "The paper proposes to use the zero-cost proxy in the operation selection in differentiable neural architecture search. Specifically, the paper formalizes the selection procedure into perturbation and discretization. By introducing the zero-cost proxy, they first find the proxy could yield a better ranking compared to both Darts and Darts-PT in the Nas-Bench 201. They also experiment with different proxies proposed by previous work and find they all perform better than the Darts-PT. In the Darts-CNN and S1-S4 experiments, they select one of the best-performed proxies and find it could yield better accuracy than other baselines.", "summary_of_the_review": "Although the paper has shown us the zero-cost proxy has great power in operation search in the differentiable neural architecture search, the paper stills lack insight on analyzing how it works. Therefore, I rate it as a borderline paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HAI0vJnx_hM", "reviewer": "Reviewer_G1Xa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nThis paper introduces the lightweight operation scoring based on zero-cost proxies, which empirically outperform existing operation scoring functions. The experiments show that the perturbation is more effective than discretization and propose Zero-Cost-PT. Extensive experiments empirically show that the proposed method outperforms the best available differentiable architecture search in terms of searching time and accuracy. \n\nCons:\nThe zero-cost proxies method has been used in the previous work. This work only borrows it into differentiable architecture search, which is only a combination. There exist some typos and it is a little difficult to understand Figure 1. Some annotations can be moved to the caption. \n", "summary_of_the_paper": "In this paper, the authors introduce new training-free proxies to the context of differentiable NAS, which can speed up the search process while improving accuracy. Further, the authors propose, evaluate and compare perturbation-based zero-cost operation scoring (Zero-Cost-PT) for differentiable NAS. Extensive experiments empirically show the effectiveness of Zero-Cost-PT in six search spaces and 3 datasets. ", "summary_of_the_review": "Generally speaking, I think it is incremental work. Although there is a lack of theoretical contributions, a large number of experiments provide empirical contributions. It is difficult for me to put forward some new opinions about these large numbers of experiments. I give the above acceptance threshold, but reject is OK.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "68Wq3y_vxPe", "reviewer": "Reviewer_86cZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThis work studied different training-free proxies and leveraged perturbation-based methods for NAS.\n\nWeakness:\nMy main concern is the novelty of this work. Specifically,\n1. This work did not propose any new proxies for evaluating networks performance. If I understand correctly, the proxies listed in Table 2 are all from previous works. The authors did not claim any novelty in these training-free proxies either. Although the authors mentioned \"opportunity to use a new class of training-free proxies\" in Sec. 3.1, there is no concrete proposal in this work.\n2. The operation sorting method (Eq. 5) is also similar to the perturbation method proposed by [1]. The main difference is that the authors adopt this perturbation at supernet's initialization.\n\n[1] Rethinking architecture selection in differentiable NAS.", "summary_of_the_paper": "This work combines training-free proxies and perturbation-based subnetwork evaluation to achieve efficient NAS. The author also conducted comprehensive study of training-free proxies under different training iterations.", "summary_of_the_review": "In addition to the novelty issue mentioned above, some results and description of this work also makes me a bit confusing. For example, when demonstrating Figure 2, the author did not mention which concrete training-free proxy they used for $S$ in Eq. 5. Also, the purpose of \"retrain for 5 epochs\" is unclear to me. Why does the author want to conduct this experiment given the \"zero-cost\" motivation in this work?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received scores of 5,5,6,8. The reviewer giving a score of 8 stated that they would've given a 7, but that that is not an option in the system. The other reviewer giving an acceptance scores mentioned that they would also be OK with a rejection. The details of the assessment are thus less enthusiastic than could be assumed with an overall average score of 6. I am therefore weakly recommending rejection.

The main criticisms of the reviewers are lack of novelty, lack of deeper analyses that really provide insights into why zero-cost operation scoring works, and lack of the number of NAS benchmarks tested. Out of these, personally, I would not criticize a lack of novelty, since it is not trivial to put together zero-cost and one-shot methods and the results appear promising.
However, even the most positive reviewer criticized that the work focuses on NAS-Bench-201 heavily (which is particularly problematic given that NAS-Bench-201 uses a fixed wiring and only allows the choice of operations; this may make the proposed method particularly applicable). During the rebuttal, the authors added NAS-Bench-1shot, which is a very good step, but the proposed technique does not actually work as well there. While this may be due to the special nature of operations in the nodes rather than in the edges for NAS-Bench-1shot1, for a revision, it would be good to add additional experiments on further NAS benchmarks in order to allow for a better understanding under which circumstances the proposed method works well. In particular, it would be interesting how well the method works on a quite different search space, such as the one of MobileNet.
Decision: Reject