Zero-Cost Proxies for Lightweight NAS
Authors: Mohamed S Abdelfattah
Abhinav Mehrotra
Łukasz Dudziak
Nicholas Donald Lane
Venue: ICLR 2021 Poster (2021)
Keywords: NAS
AutoML
proxy
pruning
efficient

Abstract: Neural Architecture Search (NAS) is quickly becoming the standard methodology to design neural network models. However, NAS is typically compute-intensive because multiple models need to be evaluated before choosing the best one. To reduce the computational power and time needed, a proxy task is often used for evaluating each model instead of full training. In this paper, we evaluate conventional reduced-training proxies and quantify how well they preserve ranking between neural network models during search when compared with the rankings produced by final trained accuracy. We propose a series of zero-cost proxies, based on recent pruning literature, that use just a single minibatch of training data to compute a model's score. Our zero-cost proxies use 3 orders of magnitude less computation but can match and even outperform conventional proxies. For example, Spearman's rank correlation coefficient between final validation accuracy and our best zero-cost proxy on NAS-Bench-201 is 0.82, compared to 0.61 for EcoNAS (a recently proposed reduced-training proxy). Finally, we use these zero-cost proxies to enhance existing NAS search algorithms such as random search, reinforcement learning, evolutionary search and predictor-based search. For all search methodologies and across three different NAS datasets, we are able to significantly improve sample efficiency, and thereby decrease computation, by using our zero-cost proxies. For example on NAS-Bench-101, we achieved the same accuracy 4$\times$ quicker than the best previous result. Our code is made public at: https://github.com/mohsaied/zero-cost-nas.

Published as a conference paper at ICLR 2021
ZERO-COST PROXIES FOR LIGHTWEIGHT NAS
Mohamed S. Abdelfattah', Abhinav Mchrotra', Lukasz Dudziak Nicholas D. Lane
1 Samsung Al Center, Cambridge University of Cambridge mohamed1 @ssamuung.com
ABSTRACT
Neural Architecture Search (NAS) is quickly becoming the standard methodology to design neural network models. However, NAS is typically compute-intensive because multiple models need to be evaluated before choosing the best one. To reduce the computational power and time needed, a proxy task is often used for evaluating each model instead of full training. In this paper, we evaluate con- ventional reduced-training proxies and quantify how well they preserve ranking between neural network models during search when compared with the rankings produced by final trained accuracy. We propose a series of zero-cost proxies, based on recent pruning literature, that use just single minihatch of training data compute model's s score. Our zero-cost proxies use 3 orders of magnitude less computation but can match and even outperform conventional proxies. For ex- ample, Spearman's 's rank correlation coefficient between final validation accuracy and our best zero-cost proxy on NAS-Bench-201 is 0.82, compared to 0.61 for EcoNAS (a recently proposed reduced- training proxy). Finally, we use these zero- cost proxies to enhance existing NAS search algorithms such as random search, reinforcement learning, evolutionary search and predictor-based search. For all search methodologies and across three different NAS datasets, we are able to sig- nificantly improve sample efficiency, and thereby decrease computation, by using our zero-cost proxies. For example NAS-Bench- 101, we achieved the same accuracy 4x quicker than the best previous result. Our code is made public at: https://githaie wtt.://....ooo.h..a.e.ceeeeeee zero cost-nas
INTRODUCTION
Instead of manually designing neural networks, neural architecture search (NAS) algorithms
used to automatically discover the best ones (Tan & Le 2019a Liu et al. 2019 Bender et al., 2018). Early work by Zoph & Le (2017) proposed using reinforcement learning (RL) controller that constructs candidate architectures, these are evaluated and then feedback is provided to the controller based on the performance of the candidate. One major problem with this basic NAS methodology is that each evaluation is very costly typically on the order of hours or days to train a single neural network fully. We focus on this evaluation phase we propose using proxies that require a single minibatch of data and single forward/backward propagation pass score neural network. This is inspired by recent pruning-at- -tt----tttiilzzattiiii work by Lee et al. (2019), Wang et al. (020) and Tanaka et al. (2020) wherein a per- per-marmetter saliency metric is computed before training to inform parameter pruning. Can we use such saliency metrics to score an entire neural network? Furthermore, can networks for use within NAS? If so, how do we best integrate these metrics within existing NAS algorithms such as RL or evolutionary search? These are the questions that we hope (empirically) tackle in this work with the goal of making NAS less compute- hungry. Our contributions are:
these "single minibatch" metrics to rank and compare multiple neural
Zero-cost proxies We adapt pruning-an soauinggat--ninn ittinitialization metrics for use with NAS. This re- quires these metries operate at the granularity of an entire network rather than individual parameters - we devise and validate approaches that aggregate parameter-level metrics in manner suitable for ranking candidates during NAS search.
Comparison to conventional proxies We perform a detailed comparison between zero- cost and conventional NAS proxies that use a form of wewluudddoompuattinnn training. First, we quantify the rank consistency of conventional proxies on large-scale datasets: 15k mod- els vs. 50 models used in (Zhou et al.) 2020). Second, we show that zero-cost proxies can match or exceed the rank consistency of conventional proxies.

---

Published as a conference paper at ICLR 2021
Ablations on NAS benchmarks We perform ablations of our zero-cost proxies on five different NAS benchmarks the zero-cost metrics under different settings, and expose properties of successful metrics.
and PyTorchCV) to both test
Integration with NAS Finally, we propose two ways to use zero-cost metrics effec- tively within NAS algorithms: random search, reinforcement learning, aging evolution and predictor-bast based search. For all algorithms and three NAS datasets we show significant speedups, up to 4x for
101 compared to current state-of-the-art.
RELATED WORK
NAS Efficiency To decrease NAS search time, various techniques were used in the literature. Pham et al. (2018) and Cai et al. (2018) use weight sharing between candidate models to decrease the training time during evaluation. Liu et al. (209)) and others use smaller datasets (CIFAR-1O) as a proxy to the full task (ImageNerIk). In EcoNAS, Zhou et al. (2020) extensively investigated reduced-trainge training proxies wherein input size, model size, number training samples and number of epochs were reduced in the NAS evaluation phase. We compare to EcoNAS in this work to elucidate how well our zero-cost proxies perform compared to familiar and widely-used conventional proxies.
Pruning The goal is to reduce the number of parameters in a neural network, way to do this is by identifying a saliency (importance) metric for each parameter, and the less-important parameters removed. For example, |Han et al. (2015), Frankle & Carbin (2019) and others use parameter magnitudes as the criterion while LeCun et al. Hassibi & Stork 1993) and Molchanov et al.) (2077) use gradients. However, the aforementioned works require training before computing the saliency criterion. A new class of pruning-at taaning--t-inn saapnng-a--iniiiinn initialization algorithms, that require no training, were introduced by [ee et al. (2099) and extended by Wang et al. (2020) and Tanaka et al. (2020). A single forward/backwarr propagation pass is used to compute a saliency criterion which is successfully used to heavily prune neural networks before training. We extend these pruning-at-inne -tiiiaaaaiiniiiiizzaiinn criteria towards scoring entire neural networks and we investigate their use with NAS algorithms.
Intersection between pruning and NAS Concepts from pruning have been used within NAS multiple times. For example, M al. (2020) use channel pruning in their AtomNAS work ar- rive at customized multi-kernel-size convolutions (mixconvs as introduced by Tan & Le 2019b)). In their Blockswap work, TTumer et al. (2020) use Fisher information initialization to score different lightweight primitives that are substituted into a neural network to decrease computation. This is the carliest work we could find that attempts to perform type of NAS by scoring neural networks without training using a pruning criterion, More recently, Mellor et al. al.](202) (2020) introduced a new met- ric for scoring neural networks initialization based on the correlation of Jacobians with different inputs. They perform "NAS without training" by performing random search with their zero-cost metric (jacob_cov _cov to rank neural networks instead of using accuracy. We include jacob_cov in our analysis and we introduce five more zero-cost metrics in this work.
3 PROXIES FOR NEURAL NETWORK ACCURACY
3.1 CONVENTIONAL NAS PROXIES (ECONAS)
In conventional sample-based NAS, a proxy training regime is often used to predict a model's 's accu- racy instead of full training. Zhou al. (2000) investigate conventional proxies in depth by com- puting the Spearman rank correlation coefficient (Spearman p) of a proxy task to final test accuracy. The proxy used is a reduced-compine reduced-computatioon training, wherein one of the following four variables is reduced: (1) number of epochs, (2) number of training samples, (3) input resolution (4) model size (controlled through the number of channels after the first convolution). Even though such proxies were used in many prior works, EcoNAS is the first systematic study of conventional proxy tasks that we found. One main finding by Zhou et al. (2020) is that using approximately 1 of the model size and input resolution, all training samples, and Ty the number of epochs was a reasonable proxy which yielded the best results for their experiment (Zhou et al. 2020).
3.2 ZERO-COST NAS PROXIES
We present alternative proxies for network accuracy that can be used to speed up NAS. A simple proxy that we use is grad norm in which we sum the Fuclidean norm of the gradients after
2

---

Published as a conference paper at ICLR 2021
single minibatch of training data. Other metrics listed below were previously introduced in the context of parameter pruning at the granularity of single parameter saliency is computed to rank parameters and remove the least important ones. We adapt these metrics to score and rank entire neural network models for NAS.
3.2.1 SNIP, GRASP AND SYNAPTIC FLOW
In their snip work, Lee et al. (2019) proposed performing parameter pruning based on a saliency metric computed at initialization using a single minibatch of data. This saliency criteria approxi- mates the change in loss when a specific parameter is removed. |Wang et al. 2020) attempted to improve the snip metric by approximating the change in gradient norm (instead of loss) when a parameter is pruned in their grasp objective. Finally, Tanaka et al. (2000) generalized these
so-called synaptic saliency scores and proposed modified version (syn low) which avoids layer collapse when performing parameter pruning. Instead of using a minibatch
training data and
cross-entropy loss (as snip grasp), with synflow we compute loss which is simply the
product of all parameters in the network; therefore, no data is needed to compute this loss or the synflov metric itself. These the three metrics:
0C
o o/. 00
0L
OL
snip: SS(00)
graap Sp( = --H
3 synflow Sp(0) = 00 00
o 0
(I)
where L is the loss function of neural network with parameters 0, H is the Hessiar Sp is the per-parameter saliency and @ is the Hadamard product. We extend these saliency metrics to score an entire neural network by summing over all parameters N the model: S, = Σ (O),
3.2.2 FISHER
Theis et al. (2018) perform channel pruning by removing activation channels (and their correspond- ing parameters) that are estimated have the least effect the loss. They build the work of Molchanov et al. (2017) and Figurnov et al. (2016). More recently, Turner et al. (2020) aggregated this fisher metric for all channels in convolution primitive quantify the importance of that primitive when it is replaced by more efficient alternative. We further aggregate the fisher met- ric for all layers in neural network score an entire network as shown in the following equations:
0C 2
M
fisher 8.(2)=
S, = 5 Sz(2)
0z
where S2 is the saliency per activation 2, and M the length the vectorized feature map.
3.2.3 JACORIAN COVARIANCE
This metric was resione to score neural networks in the context of NAS - we refer the reader the original paper for detailed reasoning and derivation of the metric which we call jacob_cov (Mellor et al. 2020). In brief, this metric captures the correlation of activations within network when subject different inputs within a minibatch of data the lower the correlation, the better the network is expected to perform as it can differentiate between different inputs well.
4 EMPIRICAL EVALUATION OF PROXY TASKS
Generally, most of the proxies presented in the previous section try capture how trainable a neural network is by inspecting the gradients at the beginning of training. In this work, we refrain from attempting to explain precisely why each metric works (or does not work) and instead focus on empirical evaluation of those metrics in different scenarios. We use the Spearman rank correlation coefficient (Spearman p) to quantify how well a proxy ranks models compared to the ground-truth ranking produced by final test accuracy (Daniel 1990).
'The full Hessian does need be explicitly constructed explained by Pearlmutter (1993).
3

---

Published as a conference paper at ICLR 2021
0.85
econase D.8).
0.80
0.75
baseliie S
econas F2C4
econes fuula
econas F.cfa
0.60
(5.0]0.61) ecortss
0.55
0.50
1/64
1/8
Epoch
Normalized FLOPS
Normalized GPU Runtime
Figure 1: Evaluation of different econas proxies on WAS-Bench-200 CIFAR- 10. FLOPS and
runtime are normalized the FLOPS/runtime of a single baseline (full training) epoch.
4.1 NAS-BENCH-200
NAS-Bench-201 is purpose-built benchmark for prototyping NAS algorithms (Dong & Yang 2020). It contains 15,625 CNN models from cell-based search space and corresponding train- ing statistics. We first use evaluate our zero-cost proxies and compare the two approaches.
to evaluate conventional proxies from EcoNAS, then we
4.1.1 ECONAS PROXY ON NAS-BENCI-2001
Even though Zhou et al. 2020) thoroughly investigated reduced- training proxies, they only evalu- ated small model zoo consisting of 50 models. To study EcoNAS more extensively we evaluate it all 15,625 models in NAS-Bench-20 search space (training details in A.I]. The full configu- ration training of AAS-Bench-201 on CIFAR-10 uses input resolution r=32, number of channels in the stem convolution c=16 and number of epochs e=200 we summarize this as: r32C10F200-
According to the EcoNAS study, the most effective configuration divides both the input resolution and stem channels by -4 and the number of epochs by 10, that is, raC4620 for NAS-Bench- models. Keeping that in mind we investigate r3C4 in Fig. (labeled econas); however, this proxy training seems to suffer from overfitting correlation final accuracy started drop after 20 epochs. Additionally, the Spearman was modest 0.61 when evaluated on all 15,625 models in NAS-Bench-201
far cry from the 0.87 achieved on the 50 models in the EcoNAS paper (Zhou et al. 2020). We additionally explore TsCs, T16 and F16Cs and find a very good proxy with T16Cs61s, labeled Fig. Mas econast. From the plots Fig. we would like to highlight that:
1. A reduced-training proxy that works well other as highlighted by the difference in Spearman p between econas and econast. CIFAR-10 image classification.
search space may not work well
This occurs though both tasks in this
2. Even though EcoNAS-style proxies reduce computation load by large factor (as seen in the middle plot in Fig. []] this does not translate fully into actual runtime improvement when
nominal desktop G We therefore plot actual GPU speedup in the third subplot in Fig. [ For example, notice that the point labeled econas (18c4e20) has the same FLOPS "To of full training epoch, but when measured GPU, takes time equivalent to 5 full training epochs 50x gap between theoretical and actual speedup.
4.1.2 ZERO-COSt PROXIES ON MAS-BENCH-OE
We now shift our focus towards our zero-cost NAS proxies which rely on gradient computations using a single minibatch of data at initialization. A clear advantage of zero-cost proxies is that they take very little time to compute- the forward/backkward pass using a single minibatch data. We the zero-cost proxies on all 15,625 models in NAS-Bench-201 for three image classification datasets and we summarize the results Table
The synf 1 low metric performed the best on all three datasets with Spearman p consistently above 0.73, j jacob_cov was second best but was also very well-correlated to final accuracy. Next came grad norm and snip with Spearman p close to 0.6. We add another metric that simply label with e that takes a majority vote between the three metrics synflow www.jacoo jacob.cov cov and
We used Nvidia Geforee GTX 1080 Ti and ran random sample of 10 models for 10 epochs to get an average pire-per-e for each proxy different batch sizes. We discuss this further in Section A.2
4

---

Published as a conference paper at ICLR 2021
Epochs
Epechs
Epochh
10 11
0
9 10
11
10
0.85
0.85
0.85
vote
vote
#Syfflow
0.75
0.70
Jarob.cow
0.70
(an0b CCM
0.70
X/robbcco
Spaarmnn
0.65
Spvarman
0.60
@cconss
0.60
0.60
snip
3rhd ma
(orasp
0.50
pracp
0.50
0.50
0.40
0
500
1000
15:0
2000
5.0
1000
1500
2000
0
5.0
1000
1500
2000
Evoluation Cost (humber of minibatches)
Evaluation (number of minibatches)
Evaluation Cost (humber of minibatches)
(a) CIFAR-10
(b) CIFAR-100
(c) ImageNet16-120
Figure 2: Correlation of validation accuracy to final test accuracy during the first 12 epochs of training for three datasets the NAS-Bench-201 search space. Zero-cost and EcoNAS proxies also labeled for comparison.
Table 1: Spearman p of zero-cost proxies NAS-Bench-201.
Dataset
grad_norm
snip
grasp
fisher
synflow
jacob.cov
vote
CIFAR-10
0.58
0.58
0.48
0.36
0.74
0.73
0.82
CIFAR-100
0.64
0.63
0.54
0.39
0.76
0.71
0.83
ImageNer16-120
0.58
0.58
0.56
0.33
0.75
0.71
0.82
snip when ranking two models. This performed better than any single metric with Spearman consistently above 0.8. At the cost just minibatches instead slightly better than econast, and much better than econas as shown in Fig. 2a In Fig. we also plot the rank correlation of validation accuracy (without any reduced training) over the first 10 epochs of training for the three datasets available in VAS-Bench-201.
this is already performing
Having set comparison point with EcoNAS and reduced- traininnnring proxies, we have shown that zero-cost proxies can match and outperform these conventional methods in a large-scale empirical analysis. However, different NAS search spaces may behave differently, so in the remainder of this section, we test the zero-cost proxies on different search spaces.
4.2 MODELS IN THE WILD (PYTORCHCV)
To study zero-cost proxies in a different setting, we scored the models in the PyTorchCV database (Sémery 2020). PytorchCV contains common state-of-the-art neural networks such as ResNets (He et al.] 2016), DenseNets (Huang et al.] (017). MobileNets (Howard et al. 2017) and EfficientNets (Tan & Le (2019)) - a representative assortment of top-performing models. We evaluated ~50 mod- els for CIFAR-IO, CIFAR- 100 Krizhevsky (2009) and SVHN (Netzer et al. 2011), and ~200 models for ImageNet1k (Deng al. 2009). Fig. shows the resulting correlation for the zero-cost metrics. synflow, snip, fisher and grad_no rm all perform similarly well on all datasets, with the ex- ception of SVHN where synf oo outperforms other metrics by a large margin. However, grasp failed in this setting completely as shown by the low mean Spearman P and high variance as shown Fig. Curiously, jacob_com also failed in this setting even though performed well on NAS- Bench-201. This suggests that this metric is better at scoring models from within search space (similar topology and size), but becomes worse when scoring unrelated models.
4.3 OTHER SEARCH SPACES
We investigate our zero-cost metrics with other NAS benchmarks. Our goal is empirically find good metric to speed up NAS algorithms reliably on different tasks and datasets.
NAS-Bench-101: This is the first and largest NAS benchmark available with over 423k CNN models and training statistics CIFAR-10 (Ying al. 2019).
NAS-Bench-NLP: Klyuchnikovv et al. (2020) investigate the architectures 14k different
recurrent cells in natural language processing (NLP) tasks such next word prediction.
5

---

Published as a conference paper at ICLR 2021
grad_ grad_narm
snip
Spearmnn
0.5
grasp
fisher
0.4
synflow
0.3
jacob ccvv
0.2
0.1
0.0
CIFAR-10
CIFAR-100
SVHNN
ImageNetik
Figure 3: Performance of zero-cost metrics PyTorchCV models (averaged over 5 seeds).
This is our in-house dataset for convolution-based automatic speech recognition models evaluated on the TIMIT dataset (Garofolo et al.) 1993). The search space includes linear, convolution, zeroize and skip-connections, forming 8242 models (Mchrotra et al. 2021
NAS-Bench-ASR:
Compared to MAS--Bnnc-200. these datasets either much larger MAS-Benchh- 101) based different task Cew--Becc-NN.P/SSELE metric (highlighted in bold) is the only consistent across all analyzed benchmarks. even for the synfl 1 Low metric. rank correlation is quite a bit lower than that for Additionally, (~0.3 vs. -0.8). Other than global rank correlation, a search space is also critically important for NAS algorithms this is because we ultimately care about finding those top models. In A..4 we perform an analysis of how top models are ranked by zero-cost proxies. Additionally, local rank correlation of top models could be important for NAS algorithms when two good models are compared using their proxy metric value. Tables and TO show that the only metric that maintains correct ranking among top models consistently across all NAS benchmarks synflow. Section 5 we deliberately evaluate 3 benchmarks that exhibit different levels of rank correlation: MAS--ench-220/1000 within NAS and achieve consistent gains for all three search spaces.
From Table we would like to highlight that the syn fl ow
posit that ranking of top models from
to see if we integrate synflow
Table 2: Spearman p of zero-cost proxies on other NAS search spaces.
grad norm
snip
grasp
fisher
synflow
jacob.cov
NAS-Bench- 0.20
0.16
0.45
0.26
0.37
0.38
TAS-Bench-NLP
-0.21
-0.19
0.16
0.34
0.56
MAS--ench-ASSE
0.07
0.01
0.02
0.40
-0.37
5 ZERO-COST NAS
Mellor al. (2020) proposed using jacob_cov score of randomly-sampled models and to greedily choose the model with the highest score. This "NAS without training" methodology is very attractive thanks its simplicity and low computational cost. In this section, we evaluate our metrics in this setting that we simply call "random search" (RAND). We extend this methodology slightly: instead of just training the top model, we keep training models (from best to worst as ranked by the zero-cost metric) until the desired accuracy is achieved. However, this approach can only produce results that are as good as the metric being used and we have no guarantees (just empirical evidence) that these metrics will perform well on all datasets. Therefore, we also investigate how to integrate zero-cost metrics within existing NAS algorithms such as reinforcement learning (RL) (Zoph & Le 2017), aging evolution (AE) search (Real et al. 2019) and predictor-base search DDuzziak et al. 2020). More specifically, we investigate enhancing these search algorithms through either (a) zero-cost warmup phase (b) zero-cost move proposal.

---

Published as a conference paper at ICLR 2021
74.
74.5
73.5
73.5
13.0
13.0
72.0
720
RAND
(103)
RAND worms 112001
RL warmu 110201
RAND warmu 130201
RL warmu 130201
RAND warmup (15525)
RL wammu (15925)
71.0
71.0
50
100
150
250
300
50
1.00
150
250
300
Trained models
Trained models
(a) Random Search
(b) Reinforcement Learning
74.5
74.5
23.0
13.0
Aug. best
22.0
22.4
(10201
(3000)
50 (25.)
wammu (15525)
5P warmu 1512)
71.0
50
100
150
200
250
300
50
100
150
250
300
Trained modes
Trained models
(c) Aging Evolution
(d) Binary Predictor
Figure Search speedup with the synfl ow zero-cost proxy on NAS-Bench-20 CIFAR-100.
5.1 ZERO-COST WARMUP
using the zero-cost proxies the beginning of the search process to initialize the search algorithm without training any models or using accuracy. The main parameter in zero-cost warmup is the number of models for which compute and the zero-cost metric (N), and the potential gain comes from the fact that this number can be usually much larger than the number of models
Generally speaking, by warmup
afford train (T < N).
Aging Evolution We score random models with our proxy metric and choose the ones ranked highest as the initial population (pool) in the aging evolution (AE) algorithm (Real et al. 2019).
Reinforcement Learning In the REINFORCE algorithm (Zoph Le 2017) ) we sample N dom models and use their zero-cost scores to reward the controller, thus biasing it towards selecting architectures which for the controller is calculated by linearly normalizing values returned by the proxy functions to the 1, 1] (with online adjustment of min and max).
likely have higher values of the chosen metrics. During warmup, reward
range [-1,
Binary Predictor We warm up a binary graph convolutional network (GCN) predictor from Dudziak al. (2020) by training it to predict relative performance of models by considering their zero-cost scores instead of accuracy. For N warmup points, we use the relative rankings (ac- cording the zero-cost metric) of all pairs of models (0.5.( (N-NNNN11 11) pairs) when performing warmup training for the predictor. As in (Dudziak et al., 2020), models ranked by the predictor after each training round (including the warmup phase) and the top models are evaluated.
5.2 ZERO-COST MOVE PROPOSAL
Whereas warmup tries to leverage global correlation of the proxy metrics to the accuracy of mod- els. move proposal focuses on local neighborhood each step. A common parameter for move proposal algorithms is denoted as R and means sample ratio, i.e., how many models can be checked using zero-cost metrics each time we select model train.
The algorithm is enhanced by performing "guided" mutations. More specif- ically, each time model is being mutated (in the baseline algorithm this is done randomly)
Aging Evolution

---

Published as a conference paper at ICLR 2021
RELD
21.
1ALD nurra
(811
21
21.4
21.6
22.4
21.4
Aep.
330
270
1.0
Trained
Tratned
(a) Random Search
(b) Aging Evolution
(c) Reinforcement Learning
Figure 5: Search speedup with the synflow zero-cost proxy on
TIMIT
consider all possible mutations with edit distance 1 from the current model, score them using the zero-cost proxies and select the best one to add to the pool.
Reinforcement Learning In the of REINFORCE, move proposal is similar to warmup instead of rewarding a controller N time before the search begins, we interleave R zero-cost rewards for each accuracy reward (R < N).
5.3 RESULTS
For all NAS experiments, we repeat experiments 32 times and we plot the median and shade between the lower/upper quartiles. Our baselines already heavily tuned and achieve the same better results than those reported in the original BAS-Bench-101/20) papers. When adding zero-cost warmup or move proposal with synflew, leave all search hyper-parameters unchanged.
NAS-Bench-201 The global/top-10% rank correlations of synflow for this dataset (0.76/0.4p) so we expect this proxy to perform quite well. Indeed, as Figure and Table 7 show, we improve search speed on all four types of searches using zero-cost warmup and move proposal. RAND and RL are both significantly improved, both in terms of sample efficiency and final achieved accuracy. But even more powerful algorithms like AE and BP exhibit 5.6x and 2.3x speedups re- spectively to arrive at 73.5% accuracy. Generally, the more zero-cost warmup, the better the results. This holds true for all algorithms except RL which degrades at 15k warmup points, suggesting that the controller is overfitting to the synflow metric instead of learning to optimize for accuracy.
NAS-Bench-101 This dataset is an order of magnitude larger than NAS-Bench-200 and has lower global/top-100 10% rank correlations of (0.37/0.14) In many ways, this provides true test as to whether these lower correlations are still useful with 7ero-cost warmup and move proposal. Table shows summary of the results and Figure 7 (in Section A.6 shows the full plots. As the table shows, even with modest correlations. there major boost all searching algorithms thus outperforming the best previously published result by a large margin and setting a new state-of-ther dataset. However, worth noting that the binary predictor exhibits no improvement (but also no degradation). Perhaps this is because it was already very sample-efficient and synflov warmup couldn't help further due to its relatively poor correlation on this dataset.
result on this
Table 3: Comparison to prior work on NAS-Bench- 101 dataset.
Wen et al. 2019
Wei el al.
Dudziak
Ours
2020
et al.
2020
RL+M (100)
AE+W (15k)
RAND+W (3k)
# Trained Models
256
50
Test Accuracy [%]
94.17
94.14
94.22
94.22
94.22
94.22
NAS-Bench-ASR We repeat our evaluation on NAS-Bench-ASR with global/top- 10% correla- tions (0.40/0.40) 40). Even though this is different task (speech recognition), synflow warmup and move proposal both yield large improvements in search speeds compared to all baselines in Figure and Table 8 For example, to achieve phoneme еитог rate (PER) of 21.3%, baseline RAND and RL required > 1000 trained models, and AE required 138 trained models; however, this is reduced to 68, 173 and 87 trained models with 2000 models of zero-cost warmup.
8

---

Published as a conference paper at ICLR 2021
DISCUSSION
In this section investigate why zero-cost NAS is effective in improving the sample efficiency of NAS algorithms by looking more closely at how top models are selected by the synflow ow proxy.
Warmup Table 4 shows the number of top-5% most-aceurate models ranked within the top 64 models by the synflow metric. If we compare random warmup versus zero-cost warmup with synflow, random warmup will only return 5% or -3 models out of 64 that are within the top 5% of models whereas syw warmup returns a higher number of top-5% models as listed in T This is key to the improvements observed when adding zero-cost warmup to algorithms like random search or AE. For example, with AE, the numbers in Table 4 are indicative of the models that may end up in the initial AE pool. By initializing the AE pool with many good models, it becomes more likely that a random mutation will lead to an even better model, thus allowing the search to find top model more quickly. Note that synfl fonloww ow is able rank many good models in its top 64 models even when global/local correlation is low (as it is the case for
Table 4: Number of top-5% most-accurate models within the top 64 models returned by synflow.
NAS-Bench-201
NAS-Bench 10--Bench-10)
NAS-Bench-ASD
CIFAR-10
CIFAR-100
ImageNet16-120
44
54
56
12
16
Move Proposal For a search algorithm like AE, search moves consist of random mutations (with edit distance I for our experiments) for a model from the AE pool. Zero-cost move proposal en- hances this by trying out all possible mutations and selecting the best one according to synflow. To investigate how this improves search efficiency, we took 1000 random points and explored their local neighbourhood cluster of possible mutations. Table[]]shows the probability that the synflow proxy correctly identifies the top model. Indeed, synfl improves the chance of selecting the best mutation from -4% to >30% for NAS-Bench-
and 12% for NAS-Bench-101. Even for NAS-Bench-ASR a random mutation has a 7.7% chance (= 1 1/13) to select the best mutation, but this increases to 10% with the synflow proxy thus speeding up convergence to top models.
Table 5: For 1000 clusters of models with edit distance I, we empirically measure the probability that the synflov proxy will select the most accurate model from each cluster.
NAS-Bench-201
NAS- Bonch-
NAS- Bench-ASR
CIFAR-I0 CIFAR-100 ImageNee1661100
Top Model Match
32%
35%
33%
12%
10%
Average Cluster Size
25
25
25
26
13
CONCLUSION
In this paper, introduced six zero-cost proxies, mainly based on recent pruning-at-initaalznn at-initialization work, that are used to rank neural network models in NAS. First, we compared to conventional prox- ies (EcoNAS) that perform reduced-computatio as synflom ow can outperform EcoNAS in maintaining rank consistency. Next, we verified our zero- cost metrics on four additional datasets of varying sizes and tasks and found that indeed out of the six initially-considreed zero-cost metrics, only synflos fow was robust across all datasets for both global and top- 10% rank correlation. Finally, we proposed two ways to integrate synf within NAS algorithms: zero-cost warmup and zero-cost move proposal. Both methods demonstrated significant speedups across four search algorithms and three NAS benchmarks, setting new state-of-the-ar re- sults for both NAS-Bench- 101 and NAS-Bench-201 datasets. Our strong and consistent empirical results suggest that the synflow metric, when combined with warmup and move proposal can be an effective and reliable methodology for speeding up different NAS algorithms. We hope that our work lays a foundation for further zero-cost techniques that expose favourable model proper- ties with little computation thus making NAS more readily accessible without exorbitant computing resources. The most immediate open question for future investigation is why the synflow proxy works well analytical insights will enable further research in zero-cost NAS proxies.
training and we found that zero-cost proxies such

---

Published as a conference paper at ICLR 2021
REFERENCES
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Under- standing and simplifying one-shot architecture search. In International Conference Machine Learning (ICML), 2018.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
Wayne W. Daniel. Applied Nonparametric Statistics. Boston: PWS-Kent, 1990.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale
Hierarchical Image Database. In CVPR, 2009.
Xuanyi Dong and Yi Yang. NAS-Bench-201: Extending the Scope of Reproducible Neural Archi- tecture Search. In International Conference on Learning Representations (ICLR), 2020.
Eukasz Dudziak, Thomas Chau, Mohamed S. Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D. Lane. BRP-NAS: Prediction-based NAS using GCNs. Neural Information Processing Systems (NeurIPS), 2020.
Mikhail Figurnov, Aizhan Ibraimova, Dmitry P Vetrov, and Pushmeet Kohli. Perforatedenns: Accel- eration through elimination of redundant convolutions. In Neural Information Processing Systems (NeurIPS), 2016.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR), 2019.
John S. Garofolo, Lori Lamel, William M. Fisher, Jonathan G. Fiscus, David S. Pallett, Nancy L. Dahlgren, and Victor Zue. Timit acoustic phonetic continuous speech corpus. Linguistic Data Consortium, 1993. URL https: /attalttl.oo.ooo.ooo.ooo 1dc. upenn www.dcpw.......uuuuuu.m edd.LLC3ddLLDD
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Neural Information Processing Systems (NeurIPS), 2015.
Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems 5, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Computer Vision and Pattern Recognition (CVPR), June 2016.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for 2017.
Mobile Vision Applications. arXiv preprint arXiv:
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Computer Vision and Pattern Recognition (CVPR), 2017.
Nikita Klyuchnikov, Ily Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, and Evgeny Burnaev. Nas-bench-nlp: Neural architecture search benchmark for natural language processing. arXiv preprint arXiv:2006.077 16, 2020.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images, 2009. URL https: //www. cs. www.ttorooo..o..o
wwww.cs.torooto.oo. edw. wedd/ddriizie cifar html
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. Neural Information Processing Systems (NeurIPS), 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Tom. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations (ICLR), 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In
International Conference Learning Representations (ICLR), 2019.
10

---

Published as a conference paper at ICLR 2021
Abhinav Mehrotra, Alberto Gil Ramos, Sourav Bhattacharya, Lukasz Dudziak, Ravichander Vip- perla, Thomas Chau, Mohamed S. Abdelfattah, Samin Ishtiaq, and Nicholas D. Lane. NAS- Bench-ASR: Reproducible Neural Architecture Search for Speech Recognition. In International Conference on Learning Representations (ICLR), 2021.
Jieru Mei, Yingwei Li, Xiaochen Lian, Xiaojie Jin, Linjie Yang. Alan Yuille, and Jianchao Yang- AtomNAS: grne-grained end-to-end neural architecture search. In International Conference Learning Representations (ICLR), 2020.
Joseph Mellor, Jack Tumner, Amos Storkey, and Elliot J. Crowley. Neural architecture search without training. arXiv preprint 0et:::2000044447, 2020.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruming convolutional neural networks for resource efficient inference. In International Conference on Learning Repre- sentations (ICLR), 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. In NeurlPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Barak A. Pearlmutter. Fast Exact Multiplication by the Hessian. Neural Computation, 1993.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing, 2018
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized Evolution for Image Classifier Architecture Search. In AÁAI Conference Artificial Intelligence (AAAT), 2019.
Oleg Sémery. PyTorchCV Convolutional neural networks for computer vision, August 2020. URL thub. casam//smr//// /1 mgc smob]
httpss: /tttbb..............//../.....bt
Mingxing Tan and Quoc Le. EfficientNet:
Rethinking model scaling for convolutional neural net-
works. In International Conference on Machine Learning (ICML). 2019a.
Mingxing Tan and Quoc V. Le. Mixconv: Mixed depthwise convolutional kernels, 2019b.
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. arXiv preprint arXiv:2006.05.. 0.66006005467.... 2020.
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning. arXiv:
2018.
Jack Turner, Elliot J. Crowley, Michael O' O'Boyle, Amos Storkey, and Gavin Gray. Blockswap: Fisher-guided block substitution for network compression on budget. International Confer- ence on Learning Representations (ICLR), 2020.
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by
preserving gradient flow. In International Conference on Learning Representations (ICLR), 2020.
Chen Wei, Chuang Niu, Yiping Tang, and Jimin Liang. NPENAS: Neural predictor guided evolution for neural architecture search. arXiv 2003. 03 2020.
Wei Wen, Hanxiao Liu, Hai Li, Yiran Chen, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. arXiv: 2019.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS- bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning (ICML), 2019.
Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. Econas: Finding proxies for economical neural architecture search. In Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. Interna- lional Conference on Learning Representations (ICLR), 2017.
11

---

Published as a conference paper at ICLR 2021
A APPENDIX
Because this paper is empirically-driver there are many results than what presented in the main text of the paper. the appendix we list many important results that support our main arguments and hypotheses in the main text of this paper.
A.1 EXPERIMENTAL DETAILS
In Table 6 we list the hyper-pasn parameeetss used in training the EcoNAS proxies to produce Figure | The only difference to the standard NAS-Bench- training pipeline (Dong & Yang 2020) is our use of fewer epochs for the learning rate annealing schedule we anneal the learning rate to zero over 40 epochs instead of 200. This is common technique used in speeding up convergence for training proxies Zhou et al. (2020). We acknowledge that slightly better correlations could have been achieved for econas and econas+ proxies in Figure if the learning rate was annealed to zero over fewer epochs (20 and 15 epochs respectively). However, we do not anticipate the results to change significantly.
Table EcoNAS training hyper-paramette for
optimizer
SGD
initial LR
0.1
Nesterov
V
final LR
momentum
0.9
LR schedule
cosine
weight decay
0.0005
epochs
random flip random crop
(p=0.5)
batch size
256
One additional comment regarding Figure Tin the main paper. While we run the training ourselves for all EcoNAS variants in the plot, take the data for the line labeled baseline directly from the NAS-Bench-201 dataset. We are not sure why the line is not smooth like the lines for the EcoNAS variants that trained but assume that this NAS-Bench-201 dataset. In any case, we do not anticipate that this would change any conclusions or observations that draw from this plot.
artifact of averaging
multiple seeds in the
Finally, we would like to note some details about our NAS experiments in Section 5 provide multiple seeds of results for each model, whenever "train" model, query random seed from the database to mimic a real NAS pipeline without caching. We refer the reader (Dudziak al. 2020), specifically Section S3.2 for more details this.
NAS datasets
A.2 GPU RUNTIME FOR ECONAS
sioure& the speedup of different EcoNAS proxies compared baseline training. Even though C4 has 64 less computation compared to ra2C16. it achieves maximum of 4x real speedup even when the batch size increased.
4.0
3.5
Speedup
r8c4
2.5
r32c16
2.0
1.5
1.0
256
512
1024
2048
Batch Size
Figure 6: Higher batch sizes when training econas proxies have diminishing returns in terms of measured speedup. This measurement is done for 10 randomly-sampled NAS-Bench-201 models on the CIFAR-10 dataset.
12

---

Published as a conference paper at ICLR 2021
A.3 TABULATED RESULTS
This subsection contains tabulated results from Figures 5 facilitate comparisons with future work. Tables and 8 highlight important data points about the NAS searches we conducted with NAS-Bench-2D 201 and NAS- NAS-Bench-ASh respectively. We highlight results in two ways: First, show the accuracy of the best model found after 50 trained models. Second, we indicate the number of trained models needed for each search method to reach a specific accuracy (73.5% CIFAR- 10 classification accuracy for NAS-Bench-22 and 21.3% TIMIT PER.) We colour the best results (red) and the second best (blue) results each table.
Table 7: Zero-cost NAS comparison with baseline algorithms show accuracy after trained models and the number of models to reach 73.5% accuracy.
LIFAR-100. We
Warmup
Move
Baseline
1000 (BP=256)
3000 (BP=512)
15k
10
100
RAND
71.31 1000+
72.98 1000+
73.18 / 1000+
73.75 /
RL
71.08 1000+
72.76 145
73.14 84
73.21 / 107
71.16/289
7 70
AE
71.53 139
72.91 115
73.40
73.63/25
71.3
BP
72.74/93
73.32 66
73.85 40
Table Zero-cost NAS comparison with baseline algorithms on WAS-Bench-ASR, We show PER after 50 trained models and the number of models to reach PER-21.3%
Warmup
Move
Baseline
500
2000
10
100
RAND
21.65 1000+
21.38
1000+
21.35
RL
21.66 1000+
21.48
1000+
21.45 173
21.62/ 169
21.43/ 161
AE
21.62 138
21.40 115
21.36/87
21.74/ 112
A.4 ANALYSIS OF THE TOP 10% OF MODELS
In the main we pointed the fact that only synflow achieves consistent rank correlation for the 10% of models across different datasets. Here, in Table / we provide the full results. Additionally, we hypothesized that a successful metric will rank many of the most-accurate models in its top models. In [ablle[0[we enumerate the percentage of top- 10% most accurate models ranked as top-10% 10% by each proxy metric. Again, synfl ow the only consistent metric for all datasets, and performs best on average.
Table 9: Spearman p of zero-cost proxies for the top 10% points on all NAS search spaces.
grad_norm snip
grasp
fisher
synflow
jacob_cov
NB2-CIFAR-10
-0.38
-0.38
-0.37
-0.38
0.18
0.17
NB2-CIFAR-100
-0.09
-0.09
-0.11
-0.16
0.42
0.08
HBB-ImaaeeNe116-120
0.13
0.13
0.10
0.02
0.55
0.05
NAS-Bench-101
0.05
-0.01
-0.01
0.07
0.14
0.08
NAS-Bench-NLP
-0.03
-0.02
0.04
0.10
0.04
MAS-Bench-ASR
0.25
0.13
-0.07
0.40
-0.03
A.5 ANALYSIS OF WARMUP AND MOVE PROPOSAL
This section provides more results relevant to our discussion in Section Table II shows the number of top-5% models ranked in the top 64 models by each metric. This is extension to Table in the main text that only shows the results for fl As shown in the table, syn flow is the most powerful metric that we tried.
Table 12 shows the rank correlation coefficient of models within 1000 sampled local clus- ters of models. This result highlights that both grad_norm and jacob_cov work well in dis- tinguishing between very similar models. However, synflow still consistently the best metric in this analysis. Furthermore, measure the percentage of times that metric correctly predicts the
13

---

Published as a conference paper at ICLR 2021
Table 10: Percentage of top-10% most-accurate models within the top-10% of models ranked by each zero-cost metric.
grad_norm
snip
grasp
fisher
synflow
jacob_cov
NB2-CIFAR-10
30%
31%
30%
5%
46%
25%
NB2-CIFAR-I00
35%
36%
34%
4%
50%
24%
1B2-ImmaeNNer16112
31%
31%
32%
5%
44%
30%
MAS-Bench-100
2%
3%
26%
3%
23%
2%
NAS- MAS--Bncc--NP
10%
10%
4%
22%
38%
MMS-Bench-ASR
0%
0%
0%
15%
46%
top model within local cluster of models in Table 13 This is an extension Table the main text. The results are averaged over 1000 pandomly-samplles local clusters. Again, synflow has the highest probability of selecting the top model compared other zero-cost metrics.
Table 11: Number of top-5% most-accurate models within the top-64 models returned by each metric.
grad norm
snip
grasp
fisher
synflow
jacob_cov
NB2-CIFAR-10
0
0
0
44
15
NB2-CIFAR-100
4
4
0
54
16
NB2-ImageNet116
13
13
14
0
56
15
0
0
6
0
12
0
NAS-Bon TAAS-Bench-ASR
0
1
13
Table 12: Rank correlation coefficient for the local neighbourhoods (edit distance of 1000 clusters in each search space.
grad norm
snip
grasp
fisher
synflow
jacob
NB2-CIFAR-10 NB2-CIFAR-100
0.51
0.51
0.37
0.37
0.66
0.62
0.58
0.58
0.44
0.41
0.69
0.61
HB82-mmagNNet16-122
0.56
0.57
0.5
0.4
0.67
0.61
0.23
0.21
0.44
0.27
0.36
0.37
BAS-Bench-ASR 0.59
0.4
0.56
0.38
0.28
Table For 1000 clusters of points with edit distance 1. count the number of times wherein the top model returned by zero-cost metric matches the top model according to validation accuracy.
This represents the probability that zero-cost move proposal will perform the best possible mutation.
grad norm
snip
grasp
fisher
synflow
jacob_cov
NB2-CIFAR-10
14.8%
14.8%
12.7%
5.7%
32.2%
14.5%
NB2-CIFAR- 100
19.1%
18.5%
14.2%
6.0%
35.4%
13.8%
NB2-ImageNNe116112
17.5%
18.5%
15.7%
5.5%
33,4%
16.7%
NAS-Bench-101
0.4%
0.9%
7.4%
0.5%
12.3%
0.5%
NAS-Bench-ASR
11.0%
9.8%
10.3%
10.3%
10.5%
14

---

Published as a conference paper at ICLR 2021
A.6 NAS-BENCI-101 SEARCH PLOTS
Figure shows the NAS search curves for all considered algorithms NAS-Bench-101 dataset.
Important points from this plot are summarized in Table 3 in the main text.
94.4
94.4
Avg. best test accuracy [%6]
Avg. best accuracy
94.0
94.0
93.8
93.8
93.6
93.6
RAND
AE move
RAND warmup (1000)
AE warmup (1000)
93.2
RAND warmup (3000)
93.2
AE warmup (3000)
RAND warmup (15000)
AE warmup (15000)
93.0
93.0
50
100
150
200
250
50
100
150
200
250
300
Trained models
Trained models
(a) Random Search
(b) Aging Evolution
94.4
94.4
Avg. best Tast accuracy [%6]
Avg. best tast accuracy [96]
94.0
94.0
93.8
93.8
RL
93.6
RL + move (10)
93.6
RL + move (100)
RL + warmup (1000)
93.2
RL + warmup (3000)
93.2
BP
RL + warmup (15000)
BP warmup (512)
93.0
93.0
50
100
150
200
250
300
50
100
150
200
250
300
Trained models
Trained models
(c) Reinforcement Learning
(d) Binary Predictor
Figure 7: Search speedup with the synflow zero-cost proxy NAS-Bench-10 CIFAR-10.
15

---

Published as a conference paper at ICLR 2021
A.7 SENSITIVITY ANALYSIS
We performed some sensitivity analysis to investigate how the zero-cost metrics perform on all points within NAS-Bench-201 with different initialization seed, initialization method and minibatch size. We comment on each table in its caption; however, to summarize, all metrics seem be relatively unaffected when initialization and minibatch size are varied. The one exception can be seen in Table L5 where sher benefits when biases are initialized with zeroes.
Table All metrics remain fairly constant when varying the initialization seed the variations are only observed at the third significant digit. Dataload is random with 128 samples and initialization done with default PyTorch initialization scheme.
seed
grad morm
grrd
snip
grasp
fisher
synflow
jacob.coy
0.578
0.581
0.487
0.361
0.737
0.735
0.580
0.583
0.488
0.354
0.740
0.728
0.582
0.584
0.486
0.358
0.738
0.726
0.581
0.584
0.491
0.356
0.738
0.73
0.581
0.583
0.486
0.356
0.738
0.727
Average 0,580
0.583
0.488
0.357
0.738
0.729
Table 15: fi fisher becomes noticeably better when biases are initialized zero; otherwise, metries seem to perform independently of initialization method. Results averaged over 3 seeds.
Weights init
Bias init
grad norm
snip
grasp
fisher
synflow
jacob cov
default kaiming
default
0.580
0.583
0.488
0.357
0.738
0.729
default
0.548
0.558
0.364
0.332
0.731
0.723
xavier default
0.543
0.568
0.424
0.345
0.736
0.729
default
zero
0.581
0.583
0.488
0.509
0.738
0.729
kaiming
zero
0.542
0.551
0.370
0.479
0.730
0.723
xavier zero
0.540
0.566
0.412
0.495
0.735
0.730
Table 16: Surprisingly, grasp becomes worse with more (random) data, while grad_nor and snip degrade very slightly. Other metrics seem to perform independently number of samples in the minibatch. Initialization is done with default PyTorch initialization scheme.
Number of Samples
grad_norm _nom
snip
grasp
fisher
synflow
jacob_cov
32
0.595
0.596
0.511
0.362
0.737
0.732
64
0.589
0.59
0.509
0.361
0.737
0.735
128
0.578
0,581
0.487
0.361
0.737
0.735
256
0.564
0.569
0.447
0.361
0.737
0.731
512
0.547
0.552
0.381
0.361
0.737
0.724
A.8 RESULTS FOR ALL ZERO-COST METRICS
Here we provide some NAS search results using all considered metrics for both RAND and AE searches on WAS-Bench- 101/201 datasets. Our experiments point to synflow as the only effective zero-cost metric across different datasets; however, provide the plots below for the reader to inspect how poorer metrics perform NAS.
16

---

Published as a conference paper at ICLR 2021
92.0
91.5
91.5
91.0
91.0
00.5
90.5
R450
A6
90.0
R4b0 warmu (5kil
80.0
wampp (rl))
(gras)
igrass
(lahe?)
(iahut)
ig
(boo0_cow)
wammu (bcob_cool
50
100
150
200
250
300
50
100
150
200
250
360
Trained models
Trained madels
(a) NAS Bench- 201 CIFAR-10
73.5
73.5
73.0
73.0
72.5
72.5
RARI
Aug. best
72.0
wamme
wemmup
1gras) (ishel
(ishut)
50
100
150
200
250
300
50
100
150
200
250
300
Trained models
Trained models
(b) Bench-2002 CIFAR-
47.0
47.0
[36]
46.5
46.5
46.0
46.0
45.5
45.5
RARI
50
100
150
200
250
300
50
100
150
200
250
300
Trained models
Trained models
(c) NAS-Bench-200 ImageNeet66112)
best [%6]
94.2
94.0
94.0
93.8
93.8
RAND
best
A2 wampp enlp)
R/KiD wammn
93.4
(gree))
93.4
wammp igrses)
[ttber)
(hher)
932
warmo igrad nerm) wamus (jacob.cov) cow)
93.2
wammu igras
(syritow)
wammp (jacob covl
50
100
150
200
250
300
50
100
150
200
250
300
Trained
Trained models
(d) NAS NAS-NANSheetee Bench-101 101 CIFAR-10
Figure 8: Evaluation of all zero-cost proxies different datasets and search algorithms: random search (RAND) and aging evolution (AE). RAND benefits greatly from a strong metric (such as synflow) but may deteriorate with a weaker metric as shown in the plot. However, AE benefits when a strong metric is used and is resilient to weaker metrics as well it is able to recover and achieve the top accuracy in most cases.
17


Reviews:
[{"review_id": "wDJZbKKki2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nTo reduce the cost of NAS, this paper focuses on zero-cost proxies to estimate the performance of network architectures without any training.\n\nSpecifically, the authors propose a series of zero-cost proxies based on recent pruning literature. They also propose two practical strategies: zero-cost warmup and zero-cost move, to integrate zero-cost proxies into existing search algorithms (e.g., random search, RL, evolution). \n\nExtensive experiments and analysis on NAS-Bench-101/201/NLP/ASR validate the usefulness of the proposed zero-cost proxies. Integrating zero-cost proxies into existing search algorithms significantly improves the sample efficiency.\n\n\n# Strong points\n1. Designing zero-cost proxies for NAS is a promising direction to reduce its cost. This paper evaluates a series zero-cost proxies and demonstrates their usefulness.\n2. The idea of zero-cost warmup and zero-cost move is very appealing. Only using zero-cost proxies might not achieve competitive performance, but this idea provides a practical way of using these zero-cost proxies that will yield nice performance and a great reduction in search cost.\n3. The experiments and analysis in this paper are extensive and solid, which provides many insights for the community to understand these zero-cost proxies.\n\n\n# Weak points\n1. “with synflow we compute a loss which is simply the product of all parameters in the network”. Why synflow using this loss? To make the paper self-contained, it will be helpful to give more clarification on this.\n\n  Small comment: Explicitly explaining the Hadamard product notation is also helpful.\n\n2. For some zero-cost proxies, we need to pass a minibatch of data to the network. Is the proxy sensitive or not to the choice of the minibatch? I am not sure if the “128 samples” are fixed or not in Table 4.\n\n3. Why can r16c8 outperform the full training baseline r32c16? This seems to be counter-intuitive. Any explanation for this?\n\n  Also, how many models are used in Figure 1, all the 15625 models in NAS-Bench-201?  It seems you will have to re-train the models under different configurations.\n\n4. For econas+ (r16c8e15), what’s the learning rate schedule: (1) anneal to 0 (or a small number) at epoch 15, or (2) still follows the LR schedule of 200 epochs? The same question applies to other points in Figure 1. The first way (anneal to 0 at epoch 15) is more common and usually the better choice. It helps people to understand how strong this baseline is if you can confirm the implementation details in this part.\n\n4. Figure 3 needs to be better organized to group the results for the same dataset together. It’s hard to compare different proxies in this form.\n\n  The text says that snyflow is the best. But I notice that snip and fisher are actually slightly better than synflow, if we ignore SVHN, which is less important than ImageNet I think.\n\n# Justification of rating\nThe zero-cost proxies used in this paper are mostly inspired by recent pruning literature. So, I consider this paper as more like an analysis paper that evaluates these known metrics for zero-cost proxies. But I need to point out that these known metrics were not designed for NAS initially. So, this paper provides new insights to the community.\n\nI generally feel positive about this work due to its extensive analysis and the nice idea of zero-cost warmup/move. Most of the weak points mentioned above are for clarification. I also encourage the authors to further refine the figures and writing to make this paper better.\n\n# Additional comments\n\n1. Figures1&2 are blurred. The visualization style needs to be improved to make it easier to interpret.\n\n2. Figure 6 y-axis is labeled as “normalized training time”. But from the text, the y-axis label should be speedup?\n\n3. Table 2, the meaning of boldface is unclear.\n\n# After rebuttal\n\nThe authors did a good job of answering my concerns. I keep my original positive rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "nbMvUgPGNEf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, this is a solid work with interesting ingredients. The main contributions are mostly in terms of empirical results rather than a new search method.\n\nStrengths\n* The paper is very well-written and easy to follow.\n* The work presented simple yet effective ways to combine zero-shot proxies with existing NAS methods: (1) to initialize standard NAS algorithms, or (2) to guide the search process. Both approaches are interesting and have well demonstrated that zero-shot proxies are complementary to existing methods.\n* The paper has provided extensive evaluations over different metrics, search spaces and datasets. The results are presented in clear manner and are fairly convincing.\n\nWeaknesses\n* The technical novelty is limited in the sense that (1) the idea of zero-shot NAS is not new (Mellor et al., 2020), and (2) the best zero-shot metric (synflow) is a straightforward application of the existing work.\n* Results for the rank correlation for top 10% models (A.3) are a bit disappointing because it seems to suggest zero-shot proxies are not able to identify high-performance architectures when used *alone* without leveraging other NAS methods, which are usually more expensive.\n* While the method provides speedup at the early-stage, it does not seems to offer gains on the high-performance region on top of strong methods (e.g., predictive methods in Table 3).", "extra_scores": {}, "extra_text": {}}, {"review_id": "LNF1PIQEANg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "############################################################\n\nSummary\n\nThis paper provides an extensive empirical evaluation of zero-cost proxies which can be combined with existing NAS methods to speed up search time. The proposed method utilizes ‘pruning-at-initialization’ works which computes gradient-computation at initialization as a proxy for performance of the given neural architectures. Through extensive experiments, this paper compares between conventional proxies and ablation study on five NAS benchmarks and shows the validation of the proposed proxy. \n \n############################################################\n\nReasons for score\n  \nOverall, I vote for weak rejecting. I like the idea of zero-cost proxies which decides the performance of neural architectures at initialization and the high utility of the zero-cost proxies available combining it with various NAS methods. However, my major concern is the limitation of zero-cost proxies where the correlations between the top 10% performance neural architectures and the values estimated by zero-cost proxies look low to use as a proxy for searching for the best neural architectures.  Hopefully, the authors can address my concern in the rebuttal period.  \n  \n############################################################\n\nStrong Points \n  \n1. This paper provides zero-cost proxies that can estimate the performance of the given neural architectures at the initialization step saving time and resources.\n2. The idea of this paper is simple yet effective, where this paper is inspired by pruning-at-initialization methods and adopt those methods as proxies for NAS methods.\n3. This paper demonstrates the effectiveness through extensive experiments. It first performs evaluations among several zero-cost proxies candidates to select the best one, synflow and then combines synflow and conventional NAS methods to conduct experiments on various NAS-benchmark datasets.\n\n############################################################\n\nWeak Points\n\n1. My main concern is that zero-cost proxies may be not enough to use for searching the best neural architecture. The correlation between synflow which is the best one of zero-cost proxies and neural architectures for CIFAR10 from NAS201-Bench201 search space is 0.74 (Table 1). However, the correlation drops to 0.18 (Table 7 of Appendix) when we compute it between synflow, and neural architectures which have top 10% ranking performance on CIFAR10 from NAS201-Bench201 search space (Table 7). In case NAS-Bench-ASR, it drops to 0.03 (Table 7) which is nearly random. I believe while zero-cost proxies are suitable for estimate the performance tendency of all neural architectures from the search space, they are needed to be improved for the problems which search the best neural architectures. During the period, I hope the authors address my concern by performing additional experiments. \n2. I think this paper has the property of the benchmark paper since this paper brings several zero-cost proxies applying it to existing NAS rather than designs a novel proxies. Thus, I expect that this paper shows more usecases combining zero-cost proxies with conventional NAS methods to demonstrate the usefulness of zero-cost proxies.  \n3. I recommend the results of cifar10 and imagenet120 of NAS-Bench 201 move to the original paper from the appendix. \n4. Could the author shows the results of the Figure 4 as the form of the Table with exact values by selecting the results?\n5. Clarity of the paper looks needed. For example, \n\n      (1) missing reference or model name\n\n      (2) dataset description of NAS-Bench 201 of 4.1\n\n      (3) the detailed description of experimental setup of 4.1.2.\n\n      (4) the detailed description of warmup technique. What meaning of (1000) of warmup (1000) in Figure 4.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gea4Pzz_mo2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides an interesting direction in the neural architecture field. In particular, it proposes a series of zero-cost proxies. To design such proxies, the authors evaluate conventional reduced-training proxies and recent pruning algorithms. These proxies use just a single minibatch of training data to compute a model’s score. \nReasons for score:\nOverall, I vote for accepting. On the one hand, I think the idea of utilizing pruning algorithms in NAS is very attractive since it offers a way to avoid the heavy proxy training and evaluation phase. On the other hand, the experiments in this paper are very sufficient and strong. My major concern is about the clarity of the paper. Hopefully, the authors can address my concern in the rebuttal period.\n\nPros:\n1. The paper takes one of the most important issues of NAS: the computation burden in proxy training. For me, the problem itself is real and practical.\n2. The proposed “zero-cost” proxies are very attractive for its extremely cheap computation cost. The cheap computation allows us to explore much large search space, which is of great importance in NAS. Moreover, these proxies require very little memory, which is also very attractive.\n3. This paper provides extensive experiments to evaluate performances of existing pruning algorithms on a vast variety of datasets. The experimental results have a high reference value.\n\nCons:\n1. In Eq. (1), it seems that you use a Hessian Matrix to compute grasp metric. However, the computation of a Hessian Matrix is very expensive. Is it a bug? Or you have some techniques to bypass the computation of the Hessian matrix? It would be better to provide more details here. \n2. For Section 3, it would be better to provide more details about how you aggregate these metrics, which seems not very clear for me.\n3. As far as I know, these techniques are not easy to integrate with some differentiable NAS algorithms, such as DARTS. However, differentiable NAS occupies an important position in all NAS methods. So, I suggest that you can point out this limitation in your paper for the sake of rigor.\n\nSome typos：\n1.  INTRODUCTION： operate at at-> operate at\n2.  3.1 CONVENTIONAL NAS PROXIES (ECONAS): \n in many prior work->  in many prior works\n  One main findings->  One main finding\n3.  5.1 ZERO-COST WARMUP:  \nthe number of models for which we compute -> the number of models for which we compute(T)\n4.  6 CONCLUSION:\n     Compute-> computing\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a well-written paper proposing a promising a series of zero-cost proxies for NAS. Overall, the reviewers were convinced that the approach is sound and the results overall support the use of zero-cost proxies (although they are a bit weak in some cases, e.g. rank correlations in A.3). Despite some concerns amongst the reviewers around the technical novelty of the method, mostly due to the use of estimators from the pruning-at-init literature, this is promising work at the intersection of different sub-communities in ML.
Decision: Accept (Poster)