Zero-Cost Proxies for Lightweight NAS

1 INTRODUCTION . Instead of manually designing neural networks , neural architecture search ( NAS ) algorithms are used to automatically discover the best ones ( Tan & Le , 2019a ; Liu et al. , 2019 ; Bender et al. , 2018 ) . Early work by Zoph & Le ( 2017 ) proposed using a reinforcement learning ( RL ) controller that constructs candidate architectures , these are evaluated and then feedback is provided to the controller based on the performance of the candidate . One major problem with this basic NAS methodology is that each evaluation is very costly – typically on the order of hours or days to train a single neural network fully . We focus on this evaluation phase – we propose using proxies that require a single minibatch of data and a single forward/backward propagation pass to score a neural network . This is inspired by recent pruning-at-initialization work by Lee et al . ( 2019 ) , Wang et al . ( 2020 ) and Tanaka et al . ( 2020 ) wherein a per-parameter saliency metric is computed before training to inform parameter pruning . Can we use such saliency metrics to score an entire neural network ? Furthermore , can we use these “ single minibatch ” metrics to rank and compare multiple neural networks for use within NAS ? If so , how do we best integrate these metrics within existing NAS algorithms such as RL or evolutionary search ? These are the questions that we hope to ( empirically ) tackle in this work with the goal of making NAS less compute-hungry . Our contributions are : • Zero-cost proxies We adapt pruning-at-initialization metrics for use with NAS . This requires these metrics to operate at the granularity of an entire network rather than individual parameters – we devise and validate approaches that aggregate parameter-level metrics in a manner suitable for ranking candidates during NAS search . • Comparison to conventional proxies We perform a detailed comparison between zerocost and conventional NAS proxies that use a form of reduced-computation training . First , we quantify the rank consistency of conventional proxies on large-scale datasets : 15k models vs. 50 models used in ( Zhou et al. , 2020 ) . Second , we show that zero-cost proxies can match or exceed the rank consistency of conventional proxies . • Ablations on NAS benchmarks We perform ablations of our zero-cost proxies on five different NAS benchmarks ( NAS-Bench-101/201/NLP/ASR and PyTorchCV ) to both test the zero-cost metrics under different settings , and expose properties of successful metrics . • Integration with NAS Finally , we propose two ways to use zero-cost metrics effectively within NAS algorithms : random search , reinforcement learning , aging evolution and predictor-based search . For all algorithms and three NAS datasets we show significant speedups , up to 4× for NAS-Bench-101 compared to current state-of-the-art . 2 RELATED WORK . NAS Efficiency To decrease NAS search time , various techniques were used in the literature . Pham et al . ( 2018 ) and Cai et al . ( 2018 ) use weight sharing between candidate models to decrease the training time during evaluation . Liu et al . ( 2019 ) and others use smaller datasets ( CIFAR-10 ) as a proxy to the full task ( ImageNet1k ) . In EcoNAS , Zhou et al . ( 2020 ) extensively investigated reduced-training proxies wherein input size , model size , number of training samples and number of epochs were reduced in the NAS evaluation phase . We compare to EcoNAS in this work to elucidate how well our zero-cost proxies perform compared to familiar and widely-used conventional proxies . Pruning The goal is to reduce the number of parameters in a neural network , one way to do this is by identifying a saliency ( importance ) metric for each parameter , and the less-important parameters are removed . For example , Han et al . ( 2015 ) , Frankle & Carbin ( 2019 ) and others use parameter magnitudes as the criterion while LeCun et al . ( 1990 ) , Hassibi & Stork ( 1993 ) and Molchanov et al . ( 2017 ) use gradients . However , the aforementioned works require training before computing the saliency criterion . A new class of pruning-at-initialization algorithms , that require no training , were introduced by Lee et al . ( 2019 ) and extended by Wang et al . ( 2020 ) and Tanaka et al . ( 2020 ) . A single forward/backward propagation pass is used to compute a saliency criterion which is successfully used to heavily prune neural networks before training . We extend these pruning-at-initialization criteria towards scoring entire neural networks and we investigate their use with NAS algorithms . Intersection between pruning and NAS Concepts from pruning have been used within NAS multiple times . For example , Mei et al . ( 2020 ) use channel pruning in their AtomNAS work to arrive at customized multi-kernel-size convolutions ( mixconvs as introduced by Tan & Le ( 2019b ) ) . In their Blockswap work , Turner et al . ( 2020 ) use Fisher information at initialization to score different lightweight primitives that are substituted into a neural network to decrease computation . This is the earliest work we could find that attempts to perform a type of NAS by scoring neural networks without training using a pruning criterion , More recently , Mellor et al . ( 2020 ) introduced a new metric for scoring neural networks at initialization based on the correlation of Jacobians with different inputs . They perform “ NAS without training ” by performing random search with their zero-cost metric ( jacob cov ) to rank neural networks instead of using accuracy . We include jacob cov in our analysis and we introduce five more zero-cost metrics in this work . 3 PROXIES FOR NEURAL NETWORK ACCURACY . 3.1 CONVENTIONAL NAS PROXIES ( ECONAS ) . In conventional sample-based NAS , a proxy training regime is often used to predict a model ’ s accuracy instead of full training . Zhou et al . ( 2020 ) investigate conventional proxies in depth by computing the Spearman rank correlation coefficient ( Spearman ρ ) of a proxy task to final test accuracy . The proxy used is a reduced-computation training , wherein one of the following four variables is reduced : ( 1 ) number of epochs , ( 2 ) number of training samples , ( 3 ) input resolution ( 4 ) model size ( controlled through the number of channels after the first convolution ) . Even though such proxies were used in many prior works , EcoNAS is the first systematic study of conventional proxy tasks that we found . One main finding by Zhou et al . ( 2020 ) is that using approximately 14 of the model size and input resolution , all training samples , and 110 the number of epochs was a reasonable proxy which yielded the best results for their experiment ( Zhou et al. , 2020 ) . 3.2 ZERO-COST NAS PROXIES . We present alternative proxies for network accuracy that can be used to speed up NAS . A simple proxy that we use is grad norm in which we sum the Euclidean norm of the gradients after a single minibatch of training data . Other metrics listed below were previously introduced in the context of parameter pruning at the granularity of a single parameter – a saliency is computed to rank parameters and remove the least important ones . We adapt these metrics to score and rank entire neural network models for NAS . 3.2.1 SNIP , GRASP AND SYNAPTIC FLOW . In their snip work , Lee et al . ( 2019 ) proposed performing parameter pruning based on a saliency metric computed at initialization using a single minibatch of data . This saliency criteria approximates the change in loss when a specific parameter is removed . Wang et al . ( 2020 ) attempted to improve on the snip metric by approximating the change in gradient norm ( instead of loss ) when a parameter is pruned in their grasp objective . Finally , Tanaka et al . ( 2020 ) generalized these so-called synaptic saliency scores and proposed a modified version ( synflow ) which avoids layer collapse when performing parameter pruning . Instead of using a minibatch of training data and cross-entropy loss ( as in snip or grasp ) , with synflow we compute a loss which is simply the product of all parameters in the network ; therefore , no data is needed to compute this loss or the synflow metric itself . These are the three metrics : snip : Sp ( θ ) = ∣∣∣∣∂L∂θ θ ∣∣∣∣ , grasp : Sp ( θ ) = − ( H∂L∂θ ) θ , synflow : Sp ( θ ) = ∂L∂θ θ ( 1 ) where L is the loss function of a neural network with parameters θ , H is the Hessian1 , Sp is the per-parameter saliency and is the Hadamard product . We extend these saliency metrics to score an entire neural network by summing over all parameters N in the model : Sn = ∑N i Sp ( θ ) i . 3.2.2 FISHER . Theis et al . ( 2018 ) perform channel pruning by removing activation channels ( and their corresponding parameters ) that are estimated to have the least effect on the loss . They build on the work of Molchanov et al . ( 2017 ) and Figurnov et al . ( 2016 ) . More recently , Turner et al . ( 2020 ) aggregated this fisher metric for all channels in a convolution primitive to quantify the importance of that primitive when it is replaced by a more efficient alternative . We further aggregate the fisher metric for all layers in a neural network to score an entire network as shown in the following equations : fisher : Sz ( z ) = ( ∂L ∂z z ) 2 , Sn = M∑ i=1 Sz ( zi ) ( 2 ) where Sz is the saliency per activation z , and M is the length of the vectorized feature map . 3.2.3 JACOBIAN COVARIANCE . This metric was purpose-designed to score neural networks in the context of NAS – we refer the reader to the original paper for detailed reasoning and derivation of the metric which we call jacob cov ( Mellor et al. , 2020 ) . In brief , this metric captures the correlation of activations within a network when subject to different inputs within a minibatch of data – the lower the correlation , the better the network is expected to perform as it can differentiate between different inputs well . 4 EMPIRICAL EVALUATION OF PROXY TASKS . Generally , most of the proxies presented in the previous section try to capture how trainable a neural network is by inspecting the gradients at the beginning of training . In this work , we refrain from attempting to explain precisely why each metric works ( or does not work ) and instead focus on the empirical evaluation of those metrics in different scenarios . We use the Spearman rank correlation coefficient ( Spearman ρ ) to quantify how well a proxy ranks models compared to the ground-truth ranking produced by final test accuracy ( Daniel , 1990 ) . 1The full Hessian does not need to be explicitly constructed as explained by Pearlmutter ( 1993 ) . 4.1 NAS-BENCH-201 . NAS-Bench-201 is a purpose-built benchmark for prototyping NAS algorithms ( Dong & Yang , 2020 ) . It contains 15,625 CNN models from a cell-based search space and corresponding training statistics . We first use NAS-Bench-201 to evaluate conventional proxies from EcoNAS , then we evaluate our zero-cost proxies and compare the two approaches .

Summary: This paper provides an extensive empirical evaluation of zero-cost proxies which can be combined with existing NAS methods to speed up search time. The proposed method utilizes ‘pruning-at-initialization’ works which computes gradient-computation at initialization as a proxy for performance of the given neural architectures. Through extensive experiments, this paper compares between conventional proxies and ablation study on five NAS benchmarks and shows the validation of the proposed proxy.