Zero-Cost Operation Scoring in Differentiable Architecture Search

Differentiable neural architecture search ( NAS ) has attracted significant attention in recent years due to its ability to quickly discover promising architectures of deep neural networks even in very large search spaces . Despite its success , many differentiable NAS methods lack robustness and may degenerate to trivial architectures with excessive parameter-free operations such as skip connections thus leading to inferior performance . In fact , selecting operations based on the magnitude of architectural parameters was recently proven to be fundamentally wrong , showcasing the need to rethink how operation scoring and selection occurs in differentiable NAS . To this end , we formalize and analyze a fundamental component of differentiable NAS : local “ operation scoring ” that occurs at each choice of operation . When comparing existing operation scoring functions , we find that existing methods can be viewed as inexact proxies for accuracy . We also find that existing methods perform poorly when analyzed empirically on NAS benchmarks . From this perspective , we introduce new training-free proxies to the context of differentiable NAS , and show that we can significantly speed up the search process while improving accuracy on multiple search spaces . We take inspiration from zero-cost proxies that were recently studied in the context of sample-based NAS but shown to degrade significantly for larger search spaces like DARTS . Our novel “ perturbation-based zero-cost operation scoring ” ( ZeroCost-PT ) improves searching time and accuracy compared to the best available differentiable architecture search for many search space sizes , including very large ones . Specifically , we are able improve accuracy compared to the best current method ( DARTS-PT ) on the DARTS CNN search space while being over 40× faster ( total searching time 25 minutes on a single GPU ) . Our code is available at : https : //github.com/avail-upon-acceptance . 1 INTRODUCTION . Since the recent dawn of deep learning , researchers have designed new architectures of neural networks on an unprecedented scale , with more efficient and accurate networks being proposed each year ( Iandola et al. , 2016 ; Howard et al. , 2017 ; Tan & Le , 2019 ; 2021 ) . However , the manual design of better DNN architectures has proven quite challenging , requiring extensive domain knowledge and persistent trial-and-error in search of the optimal hyperparameters ( Sandler et al. , 2018 ; Tan & Le , 2021 ) . Recently this process has been successfully aided through automated methods , especially neural architecture search ( NAS ) which can be found behind many of the state-of-the-art deep neural networks ( Real et al. , 2019 ; Wu et al. , 2019 ; Cai et al. , 2020 ; Moons et al. , 2020 ) . However , one of the biggest problems in NAS is the computational cost – even training a single deep network can require enormous computational resources and many NAS methods need to train tens , if not hundreds , of networks in order to converge to a good architecture ( Real et al. , 2019 ; Luo et al. , 2018 ; Dudziak et al. , 2020 ) . A related problem concerns the search space size – a larger search space would typically contain better architectures , but requires longer searching time ( Real et al. , 2019 ) . Differentiable architecture search ( DARTS ) was first proposed to tackle those challenges , showcasing promising results when searching for a network in a set of over 1018 possible variations ( Liu et al. , 2019 ) . Unfortunately , DARTS has significant robustness issues as demonstrated through many recent works ( Zela et al. , 2020a ; Shu et al. , 2020 ; Yu et al. , 2020 ) . It also requires careful selection of hyperparameters which makes it somewhat difficult to adapt to a new task . Recently , Wang et al . ( 2021 ) showed that operation selection in DARTS based on the magnitude of architectural parameters ( α ) is fundamentally wrong , and will always simply select skip connections over other more meaningful operations . They proposed an alternative operation selection method based on perturbation , where the importance of an operation is determined by the decrease of the supernet ’ s validation accuracy when it is removed . Then the most important operations are selected by exhaustively comparing them with other alternatives on each single edge of the supernet , until the final architecture is found . In a parallel line of work that aims to speed up NAS , proxies are often used instead of training accuracy to obtain an indication of performance quickly without performing expensive full training for each searched model . Conventional proxies typically consist of a reduced form of training with fewer epochs , less data or a smaller DNN architecture ( Zhou et al. , 2020 ) . Most recently , zero-cost proxies , which are extreme types of NAS proxies that do not require any training , have gained interest and are shown to empirically outperform conventional training-based proxies and deliver outstanding results on common NAS benchmarks ( Abdelfattah et al. , 2021 ; Mellor et al. , 2021 ) . However , their efficient usage on a large search spaces , typical for differentiable NAS , has been shown to be more challenging and thus remains an open problem ( Mellor et al. , 2021 ) . The objective of our paper is to shed some light onto the implicit proxies that are used for operation scoring in differentiable NAS , and to propose new proxies in this setting that have the potential of improving both search speed and quality . We decompose differentiable NAS into its two constituent parts : ( 1 ) supernet training and ( 2 ) operation scoring . We focus on the second component and we formalize the concept of “ operation scoring ” that happens during local operation selection at each edge in a supernet . Through this lens , we are able to empirically compare the efficacy of existing differentiable NAS operation scoring functions . We find that existing methods act as a proxy to accuracy and perform quite poorly on common NAS benchmarks , consequently , we propose new operation scoring functions based on zero-cost proxies that outperform existing methods in both search speed and accuracy . Our main contributions are : 1 . Formalize “ operation scoring '' in differentiable NAS and perform a first-of-its-kind analysis of the implicit proxies that are present in existing differentiable NAS methods . 2 . Propose , evaluate and compare perturbation-based zero-cost operation scoring ( Zero-Cost-PT ) for differentiable NAS building upon recent work on training-free NAS proxies . 3 . Perform a thorough empirical evaluation of Zero-Cost-PT in six search spaces and 3 datasets including DARTS , DARTS subspaces S1-S4 and NAS-Bench-201 . 2 RELATED WORK . Classic NAS and Proxies . Zoph & Lee were among the first to propose an automated method to search neural network architectures , using a reinforcement learning agent to maximize rewards coming from training different models ( Zoph & Le , 2017 ) . Since then , a number of alternative approaches have been proposed in order to reduce the significant cost introduced by training each proposed model . In general , reduced training can be found in many NAS works ( Pham et al. , 2018 ; Zhou et al. , 2020 ) , and different proxies have been proposed , e.g . searching for a model on a smaller dataset and then transfer the architecture to the larger target dataset ( Real et al. , 2019 ; Mehrotra et al. , 2021 ) or incorporating a predictor into the search process ( Wei et al. , 2020 ; Dudziak et al. , 2020 ; Wu et al. , 2021 ; Wen et al. , 2019 ) . Zero-cost Proxies . Very recently , zero-cost proxies ( Mellor et al. , 2021 ; Abdelfattah et al. , 2021 ) for NAS emerged from pruning-at-initialisation literature ( Tanaka et al. , 2020 ; Wang et al. , 2020 ; Lee et al. , 2019 ; Turner et al. , 2020 ) . Such proxies can be formulated as architecture scoring functions S ( A ) that evaluate the potential or “ saliency ” of a given architecture A in achieving accuracy at initialization , without the expensive training process . In this paper , we adopt the recently proposed zero-cost proxies ( Abdelfattah et al. , 2021 ; Mellor et al. , 2021 ) , namely grad_norm , snip , grasp , synflow , fisher and nwot . Those metrics either aggregate the saliency of model parameters to compute the score of an architecture ( Abdelfattah et al. , 2021 ) , or use the overlapping of activations between different samples within a minibatch of data as a performance indicator ( Mellor et al. , 2021 ) . In a similar vein , Chen et al . ( 2021 ) proposed the use of training-free scoring for operations based on the neural tangent kernel ( Jacot et al. , 2021 ) and number of linear regions in a DNN ; the operations with the lowest score are pruned from the supernet iteratively until a subnetwork is found . Differentiable NAS and Operation Perturbation . Liu et al . first proposed to search for a neural network ’ s architecture by parameterizing it with continuous values ( called architectural parameters α ) in a differentiable way . Their method constructs a supernet , i.e. , a superposition of all networks in the search space , and optimizes the architectural parameters ( α ) together with supernet weights ( w ) . The final architecture is extracted from the supernet by preserving operations with the largest α . Despite the significant reduction in searching time , the stability and generalizability of DARTS have been challenged , e.g. , it may produce trivial models dominated by skip connections ( Zela et al. , 2020a ) . SDARTS ( Chen & Hsieh , 2020 ) proposed to overcome such issues by smoothing the loss landscape , while SGAS ( Li et al. , 2020 ) considered a greedy algorithm to select and prune operations sequentially . The recent DARTS-PT ( Wang et al. , 2021 ) proposed a perturbation-based operation selection strategy , showing promising results on DARTS space . In DARTS-PT operations are no longer selected by optimizing architectural parameters ( α ) , but via a scoring function evaluating the impact on supernet ’ s validation accuracy when the operation is removed . 3 RETHINKING OPERATION SCORING IN DIFFERENTIABLE NAS . In the context of differentiable NAS , a supernet would contain multiple candidate operations on each edge as shown in Figure 1 . Operation scoring functions assign a score to rank operations and select the best one . In this section , we empirically quantify the effectiveness of existing operation scoring methods in differentiable NAS , with a specific focus on DARTS ( Liu et al. , 2019 ) and the recently-proposed DARTS-PT ( Wang et al. , 2021 ) . Concretely , we view these scoring functions as proxies for final subnetwork accuracies and we evaluate them on that basis to quantify how well these functions perform . We challenge many assumptions made in previous work and show that we can outperform existing methods with lightweight alternatives . 3.1 OPERATION SCORING PRELIMINARIES . For a supernet A we want to be able to start discretizing edges in order to derive a subnetwork . When discretizing we replace an edge composed of multiple candidate operations and their respective ( optional ) architectural parameters α with only one operation selected from the candidates . We will denote the process of discretization of an edge e with operation o , given a model A , as : A+ ( e , o ) . Analogously , the perturbation of a supernet A by removing an operation o from an edge e will be denoted as A − ( e , o ) . Figure 1 illustrates discretization and perturbation . Furthermore , we will use A , E and O to refer to the set of all possible network architectures , edges in the supernet and candidate operations , respectively – extra details about notation can be found in the Appendix A.1 . NAS can then be performed by iterative discretization of edges in the supernet , yielding in the process a sequence of partially discretized architectures : A0 , A1 , ... , A|E| , where A0 is the original supernet , A|E| is the final fully-discretized subnetwork ( result of NAS ) , and At is At−1 after discretizing a next edge , i.e. , At = At−1 + ( et , ot ) where t is an iteration counter . The problem of finding the sequence of ( et , ot ) that maximizes performance of the resulting network AE has an optimal substructure and can be reduced to the problem of finding the optimal policy π : A× E → O that is used to decide on an operation to assign to an edge at each iteration , given current model ( state ) . This policy function is defined by means of an analogous scoring function f : A× E ×O → R , that assigns scores to the possible values of the policy function , and the policy is then defined as argmax or argmin over f , depending on the type of scores produced by f . 1 We begin by defining the optimal scoring function that we will later use to assess quality of different , practical approaches . For a given partially-discretized model At , let us denote the set of all possible fully-discretized networks that can be obtained from At after a next edge e is discretized with an operation o as At , e , o . Our optimal scoring function can then be defined as : πbest-acc ( At , e ) = argmax o∈Oe max A|E|∈At , e , o V ∗ ( A|E| ) ( 1 ) 1Since a scoring function clearly defines a relevant policy function , we will sometimes talk about a scoring function even though the context might be directly related to a policy function – in those cases it should be understood as the policy function that follows from the relevant scoring function ( and vice versa ) . where V ∗ is validation accuracy of a network after full training ( we will use V to denote validation accuracy without training ) . It is easy to see that this policy meets Bellman ’ s principle of optimality ( Bellman , 1957 ) – in fact its definition follows directly from it – and therefore is the optimal solution to our problem . However , it might be more practical to consider the expected achievable accuracy when an operation is selected , instead of the best . Therefore we also define the function πavg : πavg-acc ( At , e ) = argmax o∈Oe E A|E|∈At , e , o V ∗ ( A|E| ) ( 2 ) In practice , we are unable to use either πbest-acc or πavg-acc since we would need to have the final validation accuracy V ∗ of all the networks in the search space . There have been many attempts at finding approximate operation scoring functions , in the following we consider the following practical alternatives from DARTS ( Liu et al. , 2019 ) and DARTS-PT ( Wang et al. , 2021 ) : πdarts ( At , e ) = argmax o∈Oe αe , o ( 3 ) πdisc-acc ( At , e ) = argmax o∈Oe V ∗ ( At + ( e , o ) ) , πdarts-pt ( At , e ) = argmin o∈Oe V ( At − ( e , o ) ) ( 4 ) where αe , o is the architectural parameter assigned to operation o on edge e as presented in DARTS ( Liu et al. , 2019 ) . πdisc-acc uses accuracy of a supernet after an operation o is assigned to an edge e – this is referred to as “ discretization accuracy ” in DARTS-PT and is assumed to be a good operation scoring function ( Wang et al. , 2021 ) , most intuitively , it could approximate favg-acc . πdarts-pt is the perturbation-based approach used by DARTS-PT – it is presented as a practical and lightweight alternative to πdisc-acc ( Wang et al. , 2021 ) . Zero-Cost Operation Scoring . We argue that the scoring functions ( 3 ) and ( 4 ) are merely proxies for the best achievable accuracy ( Equation 1 ) . As such , we see an opportunity to use a new class of training-free proxies that are very fast to compute and have been shown to work well within NAS , albeit not in differentiable NAS , nor within large search spaces . We present the following scoring functions that use a zero-cost proxy S instead of validation accuracy when discretizing an edge or perturbing an operation . Note that the supernet is randomly-initialized and untrained in this case . πdisc-zc ( At , e ) = argmax o∈Oe S ( At + ( e , o ) ) , πzc-pt ( At , e ) = argmin o∈Oe S ( At − ( e , o ) ) ( 5 ) Note that TE-NAS ( Chen et al. , 2021 ) also uses training-free scoring of operations , however , they use different scoring metrics to prune operations from a supernet as opposed to discretizing or perturbing operations as we show above . We include a comparison to TE-NAS throughout our paper .

Summary: Differentiable neural architecture search (NAS), and more recently, perturbation-based operation selection in differentiable NAS, is a popular recent area of study in NAS. In parallel, zero-cost proxies are also gaining in popularity in NAS. In this paper, the authors combine the insights from perturbation-based operation selection and zero-cost proxies, to create a new method that sees substantial time speedups. Specifically, they formalize "operation scoring", and use zero-cost proxies to score operations with the perturbation paradigm. This leads them to a new NAS method (Zero-Cost-PT) that outperforms existing methods. They evaluate their proposed algorithm on the DARTS search space (and its subsets) and NAS-Bench-201, showing that it can achieve up to 40x speedups.