Picking up the pieces: separately evaluating supernet training and architecture selection

1 INTRODUCTION . Neural architecture search ( NAS ) describes the problem of automatically selecting an effective neural network architecture for given data . In their survey of work on NAS , Elsken et . al . establish three necessary components to achieve this goal : a search space to define the set possible architectures , a search strategy to traverse over potential solutions , and a performance estimation strategy to rate the quality of potential solutions considered by the search strategy ( Elsken et al. , 2019 ) . In recent years a variety of search spaces and benchmark problems have been proposed in NAS ( Elsken et al. , 2019 ) . Exploring the space of search strategies has similarly generated vast interest , with approaches spanning evolutionary algorithms ( Real et al. , 2018 ) , Bayesian optimization ( Zela et al. , 2018 ) , reinforcement learning ( Pham et al. , 2018 ) , and , most recently , direct gradient descent ( Liu et al. , 2019 ) . Various methods for performance estimation have also been proposed , including weight inheritance ( Real et al. , 2017 ) , predictive models ( Liu et al. , 2018 ) , shortened training ( Zoph et al. , 2018 ) , and weight-sharing ( Pham et al. , 2018 ) . However , the search space , search strategy , and performance estimation components of a NAS algorithm do not operate in a vacuum , and the interactions and relationships between them may dramatically affect the behavior and performance of the overall algorithm . For example , performance estimates that extrapolate from previously seen models are biased by the search strategy ’ s method for traversing the search space and differentially sampling different models . Here , we suggest that the overwhelmingly trend in the current literature is to focus a study on innovations within a single one of these aspects ( e.g . shared-weight model training for performance estimation ) without a broader and systematic consideration , analysis , or even toolset for exploring the effects that this innovation may have on other components of the NAS algorithm ( e.g . search strategy for selecting architecture topologies ) . While we do claim to provide a perfect solution for the analysis an algorithmic innovation ’ s impact on each of the different components , here we provide one example of a systematic strategy and open-source toolset for performing such an investigation – and demonstrate a specific example of how such a perspective modifies the way that we interpret the theoretical impact of an important recent algorithmic and innovation . This paper focuses specifically on the class of algorithms which combine a weight-sharing performance estimation strategy and a gradient-based search strategy into the training of a single model . This approach trains a neural network representing a superposition of every architecture in the search space instead of training individual architectures and obtains performance estimates from this shared-weight model . This technique , which we will refer to as the supernet or supergraph method , was originally demonstrated by Pham et al . ( 2018 ) , however the gradient-based training method contributed by DARTS ( Liu et al. , 2019 ) eliminated the need to train a second reinforcement learning model to select architectures , leading to the development of the series of NAS algorithms combining performance estimating and search in a single model considered here . Note that following Bender et al . ( 2018 ) , the term “ One-shot NAS ” has been used to refer to supernet methods as they train a single model rather then iteratively training candidate architectures . However , as we will discuss iterative search algorithms implemented on top the supernet , we adopt the supernet terminology used by Wang et al . ( 2021 ) . Differentiable supernet NAS algorithms have gained popularity due to their speed attained through weight-sharing – especially relative to employing the “ gold standard ” performance measure of training a neural network from multiple different random initializations and measuring the accuracy on a held-out set of validation data that often requires thousands of GPU hours for a single run ( Zoph et al. , 2018 ) . They also benefit from and their ease of implementation within popular deep learning frameworks as a result of their reliance on gradient based optimization . Importantly for this paper ’ s perspective , these algorithms establish a special relationship between the search strategy and the performance estimation strategy – collapsing both components to the training of a single neural network . This presents a particular example of the cross-component interaction phenomenon described above during the supernet training , where the shared weights are trained to maximize the performance of the previously explored architectures , and are then employed to estimate the performance of new potential architectures . The feedback loop between these two processes thus creates an opportunity to bias search towards familiar solutions , possibly at the expense of undervaluing architectures which have high train-from-scratch performance , but may not perform well with the weights learned on previously explored architectures . This interaction is seen again post-supernet training , when the shared-weight model ( itself embodying the chosen performance estimation strategy ) is often critical in the process to select the single best architecture from the search space . With both architecture performance estimation and search strategy represented by the training of a single sueprnet model , it is perhaps not surprising that the distinct contributions of these two are not easily measured or conceptualized . We might further expect the degree to which this may be an issue to vary with the convexity of the search space , and the search strategy used to traverse that landscape . In order to facilitate a greater understanding of , and more robust comparisons between , differentiable supernet NAS algorithms , here we present two statistics designed to separately evaluate ( 1 ) the architecture search and ( 2 ) the performance estimation components of the supernet training process . We also demonstrate the value of separately evaluating performance estimation and search during supernet training by presenting a two-stage framing of differentiable stochastic NAS , delinking the process of training the supernet from the process of selecting an architecture using the supernet . Using our statistics of the Stage-1 supernet training , we present a method for approximating the reliance of the Stage-2 architecture selection methods on the performance estimation capabilities of the supernet versus the discovery of quality architectures through supernet training . We offer empirical results demonstrating the capabilities of using the presented statistics to design NAS algorithms through the composition of separate supernet training and architecture selection methods , as well as highlighting a case-study demonstrating the importance of considering the interactions of Stage-1 supernet training and Stage-2 architecture selection on the analysis of a recent algorithmic innovation . 2 RELATED WORK . 2.1 DIFFERENTIABLE SUPERNET NAS . DARTS was the first published differentiable NAS algorithm , utilizing a “ continuous relaxation ” method to parametrize the set of possible architectures , by placing trainable weights on each possible operation ( Liu et al. , 2019 ) . The final model is selected from these trained architecture weights by simply keeping only the operations with the largest weight . This method is general enough to support the discovery of a broad range of neural network topologies through the use of a “ zero ” operation , representing the lack of an edge between nodes . Numerous works since the publication of DARTS have attempted to demonstrate simple improvements on DARTS to stabilize training , avoid the hard prune at the end of training , or improve scalability . For example , PC-DARTS ( Xu et al. , 2020 ) has demonstrated partial channel training , allowing the use of larger supernet model . This work addresses the final hard prune by removing the magnitude based prune and substituting a perturbation-based pruning algorithm . While Chu et al . ( 2021 ) stabilizes the training process using an auxiliary skip connection , in contrast to prior work which relied on the Hessian eigenvalue regularizer . In motivating SNAS , Xie et al . comment on the performance estimation aspect of DARTS , demonstrating the disparity in the validation accuracy obtained from the shared weight model of DARTS using the whole shared-weight model and the pruned final architecture ( Xie et al. , 2019 ) . Stochastic differentiable NAS algorithms , like SNAS , offer one answer to this question of performance estimation in differentiable NAS through the use of sampling . These methods preserve the gradient-based approach of DARTS , while proposing a specific ( and increasingly discrete ) architecture at each step . Motivating the performances estimation abilities of shared weight models more generally , Bender et al . ( 2018 ) implement a path-based form of dropout , randomly masking portions of the shared weight model during training and demonstrated a strong correlation between performance estimates from shortened training and the shared weight model , suggesting that the two methods of performance estimation are comparable . 2.2 ZERO-SHOT NAS . Recently , methods have been proposed to select an architecture without training even a single architecture model . Much of these methods are based on proxies , like EcoNAS ( Zhou et al. , 2020 ) , while TE-NAS ( Chen et al. , 2021a ) recently demonstrated a zero-shot search based on measures emerging from deep learning theory . Abdelfattah et al . ( 2021 ) demonstrate a range of measures which have been used to compute saliency in other deep learning problems as possible measures to be useful in NAS . In this work we modify the adapted synflow measure demonstrate by Abdelfattah et al . ( 2021 ) as well their implementation Jacobian covariance measure developed by Mellor et al . ( 2021 ) to function as architecture selection techniques on a supernet . 2.3 SUPERNET NAS EVALUATION . Research on the evaluation of NAS methods has developed as a result of difficulties in comparing the performance NAS algorithms demonstrated in different architecture search spaces , as the selection of search space can have a greater effect on the final performance than the selection of NAS algorithm , which can be made clear through performing a random search ( Li & Talwalkar , 2019 ) . Yu et al . ( 2020 ) initiated the critical inquiry into the use of weight-sharing for performance estimation by evaluating popular weight-sharing NAS algorithms in a reduced search space of 32 architectures . They demonstrated a lack of correlation in ranking between the performance estimates obtained from the shared-weight model and the trained from scratch test accuracy for and an improvement in performance of the final model achieved by not using weight-sharing . Further research evaluating the rankings of small samples ( Yang et al. , 2020 ) or small search spaces ( Zhang et al. , 2020 ) showed high variance in rankings across random seeds after training the shared weight model through random sampling . Like these prior works , we also suggest the the accuracy of the top model alone can be a misleading evaluation technique for NAS algorithms . However , our basis for this assertion is not the design of the search space , but that the reported accuracy value may be due to the unintended interactions between algorithmic innovation presented and other components of the NAS algorithm ( e.g . performance estimation or search strategy ) , and not necessarily well represent the merits of the proposed methodology itself . This motivates our proposal here for additional metrics which evaluate separately the performance estimation or architecture sampling aspects of a given algorithm . This work is most aligned with direction of NAS evaluation research demonstrated by Zela et al . ( 2020 ) . In this work they propose the dominant eigenvalue of the Hessian of the validation loss w.r.t . the architecture parameters as a statistic which helps to explain the failure modes of DARTS . Specifically , they associate a large eigenvalue with a significant drop in performance resulting from pruning . We are also attempt to establish statistics which indicate the potential outcome of archi- tecture selection for a given trained supernet , however we are not focused on the pruning-based architecture selection used by DARTS . Instead of formulating our statistic to account for the failure modes of a specific NAS algorithm , we propose statistics based on the broad theory of the NAS process .

Summary: This paper studies the statistics of weight-sharing based neural architecture search (NAS) algorithms. In particular, it separates the training process into two stages: one is for supernet weight training and the other for architecture selection, and then it investigates the Spearman  correlation and other statistics. All experiments are based on NAS-Bench201.