Picking up the pieces: separately evaluating supernet training and architecture selection

1 INTRODUCTION . Neural architecture search ( NAS ) describes the problem of automatically selecting an effective neural network architecture for given data . In their survey of work on NAS , Elsken et . al . establish three necessary components to achieve this goal : a search space to define the set possible architectures , a search strategy to traverse over potential solutions , and a performance estimation strategy to rate the quality of potential solutions considered by the search strategy ( Elsken et al. , 2019 ) . In recent years a variety of search spaces and benchmark problems have been proposed in NAS ( Elsken et al. , 2019 ) . Exploring the space of search strategies has similarly generated vast interest , with approaches spanning evolutionary algorithms ( Real et al. , 2018 ) , Bayesian optimization ( Zela et al. , 2018 ) , reinforcement learning ( Pham et al. , 2018 ) , and , most recently , direct gradient descent ( Liu et al. , 2019 ) . Various methods for performance estimation have also been proposed , including weight inheritance ( Real et al. , 2017 ) , predictive models ( Liu et al. , 2018 ) , shortened training ( Zoph et al. , 2018 ) , and weight-sharing ( Pham et al. , 2018 ) . However , the search space , search strategy , and performance estimation components of a NAS algorithm do not operate in a vacuum , and the interactions and relationships between them may dramatically affect the behavior and performance of the overall algorithm . For example , performance estimates that extrapolate from previously seen models are biased by the search strategy ’ s method for traversing the search space and differentially sampling different models . Here , we suggest that the overwhelmingly trend in the current literature is to focus a study on innovations within a single one of these aspects ( e.g . shared-weight model training for performance estimation ) without a broader and systematic consideration , analysis , or even toolset for exploring the effects that this innovation may have on other components of the NAS algorithm ( e.g . search strategy for selecting architecture topologies ) . While we do claim to provide a perfect solution for the analysis an algorithmic innovation ’ s impact on each of the different components , here we provide one example of a systematic strategy and open-source toolset for performing such an investigation – and demonstrate a specific example of how such a perspective modifies the way that we interpret the theoretical impact of an important recent algorithmic and innovation . This paper focuses specifically on the class of algorithms which combine a weight-sharing performance estimation strategy and a gradient-based search strategy into the training of a single model . This approach trains a neural network representing a superposition of every architecture in the search space instead of training individual architectures and obtains performance estimates from this shared-weight model . This technique , which we will refer to as the supernet or supergraph method , was originally demonstrated by Pham et al . ( 2018 ) , however the gradient-based training method contributed by DARTS ( Liu et al. , 2019 ) eliminated the need to train a second reinforcement learning model to select architectures , leading to the development of the series of NAS algorithms combining performance estimating and search in a single model considered here . Note that following Bender et al . ( 2018 ) , the term “ One-shot NAS ” has been used to refer to supernet methods as they train a single model rather then iteratively training candidate architectures . However , as we will discuss iterative search algorithms implemented on top the supernet , we adopt the supernet terminology used by Wang et al . ( 2021 ) . Differentiable supernet NAS algorithms have gained popularity due to their speed attained through weight-sharing – especially relative to employing the “ gold standard ” performance measure of training a neural network from multiple different random initializations and measuring the accuracy on a held-out set of validation data that often requires thousands of GPU hours for a single run ( Zoph et al. , 2018 ) . They also benefit from and their ease of implementation within popular deep learning frameworks as a result of their reliance on gradient based optimization . Importantly for this paper ’ s perspective , these algorithms establish a special relationship between the search strategy and the performance estimation strategy – collapsing both components to the training of a single neural network . This presents a particular example of the cross-component interaction phenomenon described above during the supernet training , where the shared weights are trained to maximize the performance of the previously explored architectures , and are then employed to estimate the performance of new potential architectures . The feedback loop between these two processes thus creates an opportunity to bias search towards familiar solutions , possibly at the expense of undervaluing architectures which have high train-from-scratch performance , but may not perform well with the weights learned on previously explored architectures . This interaction is seen again post-supernet training , when the shared-weight model ( itself embodying the chosen performance estimation strategy ) is often critical in the process to select the single best architecture from the search space . With both architecture performance estimation and search strategy represented by the training of a single sueprnet model , it is perhaps not surprising that the distinct contributions of these two are not easily measured or conceptualized . We might further expect the degree to which this may be an issue to vary with the convexity of the search space , and the search strategy used to traverse that landscape . In order to facilitate a greater understanding of , and more robust comparisons between , differentiable supernet NAS algorithms , here we present two statistics designed to separately evaluate ( 1 ) the architecture search and ( 2 ) the performance estimation components of the supernet training process . We also demonstrate the value of separately evaluating performance estimation and search during supernet training by presenting a two-stage framing of differentiable stochastic NAS , delinking the process of training the supernet from the process of selecting an architecture using the supernet . Using our statistics of the Stage-1 supernet training , we present a method for approximating the reliance of the Stage-2 architecture selection methods on the performance estimation capabilities of the supernet versus the discovery of quality architectures through supernet training . We offer empirical results demonstrating the capabilities of using the presented statistics to design NAS algorithms through the composition of separate supernet training and architecture selection methods , as well as highlighting a case-study demonstrating the importance of considering the interactions of Stage-1 supernet training and Stage-2 architecture selection on the analysis of a recent algorithmic innovation . 2 RELATED WORK . 2.1 DIFFERENTIABLE SUPERNET NAS . DARTS was the first published differentiable NAS algorithm , utilizing a “ continuous relaxation ” method to parametrize the set of possible architectures , by placing trainable weights on each possible operation ( Liu et al. , 2019 ) . The final model is selected from these trained architecture weights by simply keeping only the operations with the largest weight . This method is general enough to support the discovery of a broad range of neural network topologies through the use of a “ zero ” operation , representing the lack of an edge between nodes . Numerous works since the publication of DARTS have attempted to demonstrate simple improvements on DARTS to stabilize training , avoid the hard prune at the end of training , or improve scalability . For example , PC-DARTS ( Xu et al. , 2020 ) has demonstrated partial channel training , allowing the use of larger supernet model . This work addresses the final hard prune by removing the magnitude based prune and substituting a perturbation-based pruning algorithm . While Chu et al . ( 2021 ) stabilizes the training process using an auxiliary skip connection , in contrast to prior work which relied on the Hessian eigenvalue regularizer . In motivating SNAS , Xie et al . comment on the performance estimation aspect of DARTS , demonstrating the disparity in the validation accuracy obtained from the shared weight model of DARTS using the whole shared-weight model and the pruned final architecture ( Xie et al. , 2019 ) . Stochastic differentiable NAS algorithms , like SNAS , offer one answer to this question of performance estimation in differentiable NAS through the use of sampling . These methods preserve the gradient-based approach of DARTS , while proposing a specific ( and increasingly discrete ) architecture at each step . Motivating the performances estimation abilities of shared weight models more generally , Bender et al . ( 2018 ) implement a path-based form of dropout , randomly masking portions of the shared weight model during training and demonstrated a strong correlation between performance estimates from shortened training and the shared weight model , suggesting that the two methods of performance estimation are comparable . 2.2 ZERO-SHOT NAS . Recently , methods have been proposed to select an architecture without training even a single architecture model . Much of these methods are based on proxies , like EcoNAS ( Zhou et al. , 2020 ) , while TE-NAS ( Chen et al. , 2021a ) recently demonstrated a zero-shot search based on measures emerging from deep learning theory . Abdelfattah et al . ( 2021 ) demonstrate a range of measures which have been used to compute saliency in other deep learning problems as possible measures to be useful in NAS . In this work we modify the adapted synflow measure demonstrate by Abdelfattah et al . ( 2021 ) as well their implementation Jacobian covariance measure developed by Mellor et al . ( 2021 ) to function as architecture selection techniques on a supernet . 2.3 SUPERNET NAS EVALUATION . Research on the evaluation of NAS methods has developed as a result of difficulties in comparing the performance NAS algorithms demonstrated in different architecture search spaces , as the selection of search space can have a greater effect on the final performance than the selection of NAS algorithm , which can be made clear through performing a random search ( Li & Talwalkar , 2019 ) . Yu et al . ( 2020 ) initiated the critical inquiry into the use of weight-sharing for performance estimation by evaluating popular weight-sharing NAS algorithms in a reduced search space of 32 architectures . They demonstrated a lack of correlation in ranking between the performance estimates obtained from the shared-weight model and the trained from scratch test accuracy for and an improvement in performance of the final model achieved by not using weight-sharing . Further research evaluating the rankings of small samples ( Yang et al. , 2020 ) or small search spaces ( Zhang et al. , 2020 ) showed high variance in rankings across random seeds after training the shared weight model through random sampling . Like these prior works , we also suggest the the accuracy of the top model alone can be a misleading evaluation technique for NAS algorithms . However , our basis for this assertion is not the design of the search space , but that the reported accuracy value may be due to the unintended interactions between algorithmic innovation presented and other components of the NAS algorithm ( e.g . performance estimation or search strategy ) , and not necessarily well represent the merits of the proposed methodology itself . This motivates our proposal here for additional metrics which evaluate separately the performance estimation or architecture sampling aspects of a given algorithm . This work is most aligned with direction of NAS evaluation research demonstrated by Zela et al . ( 2020 ) . In this work they propose the dominant eigenvalue of the Hessian of the validation loss w.r.t . the architecture parameters as a statistic which helps to explain the failure modes of DARTS . Specifically , they associate a large eigenvalue with a significant drop in performance resulting from pruning . We are also attempt to establish statistics which indicate the potential outcome of archi- tecture selection for a given trained supernet , however we are not focused on the pruning-based architecture selection used by DARTS . Instead of formulating our statistic to account for the failure modes of a specific NAS algorithm , we propose statistics based on the broad theory of the NAS process .

Summary: This paper introduces an evaluation methodology for differentiable NAS methods. Specifically, it separates the search of these methods into two stages: Stage 1, which consists of training the supernet, and Stage 2, whose goal is to select an architecture. These two stages are evaluated independently, and the results of these evaluations are combined into a score whose goal is to provide information about how well any combination of one Stage 1 strategy with one Stage 2 strategy would perform.