Exploring single-path Architecture Search ranking correlations

1 INTRODUCTION . The development and study of algorithms that automatically design neural networks , Neural Architecture Search ( NAS ) , has become a significant influence in recent years ; owed to the promise of creating better models with less human effort and in shorter time . Whereas the first generations of algorithms required training thousands of networks in thousands of GPU hours using reinforcement learning ( Zoph & Le ( 2016 ) ; Zoph et al . ( 2018 ) ) , greedy progressive optimization ( Liu et al . ( 2018a ) ) , regularized evolution ( Real et al . ( 2018 ) ) and more , the invention of weight sharing during search ( Pham et al . ( 2018 ) ) reduced the computation cost to few GPU hours , and thus made NAS accessible to a much wider audience . While this also enables gradient based NAS ( Liu et al . ( 2018b ) ) , the necessity to compare operations against each other leads to an increased memory requirement . The issue is commonly alleviated by training a small search network consisting of cells with a shared topology , later scaling the resulting architecture up by adding more cells and increasing the number of channels . Although the standalone network is often trained from scratch , reusing the search network weights can increase both training speed and final accuracy ( Yan et al . ( 2019 ) ; Hu et al . ( 2020 ) ) . More recent gradient based methods require to have only one path in memory ( Dong & Yang ( 2019 ) ; Cai et al . ( 2019 ) ; Hu et al . ( 2020 ) ) and can even be applied directly to huge data sets . However , the aforementioned weight sharing methods only yield a single result , require manually fine-tuning the loss function when there are multiple objectives , and can not guarantee results within constraints ( e.g . latency , FLOPs ) . The single-path one-shot approach seeks to combine the best of both worlds , requiring only one additional step in the search phase ( Guo et al . ( 2020 ) ) : Firstly a full sized weight-sharing model ( super-network ) is fully trained by randomly choosing one of the available operations at each layer in every training step . Then , as specific architectures can be evaluated by choosing the model ’ s operations accordingly , a hyper-parameter optimization method can be used to find combinations of operations maximizing the super-network accuracy . If the rankings of the architectures by their respective super-network accuracy and by their stand-alone model retraining results are consistent , the quality of the discovered candidates is high . However , since the single-path method ’ s search spaces are often gigantic and the network training costly ( see e.g . Guo et al . ( 2020 ) ; Chu et al . ( 2019b ; a ) ) , a study of the ranking correlation is usually limited to a handful of architectures . In this work we study the single-path one-shot super-network predictions and ranking correlation throughout an entire search space , as all stand-alone model re- sults are known in advance . This enables us to quantify the effects of several super-network training variations and search space subsets , to gain further insights on the popular single-path one-shot method itself . We briefly list the closest related work in Section 2 and introduce the measurement metric , benchmark dataset , super-network training and experiment design in Section 3 . We then systematically evaluate several variations in the single-path one-shot approach with a novel method , computing the ranking correlation of the trained super-networks with the ground-truth top-N best architectures . Experiments on search space subsets in Section 4.1 once again demonstrate that the ranking is more difficult as the search space increases in size , and that the operations that make the ranking especially hard are Zero and Pool . Section 4.2 evaluates Linear Transformers ( Chu et al . ( 2019a ) ) , which we find to perform very well in specific search space subsets , and otherwise even harmful . Furthermore , some commonly used training variations such as learning rate warmup , gradient clipping , data augmentation and regularization are evaluated in Section 4.3 , where we find that none of these provides a measurable improvement . We further test disabling cell topology sharing only during training time and find that training the network in the same way as evaluating it is more effective . We finally list some grains of salt in Section 5 and conclude the paper with Section 6 . 2 RELATED WORK . A high quality architecture ranking prediction is the foundation of any NAS algorithm . In this paper we explore the effects of several super-network training variations on the ranking prediction of the aforementioned single-path one-shot approach ( Guo et al . ( 2020 ) ) . Recent efforts have shown improvements by strictly fair operation sampling in the super-network training phase ( Chu et al . ( 2019b ) ) and adding a linear 1×1 convolution to skip connections , improving training stability ( Chu et al . ( 2019a ) ) . Other works divide the search space , exploring multiple models with different operation-subsets ( Zhao et al . ( 2020 ) ) , or one model with several smaller blocks that use a trained teacher as a guiding signal ( Li et al . ( 2020b ) ) . Due to the often gigantic search spaces and the inherent randomness of network training and hyperparameter optimization algorithms , the reproducibility of NAS methods has become a major concern . NAS Benchmarks attempt to alleviate this issue by providing statistics ( e.g . validation loss , accuracy and latency ) of several thousand different networks on multiple data sets ( Ying et al . ( 2019 ) ; Dong & Yang ( 2020 ) ) , providing the ground-truth training results that we use for our evaluation . 3 METHOD . 3.1 METRIC . As we correlate the super-network accuracy prediction and the benchmark results , but are only interested in a correct ranking , we need a ranking correlation metric . We choose Kendall ’ s Tau ( τ , KT ) , a commonly used ranking metric ( Sciuto et al . ( 2019 ) ; Chu et al . ( 2019b ) ) that counts how often all pairs of observations ( xi , yi ) and ( xj , yj ) 1. are concordant , agreeing on a sorting order ( xi < xj and yi < yj ; or xi > xj and yi > yj ) 2. are discordant , disagreeing on a sorting order ( xi < xj and yi > yj ; or xi > xj and yi < yj ) 3. are neither and is then calculated by their difference and normalized by the number of possible different pairs . τ = ( num concordant ) − ( num discordant ) ( n2 ) τ ranges from -1 in perfect disagreement to +1 in perfect agreement , and is around zero for independent X and Y . A small selection of experiments that use additional metrics can be found in Appendix D . 3.2 NAS-BENCH-201 . NAS-Bench-201 ( Dong & Yang ( 2020 ) ) is a tabular benchmark , which contains training and evaluation statistics of 15625 different architectures on the common vision data sets CIFAR10 , CIFAR100 ( Krizhevsky et al . ( 2009 ) ) and a reduced variant of ImageNet ( Deng et al . ( 2009 ) ) . The models differ in the design of the cell , a building block that is stacked several times to create a network . Within the cell , as visualized in Figure 1 , at six specific positions ( orange edges ) , one of five operations ( Zero , Skip , 1×1 Convolution , 3×3 Convolution , 3×3 Average Pooling ) is chosen ( 56 = 15625 ) . The inputs of each node , such as the cell output ( rightmost node ) are averaged . N * Stem 3x3 Conv reduction ResNet N * reduction ResNet N * Head GAP , Softmax As we are only interested in the final accuracy of each architecture , we average the benchmark test results over all seeds and the last three epochs of training . As the models ’ rankings are quite consistent across all data sets ( Dong & Yang ( 2020 ) ) , we focus on the CIFAR-10-Valid accuracy . Further results are provided in the supplementary material , see Appendix B . Since discrepancies of model rankings for the top performing architectures became apparent ( Dong & Yang ( 2020 ) ) , we measure the accuracy of the trained super-networks according to the top-N ( 10 , 25 , 50 , 150 , 250 , 500 ) benchmark architectures , as well as up to 1000 randomly sampled ones . If a reduced search space ( due to masking operations , 36 = 729 ) contains fewer than 1000 different topologies , it is fully evaluated . 3.3 TRAINING . In our experiments we train various NAS-Bench-201 networks . Small variants have 2 cells per stage ( total of 8 cells , with 3 stages and 2 fixed cells for spatial reduction ) and 32 channels in the first cell , which is roughly similar to common topology sharing methods . Medium sized networks have 4 cells per stage and start with 64 channels . All models were subject to the same training schedule . We used CIFAR10 as training set ( Krizhevsky et al . ( 2009 ) ) , of which we withheld 5000 images for validation . The batch size is 256 , we used SGD with momentum of 0.9 and learning rate of 0.025 , which was cosine annealed to 1e-5 over 250 epochs . All results are averaged over five independent runs with different seeds . Further details are listed in Appendix A . 3.4 EXPERIMENT DESIGN . All of the following experiments are structured the same way : The top-N network architectures ( ordered by top1 accuracy , measured in NAS-Bench-201 ) are selected , and an over-complete supernetwork predicts their respective accuracy values , as seen in Figure 2 . If an operation is not available to the super-network , the top-N networks are also taken from the bench results without that operation . Variations to the search space and the super-network ( structure or training process ) affect the ranking correlation τ between the bench results and the super-network predictions . In the case of Figure 2 , removing the Zero operation from the search space improves τ . To make the figures more compact , the exact benchmark and prediction values are ignored in the further figures , only average prediction accuracy and τ depending on N will be shown ( see Appendix B for further detailed figures ) , as seen in e.g . Figure 4 . We also add the additional metric τa which describes the ranking correlation of the average prediction accuracy depending on N. More formally , τa is computed as described in Section 3.1 on the series of measurements [ ( 10 , A10 ) , ( 25 , A25 ) , ( 50 , A50 ) , ... ] where AN is the accuracy of super-network M with topology Ti and weights θs on the validation data Dvalid , averaged over the top-N topologies and multiple seeds . AN = ∑ s∈seeds 1 |seeds| N∑ i=1 1 N Acc ( M , θs , Ti , Dvalid ) As we increase N ( 10 , 25 , 50 , ... ) AN should monotonically decrease ( e.g . 0.7 , 0.65 , 0.6 , ... ) , so that τa = −1 is the case where the super-network estimates match the bench results best .

Summary: + This paper studies the single-path one-shot super-network predictions and ranking correlation throughout an entire search space, as all stand-alone model results are known in advance. This is a crucial step in NAS. As we know, inaccurate architecture rating is the cause of ineffective NAS in almost all existing NAS methods. It makes nearly all previous NAS methods not better the random architecture selection (suggested by two ICLR 2020 papers and many ICLR 2021 submissions). Therefore, analyzing the architecture rating problem is of most importance in NAS. This paper takes a deep insight into the architecture rating problem, which provides a timely metric for evaluating NAS's effectiveness. (+)