How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS

1 INTRODUCTION . Neural architecture search ( NAS ) has received growing attention in the past few years , yielding stateof-the-art performance on several machine learning tasks ( Liu et al. , 2019a ; Wu et al. , 2019 ; Chen et al. , 2019b ; Ryoo et al. , 2020 ) . One of the milestones that led to the popularity of NAS is weight sharing ( Pham et al. , 2018 ; Liu et al. , 2019b ) , which , by allowing all possible network architectures to share the same parameters , has reduced the computational requirements from thousands of GPU hours to just a few . Figure 1 shows the two phases that are common to weight-sharing NAS ( WS-NAS ) algorithms : the search phase , including the design of the search space and the search algorithm ; and the evaluation phase , which encompasses the final training protocol on the proxy task 1 . While most works focus on developing a good sampling algorithm ( Cai et al. , 2019 ; Xie et al. , 2019 ) or improving existing ones ( Zela et al. , 2020a ; Nayman et al. , 2019 ; Li et al. , 2020 ) , they tend to overlook or gloss over important factors related to the design and training of the shared-weight backbone network , i.e . the super-net . For example , the literature encompasses significant variations of learning hyper-parameter settings , batch normalization and dropout usage , capacities for the initial layers of the network , and depth of the super-net . Furthermore , some of these heuristics are directly transferred from standalone network training to super-net training without carefully studying their impact in this drastically different scenario . For example , the fundamental assumption of batch normalization that the input data follows a slowly changing distribution whose statistics can be tracked during training is violated in WS-NAS , but nonetheless typically assumed to hold . In this paper , we revisit and systematically evaluate commonly-used super-net design and training heuristics and uncover the strong influence of certain factors on the success of super-net training . To this end , we leverage three benchmark search spaces , NASBench-101 ( Ying et al. , 2019 ) , NASBench201 ( Dong & Yang , 2020 ) , and DARTS-NDS ( Radosavovic et al. , 2019 ) , for which the ground-truth stand-alone performance of a large number of architectures is available . We report the results of our experiments according to two sets of metrics : i ) metrics that directly measure the quality of the super-net , such as the widely-adopted super-net accuracy 2 and a modified Kendall-Tau correlation between the searched architectures and their ground-truth performance , which we refer to as sparse 1Proxy task refers to the tasks that neural architecture search aims to optimize on . 2The mean accuracy over a small set of randomly sampled architectures during super-net training . Under review as a conference paper at ICLR 2021 < latexit sha1_base64= '' dWMZqL35IYT82cSzwIxrSfiYzQ0= '' > AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5idzCZj5rHMzCphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWye1Gv3syUz75Ypf9edAqyTISQVyNPrlr95AkVRQaQnHxnQDP7FhhrVlhNNpqZcammAyxkPadVRiQU2Yza+dojOnDFCstCtp0Vz9PZFhYcxERK5TYDsyy95M/M/rpja+CjMmk9RSSRaL4pQjq9DsdTRgmhLLJ45gopm7FZER1phYF1DJhRAsv7xKWhfVoFat3dUq9es8jiKcwCmcQwCXUIdbaEATCDzAM7zCm6e8F+/d+1i0Frx85hj+wPv8AdUFj04= < /latexit > < latexit sha1_base64= '' xordNtnCPHeHQsW9SFKXfp9hiUg= '' > AAAB8HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmIckS5idzCZD5rHMzIphyVd48aCIVz/Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWyc9xP0s0eppMu2XK37VnwOtkiAnFcjR6Je/egNFUkGlJRwb0w38xIYZ1pYRTqelXmpogskYD2nXUYkFNWE2P3iKzpwyQLHSrqRFc/X3RIaFMRMRuU6B7cgsezPxP6+b2vgqzJhMUkslWSyKU46sQrPv0YBpSiyfOIKJZu5WREZYY2JdRiUXQrD88ippXVSDWrV2V6vUr/M4inACp3AOAVxCHW6hAU0gIOAZXuHN096L9+59LFoLXj5zDH/gff4AeDWQ2g== < /latexit > < latexit sha1_base64= '' 3iE++PYZojxQ9mAgZyyC7qSUlxU= '' > AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5idzCZj5rHMzCphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWye14n72ZKb9csWv+nOgVRLkpAI5Gv3yV2+gSCqotIRjY7qBn9gww9oywum01EsNTTAZ4yHtOiqxoCbM5tdO0ZlTBihW2pW0aK7+nsiwMGYiItcpsB2ZZW8m/ud1UxtfhRmTSWqpJItFccqRVWj2OhowTYnlE0cw0czdisgIa0ysC6jkQgiWX14lrYtqUKvW7mqV+nUeRxFO4BTOIYBLqMMtNKAJBB7gGV7hzVPei/fufSxaC14+cwx/4H3+APbLj2Q= < /latexit > < latexit sha1_base64= '' 3iE++PYZojxQ9mAgZyyC7qSUlxU= '' > AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5idzCZj5rHMzCphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWye14n72ZKb9csWv+nOgVRLkpAI5Gv3yV2+gSCqotIRjY7qBn9gww9oywum01EsNTTAZ4yHtOiqxoCbM5tdO0ZlTBihW2pW0aK7+nsiwMGYiItcpsB2ZZW8m/ud1UxtfhRmTSWqpJItFccqRVWj2OhowTYnlE0cw0czdisgIa0ysC6jkQgiWX14lrYtqUKvW7mqV+nUeRxFO4BTOIYBLqMMtNKAJBB7gGV7hzVPei/fufSxaC14+cwx/4H3+APbLj2Q= < /latexit > < latexit sha1_base64= '' iU9d81SBeMPs61pcpys14gjB6js= '' > AAAB8HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmIckS5idzCZD5rHMzIphyVd48aCIVz/Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWyc9NPpZotXTZNovV/yqPwdaJUFOKpCj0S9/9QaKpIJKSzg2phv4iQ0zrC0jnE5LvdTQBJMxHtKuoxILasJsfvAUnTllgGKlXUmL5urviQwLYyYicp0C25FZ9mbif143tfFVmDGZpJZKslgUpxxZhWbfowHTlFg+cQQTzdytiIywxsS6jEouhGD55VXSuqgGtWrtrlapX+dxFOEETuEcAriEOtxCA5pAQMAzvMKbp70X7937WLQWvHzmGP7A+/wBVi2QxA== < /latexit > < latexit sha1_base64= '' xordNtnCPHeHQsW9SFKXfp9hiUg= '' > AAAB8HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmIckS5idzCZD5rHMzIphyVd48aCIVz/Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWyc9xP0s0eppMu2XK37VnwOtkiAnFcjR6Je/egNFUkGlJRwb0w38xIYZ1pYRTqelXmpogskYD2nXUYkFNWE2P3iKzpwyQLHSrqRFc/X3RIaFMRMRuU6B7cgsezPxP6+b2vgqzJhMUkslWSyKU46sQrPv0YBpSiyfOIKJZu5WREZYY2JdRiUXQrD88ippXVSDWrV2V6vUr/M4inACp3AOAVxCHW6hAU0gIOAZXuHN096L9+59LFoLXj5zDH/gff4AeDWQ2g== < /latexit > < latexit sha1_base64= '' dWMZqL35IYT82cSzwIxrSfiYzQ0= '' > AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5idzCZj5rHMzCphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWye1Gv3syUz75Ypf9edAqyTISQVyNPrlr95AkVRQaQnHxnQDP7FhhrVlhNNpqZcammAyxkPadVRiQU2Yza+dojOnDFCstCtp0Vz9PZFhYcxERK5TYDsyy95M/M/rpja+CjMmk9RSSRaL4pQjq9DsdTRgmhLLJ45gopm7FZER1phYF1DJhRAsv7xKWhfVoFat3dUq9es8jiKcwCmcQwCXUIdbaEATCDzAM7zCm6e8F+/d+1i0Frx85hj+wPv8AdUFj04= < /latexit > < latexit sha1_base64= '' 3iE++PYZojxQ9mAgZyyC7qSUlxU= '' > AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5idzCZj5rHMzCphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWye14n72ZKb9csWv+nOgVRLkpAI5Gv3yV2+gSCqotIRjY7qBn9gww9oywum01EsNTTAZ4yHtOiqxoCbM5tdO0ZlTBihW2pW0aK7+nsiwMGYiItcpsB2ZZW8m/ud1UxtfhRmTSWqpJItFccqRVWj2OhowTYnlE0cw0czdisgIa0ysC6jkQgiWX14lrYtqUKvW7mqV+nUeRxFO4BTOIYBLqMMtNKAJBB7gGV7hzVPei/fufSxaC14+cwx/4H3+APbLj2Q= < /latexit > < latexit sha1_base64= '' iU9d81SBeMPs61pcpys14gjB6js= '' > AAAB8HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmIckS5idzCZD5rHMzIphyVd48aCIVz/Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWyc9NPpZotXTZNovV/yqPwdaJUFOKpCj0S9/9QaKpIJKSzg2phv4iQ0zrC0jnE5LvdTQBJMxHtKuoxILasJsfvAUnTllgGKlXUmL5urviQwLYyYicp0C25FZ9mbif143tfFVmDGZpJZKslgUpxxZhWbfowHTlFg+cQQTzdytiIywxsS6jEouhGD55VXSuqgGtWrtrlapX+dxFOEETuEcAriEOtxCA5pAQMAzvMKbp70X7937WLQWvHzmGP7A+/wBVi2QxA== < /latexit > < latexit sha1_base64= '' xordNtnCPHeHQsW9SFKXfp9hiUg= '' > AAAB8HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmIckS5idzCZD5rHMzIphyVd48aCIVz/Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWyc9xP0s0eppMu2XK37VnwOtkiAnFcjR6Je/egNFUkGlJRwb0w38xIYZ1pYRTqelXmpogskYD2nXUYkFNWE2P3iKzpwyQLHSrqRFc/X3RIaFMRMRuU6B7cgsezPxP6+b2vgqzJhMUkslWSyKU46sQrPv0YBpSiyfOIKJZu5WREZYY2JdRiUXQrD88ippXVSDWrV2V6vUr/M4inACp3AOAVxCHW6hAU0gIOAZXuHN096L9+59LFoLXj5zDH/gff4AeDWQ2g== < /latexit > < latexit sha1_base64= '' dWMZqL35IYT82cSzwIxrSfiYzQ0= '' > AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5idzCZj5rHMzCphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWye1Gv3syUz75Ypf9edAqyTISQVyNPrlr95AkVRQaQnHxnQDP7FhhrVlhNNpqZcammAyxkPadVRiQU2Yza+dojOnDFCstCtp0Vz9PZFhYcxERK5TYDsyy95M/M/rpja+CjMmk9RSSRaL4pQjq9DsdTRgmhLLJ45gopm7FZER1phYF1DJhRAsv7xKWhfVoFat3dUq9es8jiKcwCmcQwCXUIdbaEATCDzAM7zCm6e8F+/d+1i0Frx85hj+wPv8AdUFj04= < /latexit > Kendall-Tau ; ii ) proxy metrics such as the ability to surpass random search and the stand-alone accuracy of the model found by the WS-NAS algorithm . Via our extensive experiments ( over 700 GPU days ) , we uncover that ( i ) the training behavior of a super-net drastically differs from that of a standalone network , e.g. , in terms of feature statistics and loss landscape , thus allowing us to define training factor settings , e.g. , for batch-normalization ( BN ) and learning rate , that are better suited for super-nets ; ( ii ) while some neglected factors , such as the number of training epochs , have a strong impact on the final performance , others , believed to be important , such as path sampling , only have a marginal effect , and some commonly-used heuristics , such as the use of low-fidelity estimates , negatively impact it ; ( iii ) the commonly-adopted super-net accuracy is unreliable to evaluate the super-net quality . Altogether , our work is the first to systematically analyze the impact of the diverse factors of super-net design and training , and we uncover the factors that are crucial to design a super-net , as well as the non-important ones . Aggregating these findings allows us to boost the performance of simple weight-sharing random search to the point where it reaches that of complex state-of-the-art NAS algorithms across all tested search spaces . We will release our code and trained models so as to establish a solid baseline to facilitate further research . 2 PRELIMINARIES AND RELATED WORK . We first introduce the necessary concepts that will be used throughout the paper . As shown in Figure 1 ( a ) , weight-sharing NAS algorithms consist of three key components : a search algorithm that samples an architecture from the search space in the form of an encoding , a mapping function fproxy that maps the encoding into its corresponding neural network , and a training protocol for a proxy task Pproxy for which the network is optimized . To train the search algorithm , one needs to additionally define the mapping function fws that generates the shared-weight network . Note that the mapping fproxy frequently differs from fws , since in practice the final model contains many more layers and parameters so as to yield competitive results on the proxy task . After fixing fws , a training protocol Pws is required to learn the super-net . In practice , Pws often hides factors that are critical for the final performance of an approach , such as hyper-parameter settings or the use of data augmentation strategies to achieve state-of-the-art performance ( Liu et al. , 2019b ; Chu et al. , 2019 ; Zela et al. , 2020a ) . Again , Pws may differ from Pproxy , which is used to train the architecture that has been found by the search . For example , our experiments reveal that the learning rate and the total number of epochs frequently differ due to the different training behavior of the super-net and stand-alone architectures . Many strategies have been proposed to implement the search algorithm , such as reinforcement learning ( Zoph & Le , 2017 ; Zoph et al. , 2018 ) , evolutionary algorithms ( Real et al. , 2017 ; Miikkulainen et al. , 2019 ; So et al. , 2019 ; Liu et al. , 2018 ; Lu et al. , 2018 ) , gradient-based optimization ( Liu et al. , 2019b ; Xu et al. , 2020 ; Li et al. , 2020 ) , Bayesian optimization ( Kandasamy et al. , 2018 ; Jin et al. , 2019 ; Zhou et al. , 2019 ; Wang et al. , 2020 ) , and separate performance predictors ( Liu et al. , 2018 ; Luo et al. , 2018 ) . Until very recently , the common trend to evaluate NAS consisted of reporting the searched architecture ’ s performance on the proxy task ( Xie et al. , 2019 ; Real et al. , 2019 ; Ryoo et al. , 2020 ) . This , however , hardly provides real insights about the NAS algorithms themselves , because of the many components involved in them . Many factors that differ from one algorithm to another can influence the performance . In practice , the literature even commonly compares NAS methods that employ different protocols to train the final model . Li & Talwalkar ( 2019 ) and Yu et al . ( 2020b ) were the first to systematically compare different algorithms with the same settings for the proxy task and using several random initializations . Their surprising results revealed that many NAS algorithms produce architectures that do not significantly outperform a randomly-sampled architecture . Yang et al . ( 2020 ) highlighted the importance of the training protocol Pproxy . They showed that optimizing the training protocol can improve the final architecture performance on the proxy task by three percent on CIFAR-10 . This non-trivial improvement can be achieved regardless of the chosen sampler , which provides clear evidence for the importance of unifying the protocol to build a solid foundation for comparing NAS algorithms . In parallel to this line of research , the recent series of “ NASBench ” works ( Ying et al. , 2019 ; Zela et al. , 2020b ; Dong & Yang , 2020 ) proposed to benchmark NAS approaches by providing a complete , tabular characterization of a search space . This was achieved by training every realizable stand-alone architecture using a fixed protocol Pproxy . Similarly , other works proposed to provide a partial characterization by sampling and training a sufficient number of architectures in a given search space using a fixed protocol ( Radosavovic et al. , 2019 ; Zela et al. , 2020a ; Wang et al. , 2020 ) . While recent advances for systematic evaluation are promising , no work has yet thoroughly studied the influence of the super-net training protocol Pws and the mapping function fws . Previous works ( Zela et al. , 2020a ; Li & Talwalkar , 2019 ) performed hyper-parameter tuning to evaluate their own algorithms , and focused only on a few parameters . We fill this gap by benchmarking different choices of Pws and fws and by proposing novel variations to improve the super-net quality . Recent works have shown that sub-nets of super-net training can surpass some human designed models without retraining ( Yu et al. , 2020a ; Cai et al. , 2020 ) and that reinforcement learning can surpass the performance of random search ( Bender et al. , 2020 ) . However , these findings are still only shown on MobileNet-like search spaces where we only search for the size of convolution kernels and the channel ratio for each layer . This is an effective approach to discover a compact network , but it does not change the fact that on cell-based search space super-net quality remains low . 3 EVALUATION METHODOLOGY . We first isolate 14 factors that need to be considered during the design and training of a super-net , and then introduce the metrics to evaluate the quality of the trained super-net . Note that these factors are agnostic to the search policy that is used after training the super-net . 3.1 DISENTANGLING THE SUPER-NET FROM THE SEARCH ALGORITHM . Our goal is to evaluate the influence of the super-net mapping fws and weight-sharing training protocol Pws . As shown in Figure 2 , fws translates an architecture encoding , which typically consists of a discrete number of choices or parameters , into a neural network . Based on a well-defined mapping , the super-net is a network in which every sub-path has a one-to-one mapping with an architecture encoding ( Pham et al. , 2018 ) . Recent works ( Xu et al. , 2020 ; Li et al. , 2020 ; Ying et al. , Figure 2 : Constructing a super-net 2019 ) separate the encoding into cell parameters , which define the basic building blocks of a network , and macro parameters , which define how cells are assembled into a complete architecture . Weight-sharing mapping fws . To make the search space manageable , all cell and macro parameters are fixed during the search , except for the topology of the cell and its possible operations . However , the exact choices for each of these fixed factors differ between algorithms and search spaces . We report the common factors in the left part of Table 1 . They include various implementation choices , e.g. , the use of convolutions with a dynamic number of channels ( Dynamic Channeling ) , super-convolutional layers that support dynamic kernel sizes ( OFA Kernel ) ( Cai et al. , 2020 ) , weight-sharing batchnormalization ( WSBN ) that tracks independent running statistics and affine parameters for different incoming edges ( Luo et al. , 2018 ) , and path and global dropout ( Pham et al. , 2018 ; Luo et al. , 2018 ; Liu et al. , 2019b ) . They also include the use of low-fidelity estimates ( Elsken et al. , 2019 ) to reduce the complexity of super-net training , e.g. , by reducing the number of layers ( Liu et al. , 2019b ) and channels ( Yang et al. , 2020 ; Chen et al. , 2019a ) , the portion of the training set used for super-net training ( Liu et al. , 2019b ) , or the batch size ( Liu et al. , 2019b ; Pham et al. , 2018 ; Yang et al. , 2020 ) . Weight-sharing protocol Pws . Given a mapping fws , different training protocols Pws can be employed to train the super-net . Protocols can differ in the training hyper-parameters and the sampling strategies they rely on . We will evaluate the different hyper-parameter choices listed in the right part of Table 1 . This includes the initial learning rate , the hyper-parameters of batch normalization , the total number of training epochs , and the amount of weight decay . We randomly sample one path to train the super-net ( Guo et al. , 2019 ) , which is also known as single-path one-shot ( SPOS ) or Random-NAS ( Li & Talwalkar , 2019 ) . The reason for this choice is that Random-NAS is equivalent to the initial state of many search algorithms ( Liu et al. , 2019b ; Pham et al. , 2018 ; Luo et al. , 2018 ) , some of which even freeze the sampler training so as to use random sampling to warm-up the super-net ( Xu et al. , 2020 ; Dong & Yang , 2019b ) . Note that we also evaluated two variants of Random-NAS , but found their improvement to be only marginal . Please see Appendix C.2 for more detail . In our experiments , for the sake of reproducibility , we ensure that Pws and Pproxy , as well as fws and fproxy , are as close to each other as possible . For the hyper-parameters of Pws , we cross-validate each factor following the order in Table 1 , and after each validation , use the value that yields the best performance in Pproxy . For all other factors , we change one factor at a time . Search spaces . We use three commonly-used search spaces , for which a large number of stand-alone architectures have been trained and evaluated on CIFAR-10 ( Krizhevsky et al. , 2009 ) to obtain their ground-truth performance . In particular , we use NASBench-101 ( Ying et al. , 2019 ) , which consists of 423 , 624 architectures and is compatible with weight-sharing NAS ( Yu et al. , 2020b ; Zela et al. , 2020b ) ; NASBench-201 ( Dong & Yang , 2020 ) , which contains more operations than NASBench-101 but fewer nodes ; and DARTS-NDS ( Radosavovic et al. , 2019 ) that contains over 1013 architectures of which a subset of 5000 models was sampled and trained in a stand-alone fashion . See Appendix A.2 for a detailed discussion .

Summary: Neural Architecture Search (NAS) aims to find a model with the best possible accuracy (or best possible accuracy/size tradeoff) from within a human-defined search space. One popular strategy for speeding up NAS is to train a one-shot model -- a single set of shared weights -- that can then be used to evaluate any candidate architecture within the search space without retraining or fine-tuning. The submission investigates how various decisions made when training a one-shot model can affect its ability to rank different candidate architectures within a search space.