How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS

1 INTRODUCTION . Neural architecture search ( NAS ) has received growing attention in the past few years , yielding stateof-the-art performance on several machine learning tasks ( Liu et al. , 2019a ; Wu et al. , 2019 ; Chen et al. , 2019b ; Ryoo et al. , 2020 ) . One of the milestones that led to the popularity of NAS is weight sharing ( Pham et al. , 2018 ; Liu et al. , 2019b ) , which , by allowing all possible network architectures to share the same parameters , has reduced the computational requirements from thousands of GPU hours to just a few . Figure 1 shows the two phases that are common to weight-sharing NAS ( WS-NAS ) algorithms : the search phase , including the design of the search space and the search algorithm ; and the evaluation phase , which encompasses the final training protocol on the proxy task 1 . While most works focus on developing a good sampling algorithm ( Cai et al. , 2019 ; Xie et al. , 2019 ) or improving existing ones ( Zela et al. , 2020a ; Nayman et al. , 2019 ; Li et al. , 2020 ) , they tend to overlook or gloss over important factors related to the design and training of the shared-weight backbone network , i.e . the super-net . For example , the literature encompasses significant variations of learning hyper-parameter settings , batch normalization and dropout usage , capacities for the initial layers of the network , and depth of the super-net . Furthermore , some of these heuristics are directly transferred from standalone network training to super-net training without carefully studying their impact in this drastically different scenario . For example , the fundamental assumption of batch normalization that the input data follows a slowly changing distribution whose statistics can be tracked during training is violated in WS-NAS , but nonetheless typically assumed to hold . In this paper , we revisit and systematically evaluate commonly-used super-net design and training heuristics and uncover the strong influence of certain factors on the success of super-net training . To this end , we leverage three benchmark search spaces , NASBench-101 ( Ying et al. , 2019 ) , NASBench201 ( Dong & Yang , 2020 ) , and DARTS-NDS ( Radosavovic et al. , 2019 ) , for which the ground-truth stand-alone performance of a large number of architectures is available . We report the results of our experiments according to two sets of metrics : i ) metrics that directly measure the quality of the super-net , such as the widely-adopted super-net accuracy 2 and a modified Kendall-Tau correlation between the searched architectures and their ground-truth performance , which we refer to as sparse 1Proxy task refers to the tasks that neural architecture search aims to optimize on . 2The mean accuracy over a small set of randomly sampled architectures during super-net training . Under review as a conference paper at ICLR 2021 < latexit sha1_base64= '' dWMZqL35IYT82cSzwIxrSfiYzQ0= '' > AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5idzCZj5rHMzCphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWye1Gv3syUz75Ypf9edAqyTISQVyNPrlr95AkVRQaQnHxnQDP7FhhrVlhNNpqZcammAyxkPadVRiQU2Yza+dojOnDFCstCtp0Vz9PZFhYcxERK5TYDsyy95M/M/rpja+CjMmk9RSSRaL4pQjq9DsdTRgmhLLJ45gopm7FZER1phYF1DJhRAsv7xKWhfVoFat3dUq9es8jiKcwCmcQwCXUIdbaEATCDzAM7zCm6e8F+/d+1i0Frx85hj+wPv8AdUFj04= < /latexit > < latexit sha1_base64= '' xordNtnCPHeHQsW9SFKXfp9hiUg= '' > AAAB8HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmIckS5idzCZD5rHMzIphyVd48aCIVz/Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWyc9xP0s0eppMu2XK37VnwOtkiAnFcjR6Je/egNFUkGlJRwb0w38xIYZ1pYRTqelXmpogskYD2nXUYkFNWE2P3iKzpwyQLHSrqRFc/X3RIaFMRMRuU6B7cgsezPxP6+b2vgqzJhMUkslWSyKU46sQrPv0YBpSiyfOIKJZu5WREZYY2JdRiUXQrD88ippXVSDWrV2V6vUr/M4inACp3AOAVxCHW6hAU0gIOAZXuHN096L9+59LFoLXj5zDH/gff4AeDWQ2g== < /latexit > < latexit sha1_base64= '' 3iE++PYZojxQ9mAgZyyC7qSUlxU= '' > AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5idzCZj5rHMzCphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWye14n72ZKb9csWv+nOgVRLkpAI5Gv3yV2+gSCqotIRjY7qBn9gww9oywum01EsNTTAZ4yHtOiqxoCbM5tdO0ZlTBihW2pW0aK7+nsiwMGYiItcpsB2ZZW8m/ud1UxtfhRmTSWqpJItFccqRVWj2OhowTYnlE0cw0czdisgIa0ysC6jkQgiWX14lrYtqUKvW7mqV+nUeRxFO4BTOIYBLqMMtNKAJBB7gGV7hzVPei/fufSxaC14+cwx/4H3+APbLj2Q= < /latexit > < latexit sha1_base64= '' 3iE++PYZojxQ9mAgZyyC7qSUlxU= '' > AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5idzCZj5rHMzCphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWye14n72ZKb9csWv+nOgVRLkpAI5Gv3yV2+gSCqotIRjY7qBn9gww9oywum01EsNTTAZ4yHtOiqxoCbM5tdO0ZlTBihW2pW0aK7+nsiwMGYiItcpsB2ZZW8m/ud1UxtfhRmTSWqpJItFccqRVWj2OhowTYnlE0cw0czdisgIa0ysC6jkQgiWX14lrYtqUKvW7mqV+nUeRxFO4BTOIYBLqMMtNKAJBB7gGV7hzVPei/fufSxaC14+cwx/4H3+APbLj2Q= < /latexit > < latexit sha1_base64= '' iU9d81SBeMPs61pcpys14gjB6js= '' > AAAB8HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmIckS5idzCZD5rHMzIphyVd48aCIVz/Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWyc9NPpZotXTZNovV/yqPwdaJUFOKpCj0S9/9QaKpIJKSzg2phv4iQ0zrC0jnE5LvdTQBJMxHtKuoxILasJsfvAUnTllgGKlXUmL5urviQwLYyYicp0C25FZ9mbif143tfFVmDGZpJZKslgUpxxZhWbfowHTlFg+cQQTzdytiIywxsS6jEouhGD55VXSuqgGtWrtrlapX+dxFOEETuEcAriEOtxCA5pAQMAzvMKbp70X7937WLQWvHzmGP7A+/wBVi2QxA== < /latexit > < latexit sha1_base64= '' xordNtnCPHeHQsW9SFKXfp9hiUg= '' > AAAB8HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmIckS5idzCZD5rHMzIphyVd48aCIVz/Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWyc9xP0s0eppMu2XK37VnwOtkiAnFcjR6Je/egNFUkGlJRwb0w38xIYZ1pYRTqelXmpogskYD2nXUYkFNWE2P3iKzpwyQLHSrqRFc/X3RIaFMRMRuU6B7cgsezPxP6+b2vgqzJhMUkslWSyKU46sQrPv0YBpSiyfOIKJZu5WREZYY2JdRiUXQrD88ippXVSDWrV2V6vUr/M4inACp3AOAVxCHW6hAU0gIOAZXuHN096L9+59LFoLXj5zDH/gff4AeDWQ2g== < /latexit > < latexit sha1_base64= '' dWMZqL35IYT82cSzwIxrSfiYzQ0= '' > AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5idzCZj5rHMzCphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWye1Gv3syUz75Ypf9edAqyTISQVyNPrlr95AkVRQaQnHxnQDP7FhhrVlhNNpqZcammAyxkPadVRiQU2Yza+dojOnDFCstCtp0Vz9PZFhYcxERK5TYDsyy95M/M/rpja+CjMmk9RSSRaL4pQjq9DsdTRgmhLLJ45gopm7FZER1phYF1DJhRAsv7xKWhfVoFat3dUq9es8jiKcwCmcQwCXUIdbaEATCDzAM7zCm6e8F+/d+1i0Frx85hj+wPv8AdUFj04= < /latexit > < latexit sha1_base64= '' 3iE++PYZojxQ9mAgZyyC7qSUlxU= '' > AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5idzCZj5rHMzCphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWye14n72ZKb9csWv+nOgVRLkpAI5Gv3yV2+gSCqotIRjY7qBn9gww9oywum01EsNTTAZ4yHtOiqxoCbM5tdO0ZlTBihW2pW0aK7+nsiwMGYiItcpsB2ZZW8m/ud1UxtfhRmTSWqpJItFccqRVWj2OhowTYnlE0cw0czdisgIa0ysC6jkQgiWX14lrYtqUKvW7mqV+nUeRxFO4BTOIYBLqMMtNKAJBB7gGV7hzVPei/fufSxaC14+cwx/4H3+APbLj2Q= < /latexit > < latexit sha1_base64= '' iU9d81SBeMPs61pcpys14gjB6js= '' > AAAB8HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmIckS5idzCZD5rHMzIphyVd48aCIVz/Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWyc9NPpZotXTZNovV/yqPwdaJUFOKpCj0S9/9QaKpIJKSzg2phv4iQ0zrC0jnE5LvdTQBJMxHtKuoxILasJsfvAUnTllgGKlXUmL5urviQwLYyYicp0C25FZ9mbif143tfFVmDGZpJZKslgUpxxZhWbfowHTlFg+cQQTzdytiIywxsS6jEouhGD55VXSuqgGtWrtrlapX+dxFOEETuEcAriEOtxCA5pAQMAzvMKbp70X7937WLQWvHzmGP7A+/wBVi2QxA== < /latexit > < latexit sha1_base64= '' xordNtnCPHeHQsW9SFKXfp9hiUg= '' > AAAB8HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmIckS5idzCZD5rHMzIphyVd48aCIVz/Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWyc9xP0s0eppMu2XK37VnwOtkiAnFcjR6Je/egNFUkGlJRwb0w38xIYZ1pYRTqelXmpogskYD2nXUYkFNWE2P3iKzpwyQLHSrqRFc/X3RIaFMRMRuU6B7cgsezPxP6+b2vgqzJhMUkslWSyKU46sQrPv0YBpSiyfOIKJZu5WREZYY2JdRiUXQrD88ippXVSDWrV2V6vUr/M4inACp3AOAVxCHW6hAU0gIOAZXuHN096L9+59LFoLXj5zDH/gff4AeDWQ2g== < /latexit > < latexit sha1_base64= '' dWMZqL35IYT82cSzwIxrSfiYzQ0= '' > AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5idzCZj5rHMzCphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+mfntR6oNU/LeThIaCjyULGYEWye1Gv3syUz75Ypf9edAqyTISQVyNPrlr95AkVRQaQnHxnQDP7FhhrVlhNNpqZcammAyxkPadVRiQU2Yza+dojOnDFCstCtp0Vz9PZFhYcxERK5TYDsyy95M/M/rpja+CjMmk9RSSRaL4pQjq9DsdTRgmhLLJ45gopm7FZER1phYF1DJhRAsv7xKWhfVoFat3dUq9es8jiKcwCmcQwCXUIdbaEATCDzAM7zCm6e8F+/d+1i0Frx85hj+wPv8AdUFj04= < /latexit > Kendall-Tau ; ii ) proxy metrics such as the ability to surpass random search and the stand-alone accuracy of the model found by the WS-NAS algorithm . Via our extensive experiments ( over 700 GPU days ) , we uncover that ( i ) the training behavior of a super-net drastically differs from that of a standalone network , e.g. , in terms of feature statistics and loss landscape , thus allowing us to define training factor settings , e.g. , for batch-normalization ( BN ) and learning rate , that are better suited for super-nets ; ( ii ) while some neglected factors , such as the number of training epochs , have a strong impact on the final performance , others , believed to be important , such as path sampling , only have a marginal effect , and some commonly-used heuristics , such as the use of low-fidelity estimates , negatively impact it ; ( iii ) the commonly-adopted super-net accuracy is unreliable to evaluate the super-net quality . Altogether , our work is the first to systematically analyze the impact of the diverse factors of super-net design and training , and we uncover the factors that are crucial to design a super-net , as well as the non-important ones . Aggregating these findings allows us to boost the performance of simple weight-sharing random search to the point where it reaches that of complex state-of-the-art NAS algorithms across all tested search spaces . We will release our code and trained models so as to establish a solid baseline to facilitate further research . 2 PRELIMINARIES AND RELATED WORK . We first introduce the necessary concepts that will be used throughout the paper . As shown in Figure 1 ( a ) , weight-sharing NAS algorithms consist of three key components : a search algorithm that samples an architecture from the search space in the form of an encoding , a mapping function fproxy that maps the encoding into its corresponding neural network , and a training protocol for a proxy task Pproxy for which the network is optimized . To train the search algorithm , one needs to additionally define the mapping function fws that generates the shared-weight network . Note that the mapping fproxy frequently differs from fws , since in practice the final model contains many more layers and parameters so as to yield competitive results on the proxy task . After fixing fws , a training protocol Pws is required to learn the super-net . In practice , Pws often hides factors that are critical for the final performance of an approach , such as hyper-parameter settings or the use of data augmentation strategies to achieve state-of-the-art performance ( Liu et al. , 2019b ; Chu et al. , 2019 ; Zela et al. , 2020a ) . Again , Pws may differ from Pproxy , which is used to train the architecture that has been found by the search . For example , our experiments reveal that the learning rate and the total number of epochs frequently differ due to the different training behavior of the super-net and stand-alone architectures . Many strategies have been proposed to implement the search algorithm , such as reinforcement learning ( Zoph & Le , 2017 ; Zoph et al. , 2018 ) , evolutionary algorithms ( Real et al. , 2017 ; Miikkulainen et al. , 2019 ; So et al. , 2019 ; Liu et al. , 2018 ; Lu et al. , 2018 ) , gradient-based optimization ( Liu et al. , 2019b ; Xu et al. , 2020 ; Li et al. , 2020 ) , Bayesian optimization ( Kandasamy et al. , 2018 ; Jin et al. , 2019 ; Zhou et al. , 2019 ; Wang et al. , 2020 ) , and separate performance predictors ( Liu et al. , 2018 ; Luo et al. , 2018 ) . Until very recently , the common trend to evaluate NAS consisted of reporting the searched architecture ’ s performance on the proxy task ( Xie et al. , 2019 ; Real et al. , 2019 ; Ryoo et al. , 2020 ) . This , however , hardly provides real insights about the NAS algorithms themselves , because of the many components involved in them . Many factors that differ from one algorithm to another can influence the performance . In practice , the literature even commonly compares NAS methods that employ different protocols to train the final model . Li & Talwalkar ( 2019 ) and Yu et al . ( 2020b ) were the first to systematically compare different algorithms with the same settings for the proxy task and using several random initializations . Their surprising results revealed that many NAS algorithms produce architectures that do not significantly outperform a randomly-sampled architecture . Yang et al . ( 2020 ) highlighted the importance of the training protocol Pproxy . They showed that optimizing the training protocol can improve the final architecture performance on the proxy task by three percent on CIFAR-10 . This non-trivial improvement can be achieved regardless of the chosen sampler , which provides clear evidence for the importance of unifying the protocol to build a solid foundation for comparing NAS algorithms . In parallel to this line of research , the recent series of “ NASBench ” works ( Ying et al. , 2019 ; Zela et al. , 2020b ; Dong & Yang , 2020 ) proposed to benchmark NAS approaches by providing a complete , tabular characterization of a search space . This was achieved by training every realizable stand-alone architecture using a fixed protocol Pproxy . Similarly , other works proposed to provide a partial characterization by sampling and training a sufficient number of architectures in a given search space using a fixed protocol ( Radosavovic et al. , 2019 ; Zela et al. , 2020a ; Wang et al. , 2020 ) . While recent advances for systematic evaluation are promising , no work has yet thoroughly studied the influence of the super-net training protocol Pws and the mapping function fws . Previous works ( Zela et al. , 2020a ; Li & Talwalkar , 2019 ) performed hyper-parameter tuning to evaluate their own algorithms , and focused only on a few parameters . We fill this gap by benchmarking different choices of Pws and fws and by proposing novel variations to improve the super-net quality . Recent works have shown that sub-nets of super-net training can surpass some human designed models without retraining ( Yu et al. , 2020a ; Cai et al. , 2020 ) and that reinforcement learning can surpass the performance of random search ( Bender et al. , 2020 ) . However , these findings are still only shown on MobileNet-like search spaces where we only search for the size of convolution kernels and the channel ratio for each layer . This is an effective approach to discover a compact network , but it does not change the fact that on cell-based search space super-net quality remains low . 3 EVALUATION METHODOLOGY . We first isolate 14 factors that need to be considered during the design and training of a super-net , and then introduce the metrics to evaluate the quality of the trained super-net . Note that these factors are agnostic to the search policy that is used after training the super-net . 3.1 DISENTANGLING THE SUPER-NET FROM THE SEARCH ALGORITHM . Our goal is to evaluate the influence of the super-net mapping fws and weight-sharing training protocol Pws . As shown in Figure 2 , fws translates an architecture encoding , which typically consists of a discrete number of choices or parameters , into a neural network . Based on a well-defined mapping , the super-net is a network in which every sub-path has a one-to-one mapping with an architecture encoding ( Pham et al. , 2018 ) . Recent works ( Xu et al. , 2020 ; Li et al. , 2020 ; Ying et al. , Figure 2 : Constructing a super-net 2019 ) separate the encoding into cell parameters , which define the basic building blocks of a network , and macro parameters , which define how cells are assembled into a complete architecture . Weight-sharing mapping fws . To make the search space manageable , all cell and macro parameters are fixed during the search , except for the topology of the cell and its possible operations . However , the exact choices for each of these fixed factors differ between algorithms and search spaces . We report the common factors in the left part of Table 1 . They include various implementation choices , e.g. , the use of convolutions with a dynamic number of channels ( Dynamic Channeling ) , super-convolutional layers that support dynamic kernel sizes ( OFA Kernel ) ( Cai et al. , 2020 ) , weight-sharing batchnormalization ( WSBN ) that tracks independent running statistics and affine parameters for different incoming edges ( Luo et al. , 2018 ) , and path and global dropout ( Pham et al. , 2018 ; Luo et al. , 2018 ; Liu et al. , 2019b ) . They also include the use of low-fidelity estimates ( Elsken et al. , 2019 ) to reduce the complexity of super-net training , e.g. , by reducing the number of layers ( Liu et al. , 2019b ) and channels ( Yang et al. , 2020 ; Chen et al. , 2019a ) , the portion of the training set used for super-net training ( Liu et al. , 2019b ) , or the batch size ( Liu et al. , 2019b ; Pham et al. , 2018 ; Yang et al. , 2020 ) . Weight-sharing protocol Pws . Given a mapping fws , different training protocols Pws can be employed to train the super-net . Protocols can differ in the training hyper-parameters and the sampling strategies they rely on . We will evaluate the different hyper-parameter choices listed in the right part of Table 1 . This includes the initial learning rate , the hyper-parameters of batch normalization , the total number of training epochs , and the amount of weight decay . We randomly sample one path to train the super-net ( Guo et al. , 2019 ) , which is also known as single-path one-shot ( SPOS ) or Random-NAS ( Li & Talwalkar , 2019 ) . The reason for this choice is that Random-NAS is equivalent to the initial state of many search algorithms ( Liu et al. , 2019b ; Pham et al. , 2018 ; Luo et al. , 2018 ) , some of which even freeze the sampler training so as to use random sampling to warm-up the super-net ( Xu et al. , 2020 ; Dong & Yang , 2019b ) . Note that we also evaluated two variants of Random-NAS , but found their improvement to be only marginal . Please see Appendix C.2 for more detail . In our experiments , for the sake of reproducibility , we ensure that Pws and Pproxy , as well as fws and fproxy , are as close to each other as possible . For the hyper-parameters of Pws , we cross-validate each factor following the order in Table 1 , and after each validation , use the value that yields the best performance in Pproxy . For all other factors , we change one factor at a time . Search spaces . We use three commonly-used search spaces , for which a large number of stand-alone architectures have been trained and evaluated on CIFAR-10 ( Krizhevsky et al. , 2009 ) to obtain their ground-truth performance . In particular , we use NASBench-101 ( Ying et al. , 2019 ) , which consists of 423 , 624 architectures and is compatible with weight-sharing NAS ( Yu et al. , 2020b ; Zela et al. , 2020b ) ; NASBench-201 ( Dong & Yang , 2020 ) , which contains more operations than NASBench-101 but fewer nodes ; and DARTS-NDS ( Radosavovic et al. , 2019 ) that contains over 1013 architectures of which a subset of 5000 models was sampled and trained in a stand-alone fashion . See Appendix A.2 for a detailed discussion .

Summary: This work analyzes commonly used heuristics for training the supernet in weight sharing NAS. The authors first proposes a new metric, sparse Kendall-Tau, to measure the quality of the supernet. Then extensive experiments are conducted on three NAS benchmarks to empirically evaluate the heuristics, and pick the best settings. To highlight the significance of the training quality of supernet, the author showed that random search, when combined with the best settings, can performs competitive to SOTA results.