Improving Random-Sampling Neural Architecture Search by Evolving the Proxy Search Space

1 INTRODUCTION . Neural architecture search ( NAS ) has been successfully utilized to discover novel DNN architectures in complex search spaces and outperformed human-crafted designs . Early NAS works like NASNet ( Zoph et al . ( 2018 ) ) and AmoebaNet ( Real et al . ( 2019 ) ) used reinforcement learning or evolutionary algorithms to search for the DNN architectures by training a substantial amount of independent network architectures from scratch . Although these searched architectures can deliver high accuracy , they come with tremendous computation and time costs . Therefore , researchers gradually shift their focuses to one-shot NAS , which is more efficient and can deliver satisfying outputs within a few GPU-days . There are two main types of one-shot NAS . One is the differentiable NAS ( DNAS ) , such as Liu et al . ( 2019b ) ; Cai et al . ( 2018 ) ; Xie et al . ( 2018 ) ; Dong & Yang ( 2019b ) ; Xu et al . ( 2019 ) ; Chen et al . ( 2019a ) , which uses a continuous relaxation of the architecture representations and introduces architecture parameters to distinguish the architectures . The other is Random-Sampling NAS ( RandomNAS ) , such as Li & Talwalkar ( 2019 ) ; Chen et al . ( 2019b ) ; Zhang et al . ( 2020 ) ; Guo et al . ( 2019 ) ; Bender ( 2019 ) ; Yang et al . ( 2020 ) . RandomNAS approaches typically have two phases : ( 1 ) Training phase : in each iteration , RandomNAS randomly samples one architecture or a set of architectures and updates their shared weights in the supernet ; ( 2 ) Search phase : after supernet training , the desired architectures are selected based on their performance ranking on the validation dataset using inherited weights from the supernet , which is called weight-sharing performance . Finally , the selected architectures will be retrained from scratch to get their actual ( retrained ) performance for deployment . Compared to DNAS , RandomNAS usually consumes less GPU memories by partially updating the weights . Also , it generates multiple target architectures , while DNAS generally retrieves a single architecture based on the maxima of the representation distribution . There are , however , two major drawbacks preventing RandomNAS from achieving higher search efficiency . First , although RandomNAS achieves a promising ranking correlation between the weight-sharing estimation and the retrained performance over all architecture candidates , it delivers a low ranking correlation among “ good '' architectures ( e.g. , top-20 % -performing architectures among the search space ) that researchers are more interested in . Second , by following the RandomNAS approach , smaller network architectures ( with less parameters ) tend to converge faster than the larger ones , which significantly degrades the ranking correlation . To address the drawbacks of RandomNAS , we first introduce a proxy search space ( PS ) : a subset of the architectures flexibly sampled from the global search space ( GS ) to study the features of RandomNAS . We then evaluate the RandomNAS with the proposed PS using the NASBench-201 benchmark ( Dong & Yang ( 2020 ) ) and notice two interesting phenomena : ( 1 ) When uniformly sampling a PS from a global search space , RandomNAS in the PS will maintain a similar ranking correlation with the one in the GS , even the size of the PS is extremely small ( e.g . 16 architectures ) . ( 2 ) If the PS consists of the “ good '' architectures , the PS-based search can significantly improve the ranking correlation among the PS compared to the RandomNAS trained in GS and validated in PS . Based on these observations , the PS constructed from “ good ” architectures can help overcome the first drawback of RandomNAS to search for more promising architectures . So , the remaining question is how to find a suitable PS containing sufficient “ better '' architectures ? In this paper , we consider it a natural selection problem and solve it by the evolutionary algorithm . The architectures in the initial PS are iteratively evolved and gradually upgraded , while the average ranking of the architectures in the PS is improved . Meanwhile , it also helps improve the ranking correlation of the PS . We propose a new RandomNAS approach , named Evolving the Proxy Search Space ( EPS ) . EPS runs in three stages iteratively : Training the supernet by randomly sampling from a PS ; Validating the architectures among the PS on a subset of the validation dataset in the training interval ; Evolving the PS by a tournament selection evolutionary algorithm with the aging mechanism . In this way , EPS gradually includes more high quality architectures in the proxy search space , while improves its ranking correlation . To solve the second issue in which smaller architectures converge faster than larger ones , we introduce a simple model-size-based regularization in the final selection stage . Our result on NASBench-201 shows 17.2 % improvement in ranking correlation measured by Spearman ’ s ρ by adding the regularization on NASBench-201 . In the experiments , we demonstrate that EPS delivers a near-optimal performance on NASBench201 . We also extend the EPS on DARTS search space . By using the 5-search-runs measurement , EPS demonstrates a robust search ability compared with the previous works in 8 hours search time with little hyper-parameters fine-tuning effort . Also , EPS is evaluated in 4 DARTS sub search space ( Zela et al . ( 2019 ) ) using 3 datasets , on which DARTS easily fails . EPS surpasses the DARTSADA and DARTS-ES on most cases and can often find the global state-of-the-art architectures . EPS also shows a high performance on a language modeling task which consolidates the generalization ability and robustness of EPS . 2 RANDOMNAS ON NASBENCH-201 . In this section , we present two major drawbacks we found in the existing RandomNAS methods and investigate the ranking correlation using a proxy search space . Proxy search space ( PS ) is a subset of the global search space ( GS ) and the RandomNAS is more flexible to be analyzed by training in the PS including different architectures from the GS . We run the following experiments on NASBench201 , which is a unified and fair benchmark designed for NAS algorithm evaluation and contains the ground-truth architecture accuracy on three datasets . The GS of NASBench-201 contains 15,625 architectures and they are constructed by 4 nodes and 5 associated operation options . ( Please refer to Fig . 1 in Dong & Yang ( 2020 ) for details . ) 2.1 RANKING CORRELATION IN DIFFERENT PROXY SEARCH SPACES First , we want to study whether RandomNAS in a PS with architectures uniformly sampled from the GS can achieve a higher ranking performance than the RandomNAS in the GS . We design an experiment to train the RandomNAS in the PS with 10 different sizes from 24 to 213 independently and compare the results with the RandomNAS in the GS . The experiment for each scenario is conducted 10 times with different random seeds ( 10×11 total runs ) . We calculate the Spearman ’ s ρ ranking correlation between the RandomNAS validation loss and the ground-truth test accuracy with 16 architectures randomly sampled from each search space . Fig . 1a shows the raw data with ± one standard deviation error bar for different proxy search space . The ranking correlations are close and there is no significant difference among them through statistical testing with a Pearson coefficient as -0.10 . We observe that when the architectures are randomly sampled from the global search space , RandomNAS in a PS maintains a similar global-ranking performance with RandomNAS in the GS . Second , although RandomNAS trained in the GS achieves 0.783 Spearman ’ s ρ when validating in the GS , it shows a poor Spearman ’ s ρ ( 0.347 ) in top-20 % of the GS ( which consists of top-20 % performing architectures ranked by ground-truth accuracy from the GS ) . Hence , we ’ d like to see if RandomNAS in a PS can have a better ranking performance . Fig . 1b shows another six different scenarios , demonstrating the ranking correlation differences when sampling from GS or PS . The first three are : the RandomNAS trained in the GS with Spearman ’ s ρ calculated for 64 architectures randomly samples from top-100 % ( global ) , top-60 % , top-20 % in the GS independently . The other three are from PS : the RandomNAS trained in a PS with 256 architectures sampled from top100 % , top-60 % , top-20 % the from GS and Spearman ’ s ρ calculated with 64 architectures randomly sampled from the same PS . It shows that the RandomNAS in the GS achieves 0.783 ( the blue line ) Spearman ’ s ρwith the validation architectures sampled from global . However , it only achieves 0.347 ( purple ) with the validation architectures sampled from top-20 % . On the contrast , the RandomNAS in the PS shows that its Spearman ’ s ρ curve under PS top-100 % ( orange , 0.785 ) is similar with the GS top-100 % , which is consistent with the Fig . 1b results . However , the Spearman ’ s ρ curve of the PS top-60 % ( red , 0.687 ) and top-20 % ( brown , 0.607 ) both surpass the GS top-60 % ( green , 0.530 ) and GS top-20 % . We suspect that such an improvement may be due to the better constructed PS sampled from top performing GS , an insight we will take advantage of in designing our new solution below . 2.2 SMALL ARCHITECTURES GAIN LOWER LOSS . We observe another drawback of RandomNAS in that smaller architectures inside the supernet converge faster and tend to gain lower validation loss than the larger ones in the early training phase . Fig . 2a shows the validation losses of 4000 architectures from RandomNAS in the GS after training . Although the architectures with a model size of 1.05Mb achieves higher accuracy ( queried from the ground truth in NASBench201 ) , the Pareto front ( shown as orange dots ) only reaches the 0.83Mb architecture . One possible explanation is that the light-weight operations converge faster . Fig . 2b shows the average size of the validation Pareto front among 320 architectures in each validation interval . We can see that the average sizes of the Pareto-front architectures are small in the early search stage and will increase after more iterations , meaning that the larger architectures gradually begin to converge . 3 OUR METHOD Algorithm 1 : EPS Initialize a supernet Initialize an empty architecture population queue , Q_pop Initialize an empty history for i = 1 , 2 , · · · , P do new_architecture← RandomInitArch ( supernet ) Enqueue ( Q_pop , new_architecture ) end sample_set← RandomSample ( Q_pop , S ) for i = 1 , 2 , · · · , T _iter do architecture ← RandomSample ( sample_set , 1 ) TrainOneBatch ( supernet , architecture ) if i % I_val = 0 then for k = 1 , 2 , · · · , S do architecture← sample_set [ k ] val_loss← ValOneBatch ( supernet , architecture ) architecture.loss← val_loss end Sort sample_set by architecture.loss in ascending order history.append ( sample_set ) for k in 1,2 ... M do new_architecture← Mutate ( sample_set [ k ] ) Enqueue ( Q_pop , new_architecture ) Dequeue ( Q_pop ) end sample_set← RandomSample ( Q_pop , S ) end end selected_architectures← ArchitectureSelection ( history , T ) ArchitectureRetrain ( selected_architectures ) Motivated by the observations in 2.1 , we propose a simple yet effective random sampling approach , called EPS , to iteratively improve the architecture quality within the proxy search space . Our proposed solution can distinguish the “ better '' architectures with higher ranking correlation compared to the existing solutions using global search . Algorithm 1 shows the overall flow of EPS . Main Hyper-parameters . ( 1 ) T_iter : the number of iterations for supernet training . ( 2 ) I_val : validation interval . ( 3 ) P : maximum population size , where the population is a proxy search space . ( 4 ) S : sample size ( the number of sampled architectures from population ) . ( 5 ) M : the number of mutated architectures . The ablation study on the search hyperparameters will be discussed in Section 4 . Overall Flow . EPS performs three major stages iteratively , as training , validation and evolving , followed by a final architecture selection and retraining . At the beginning , we build a supernet which contains all possible architectures in the search space and initialize Q_pop and sample_set . RandomInitArch uniformly select a architecture from the GS , and then the architecture is enqueued into the Q_pop . Q_pop is the proxy search space and sample_set is randomly sampled from Q_pop for the tournament selection . ( 1 ) Training . The supernet is trained by a total of T_iter iterations . In each iteration , one architecture is sampled from the sample_set and its weights are updated in the weight-sharing supernet by gradient descent . ( 2 ) Validation . Every I_val iterations of training , the architectures inside the sample_set will be evaluated sequentially . In each validation iteration , one architecture is evaluated on a batch of the randomly sampled validation data , and its validation loss is recorded . We validate the architecture on a single batch instead of the whole validation dataset to find a trade-off between the search time and completeness of validation . ( 3 ) Evolving . The architecture inside sample_set are sorted in ascending order according to their latest loss values , and the top-M architectures are mutated and en-queued . Meanwhile , it removes M oldest architecture from Q_pop . The mutation operation is described in Appendix B . ( 4 ) Selection . After supernet training , we revisit the history of evolution and select all the “ winner '' architectures which get the lowest loss in the Q_pop in the each of last T validation intervals . However , if T is large , it still remains numerous number of “ winners '' . Based on the observation from Section 2.2 , we propose a simple size regularization : architecture.loss = architecture.loss+ αe−βsize ( architecture ) /max_size . max_size is the maximum architecture size in the global search space . The regularization imposes a penalty on small architectures , and the experiment results in Section 4 shows the significant improvement from the regularization . By adding the regularization , we select the one with the lowest regularized loss as the ideal architecture from winners .

Summary: This paper proposes Evolving the Proxy Search Space (EPS) as a new RandomNAS-based approach. The goal is to find an effective proxy search space (PS) that is only a small subset of GS to dramatically improve RandomNAS’s search efficiency while at the same time keeping a good correlation for the top-performing architectures. EPS runs in three stages iteratively: Training the supernet by randomly sampling from a PS; Validating the architectures among the PS on a subset of the validation dataset in the training interval; Evolving the PS by a tournament selection evolutionary algorithm with the aging mechanism.