Deeper Insights into Weight Sharing in Neural Architecture Search

1 INTRODUCTION . Learning to design neural architectures automatically has aroused wide interests recently due to its success in many different machine learning tasks . One stream of neural architectures search ( NAS ) methods is based on reinforcement learning ( RL ) ( Zoph & Le , 2016 ; Zoph et al. , 2018 ; Tan et al. , 2019 ) , where a neural architecture is built from actions and its performance is used as reward . This approach usually demands considerable computation power — each search process takes days with hundreds of GPUs . Population based algorithms ( Gaier & Ha , 2019 ; Liang et al. , 2018 ; Jaderberg et al. , 2017 ) are another popular approach for NAS , new trials could inherit neural architecture from better performing ones as well as their weights , and mutate the architecture to explore better ones . It also has high computation cost . To speed up the search process , a family of methods attracts increasing attention with greatly reduced computation ( Pham et al. , 2018 ; Liu et al. , 2018c ; Bender et al. , 2018 ) . Instead of training every child model , they build a single model , called super-net , from neural architecture search space , and maintain a single copy of weights on the super-net . Several training approaches have been proposed on this model , e.g. , training with RL controller ( Pham et al. , 2018 ) , training by applying dropout ( Bender et al. , 2018 ) or architecture weights on candidate choices ( Liu et al. , 2018c ) . In these approaches , weight-sharing is the key for the speedup . However , weight sharing has no theoretical guarantee and its impact has not been well studied before . The directions of improving such methods would be more clear if some key questions had been answered : 1 ) How far is the accuracy of found architecture from the best one within search space ? 2 ) Could the best architecture be stably found in multiple runs of search process ? 3 ) How does weight sharing affect the accuracy and stability of the found architecture ? In this paper , we answer the above-mentioned questions using comprehensive experiments and analysis . To understand the behavior of weight sharing approaches , we use a small search space , which makes it possible to have ground truth for comparison . It is a simplified NAS problem , therefore , making it easy to show the ability of the NAS algorithms with weight sharing . As a result , we find that the rank of child models is very unstable in different runs of the search process , and also very different from ground truth . In fact , the instability 1 commonly exists not only in different runs , 1In this paper , we use instability and variance interchangeably . but also in consecutive training epochs within the same run . Also worthy of note , in spite of high variance , we can extract statistical information from the variance , the statistics can be innovatively leveraged to prune search space and improve the search result . To further understand where the variance comes from , we record and analyze more metric data from the experiments . It is witnessed that some child models have interference with each other , and the degree of this interference varies depending on different child models . At the very end of the super-net training , training each child model in one mini-batch can make this model be the best performing one on the validation data . Based on the insights , we further explore partial weight sharing , that is , each child model could selectively share weights with others , rather than all of them sharing the same copy of weights . It can be seen as reduced degree of weight sharing . One method we have explored is sharing weights of common prefix layers among child models . Another method is to cluster child models into groups , each of which shares a copy of weights . Experiment results show that partial weight sharing makes the rank of child models more stable and becomes closer to ground truth . It implies that with proper degree or control of weight sharing , better child models can be more stably found . To summarize , our main contributions are as follows : • We define new metrics for evaluating the performance of the NAS methods based on weight sharing , and propose a down-scaled search space which makes it possible to have a deeper analysis by comparing it with ground truth . • We design various experiments , and deliver some interesting observations and insights . More importantly , we reveal that valuable statistics can be extracted from training the super-net , which can be leveraged to improve performance . • We take a step further to explain the reasons of high variance . Then we use decreased degree of weight sharing , which shows lower variance and better performance , to support the reasoning . 2 RELATED WORKS . Neural Architecture Search ( NAS ) is invented to relieve human experts from laborious job of engineering neural network components and architectures by automatically searching optimal neural architecture from a human-defined search space . Arguably , the recent growing interesting in NAS research begins from the work by Zoph and Le ( Zoph & Le , 2016 ) where they train a controller using policy gradients ( Williams , 1992 ) to discover and generate network models that achieve state-ofthe-art performance . Following these works , there is a growing interest in using RL in NAS ( Pham et al. , 2018 ; Baker et al. , 2017 ; Tan et al. , 2019 ; Zoph et al. , 2018 ) . There have also been studies in evolutionary approaches ( Real et al. , 2019 ; 2017 ; Miikkulainen et al. , 2017 ; Xie & Yuille , 2017 ; Liu et al. , 2018b ) . Most of these works still demand high computational cost that is not affordable for large networks or datasets . Weight sharing approaches Weight sharing means sharing architecture weights among different components or models . Pham et al . ( 2018 ) combined this approach with previous work of NAS ( Zoph & Le , 2016 ) and proposed Efficient Neural Architecture Search ( ENAS ) , where a super-net is constructed which contains every possible architecture in the search space as its child model , and thus all the architectures sampled from this super-net share the weights of their common graph nodes . It significantly reduces the computational complexity of NAS by directly training and evaluating sampled child models directly on the shared weight . After the training is done , a subset of child models is chosen and they are either finetuned or trained from scratch to get the final model . Many follow-up works leverage weight sharing as a useful technique that can be decoupled from RL controllers , including applying dropout on candidate choices ( Bender et al. , 2018 ) , converting the discrete search space into a differentiable one ( Liu et al. , 2018c ; Wu et al. , 2018 ; Xie et al. , 2019 ) , searching via sparse optimization ( Zhang et al. , 2018 ) , and directly searching for child models for large-scale target tasks and hardwares ( Cai et al. , 2019 ) . Previous studies on stability of weight sharing All the weight-sharing approaches are based on the assumption that the rank of child models obtained by evaluating a child model of the trained super-net is valid , or at least , capable of finding one of the best child models in the search space . However , this assumption does not generally hold . For example , Guo et al . ( 2019 ) believed that child models are deeply coupled during optimization , causing high interference among each other . Sciuto et al . ( 2019 ) discovered that there is little correlation between the rank found by weight sharing and rank of actual performance . Anonymous ( 2020 ) also conducted a benchmark experiment that shows a similar instability when the search space contains thousands of child models , by leveraging ground truth results measured by Ying et al . ( 2019 ) . However on the other hand , research on transfer learning ( Razavian et al. , 2014 ) , where a particular model trained on a particular task can work well on another task , and multitask learning ( Luong et al. , 2015 ) , where multiple models trained for multiple tasks share the same weights during training , suggest otherwise and encourage the weights to be shared among child models , to reduce the long training time from scratch to convergence ( Pham et al. , 2018 ) . Therefore , in this paper we show whether weight sharing helps and why , using comprehensive experiments . 3 WEIGHT-SHARING : VARIANCE AND INVARIANCE . 3.1 METHODOLOGY . The space of a typical neural architecture search task usually has more than 1010 different child models ( Tan et al. , 2019 ; Liu et al. , 2018a ; 2019 ) , thus , it is impossible to train them all , which leads to the problem that without ground truth it is hard to assess how good the found child model is in the search space . To solve this problem , we down-scale search space under the assumption that small search space is easier than large search space , if the search methods works in large search space they are also supposed to work in small search space . Following DARTS ( Liu et al. , 2018c ) , we design a search space for a cell , as shown in Figure 1 , and stack four cells each of which has the same chosen structure , forming a convolutional neural network . A cell is defined as a directed acyclic graph ( DAG ) of n nodes ( tensors ) x1 , . . . , xn . A cell starts with x0 , which is the output tensor of its previous cell fed through a 1x1 conv layer to match the targeted number of channels in the current layer . The output of the cell is simply the sum of x1 , . . . , xn . The DAG is designed to be densely connected , i.e. , xj = ∑ 0≤i < j O ( i , j ) ( xi ) ( 1 ) where O ( i , j ) is the selected operation at edge ( i , j ) . In the down-scaled search space , each cell contains only two nodes ( i.e. , n = 2 ) and O ( i , j ) is one of the four primitive operations in Figure 1 . Thus , a child model only has 43 = 64 possible choices , which makes it easy to have ground truth . For convenience , we also name all the child models with three digits ( each digit is in [ 1 , 4 ] ) , denoting the choice of O ( 0,1 ) , O ( 0,2 ) , O ( 1,2 ) respectively . As introduced in the previous section , sharing a single copy of weights can be seen as training an expanded super-net . To better understand the effect of weight sharing , we simplify the training process . Specifically , we uniformly generate child models . Each mini-batch trains one child model and only the weights of this model are updated by back-propagation . After training the shared weights for a number of epochs , we use these shared weights to evaluate the performance of all child models on the validation set . On the other hand , the ground truth performance of each child model is obtained by training each of them independently from scratch with the same setting as weight sharing , and averaging over 10 runs with different random seeds for initializations . The lookup table can be found in Appendix C. For the rest of the experiments listed in this paper , if not otherwise specified , the models are trained with the dataset of CIFAR-10 on an NVidia K80 GPU . We use SGD with momentum 0.9 and weight decay 10−3 as our optimizer . The initial learning rate is set to 0.025 and annealed down to 0.001 following a cosine schedule without restart ( Loshchilov & Hutter , 2016 ) . The batch size is set to 256 . Number of epochs is 200 . Detailed experiment settings are described in Appendix A .

Summary: Many NAS methods rely on weight sharing. Notably in ENAS, a single weight tensor is used for all candidate operations each edge of a cell. In this paper, the authors take a small NAS search space (64 possible networks) and train each network separately to obtain their individual rankings. They then examine how this ranking correlates when  the same network is trained as part of a super-net with weight sharing, as in NAS algorithms.