PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search

Differentiable architecture search ( DARTS ) provided a fast solution in finding effective network architectures , but suffered from large memory and computing overheads in jointly training a super-network and searching for an optimal architecture . In this paper , we present a novel approach , namely , Partially-Connected DARTS , by sampling a small part of super-network to reduce the redundancy in exploring the network space , thereby performing a more efficient search without comprising the performance . In particular , we perform operation search in a subset of channels while bypassing the held out part in a shortcut . This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels . We alleviate it using edge normalization , which adds a new set of edge-level parameters to reduce uncertainty in search . Thanks to the reduced memory cost , PC-DARTS can be trained with a larger batch size and , consequently , enjoys both faster speed and higher training stability . Experimental results demonstrate the effectiveness of the proposed method . Specifically , we achieve an error rate of 2.57 % on CIFAR10 with merely 0.1 GPU-days for architecture search , and a state-of-the-art top-1 error rate of 24.2 % on ImageNet ( under the mobile setting ) using 3.8 GPU-days for search . Our code has been made available at https : //github.com/yuhuixu1993/PC-DARTS . 1 INTRODUCTION . Neural architecture search ( NAS ) emerged as an important branch of automatic machine learning ( AutoML ) , and has been attracting increasing attentions from both academia and industry . The key methodology of NAS is to build a large space of network architectures , develop an efficient algorithm to explore the space , and discover the optimal structure under a combination of training data and constraints ( e.g. , network size and latency ) . Different from early approaches that often incur large computation overheads ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Real et al. , 2019 ) , recent oneshot approaches ( Pham et al. , 2018 ; Liu et al. , 2019 ) have reduced the search costs by orders of magnitudes , which advances its applications to many real-world problems . In particular , DARTS ( Liu et al. , 2019 ) converts the operation selection into weighting a fixed set of operations . This makes the entire framework differentiable to architecture hyper-parameters and thus the network search can be efficiently accomplished in an end-to-end fashion . Despite its sophisticated design , DARTS is still subject to a large yet redundant space of network architectures and thus suffers from heavy memory and computation overheads . This prevents the search process from using larger batch sizes for either speedup or higher stability . Prior work ( Chen et al. , 2019 ) proposed to reduce the search space , which leads to an approximation that may sacrifice the optimality of the discovered architecture . ∗ This work was done when Yuhui Xu and Xin Chen were interns at Huawei Noah ’ s Ark Lab . Qi Tian is the corresponding author . This work was supported in part by the National NSFC under Grant Nos . 61971285 , 61425011 , 61529101 , 61622112 , 61720106001 , 61932022 , and in part by the Program of Shanghai Academic Research Leader under Grant 17XD1401900 . We thank Longhui Wei and Bowen Shi for instructive discussions . In this paper , we present a simple yet effective approach named Partially-Connected DARTS ( PCDARTS ) to reduce the burdens of memory and computation . The core idea is intuitive : instead of sending all channels into the block of operation selection , we randomly sample a subset of them in each step , while bypassing the rest directly in a shortcut . We assume the computation on this subset is a surrogate approximating that on all the channels . Besides the tremendous reduction in memory and computation costs , channel sampling brings another benefit – operation search is regularized and less likely to fall into local optima . However , PC-DARTS incurs a side effect , where the selection of channel connectivity would become unstable as different subsets of channels are sampled across iterations . Thus , we introduce edge normalization to stabilize the search for network connectivity by explicitly learning an extra set of edge-selection hyper-parameters . By sharing these hyper-parameters throughout the training process , the sought network architecture is insensitive to the sampled channels across iterations and thus is more stable . Benefiting from the partial connection strategy , we are able to greatly increase the batch size . Specifically , as only 1/K of channels are randomly sampled for an operation selection , it reduces the memory burden by almost K times . This allows us to use a K times larger batch size during search , which not only accelerates the network search but also stabilizes the process particularly for largescale datasets . Experiments on benchmark datasets demonstrate the effectiveness of PC-DARTS . Specifically , we achieve an error rate of 2.57 % in less than 0.1 GPU-days ( around 1.5 hours ) on a single Tesla V100 GPU , surpassing the result of 2.76 % reported by DARTS that required 1.0 GPUday . Furthermore , PC-DARTS allows a direct search on ImageNet ( while DARTS failed due to low stability ) , and sets the state-of-the-art record with a top-1 error of 24.2 % ( under the mobile setting ) in only 3.8 GPU-days ( 11.5 hours on eight Tesla V100 GPUs ) . 2 RELATED WORK . Thanks to the rapid development of deep learning , significant gain in performance has been brought to a wide range of computer vision problems , most of which owed to manually desgined network architectures ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; He et al. , 2016 ; Huang et al. , 2017 ) . Recently , a new research field named neural architecture search ( NAS ) has been attracting increasing attentions . The goal is to find automatic ways of designing neural architectures to replace conventional handcrafted ones . According to the heuristics to explore the large architecture space , existing NAS approaches can be roughly divided into three categories , namely , evolution-based approaches , reinforcement-learning-based approaches and one-shot approaches . The first type of architecture search methods ( Liu et al. , 2018b ; Xie & Yuille , 2017 ; Real et al. , 2017 ; Elsken et al. , 2019 ; Real et al. , 2019 ; Miikkulainen et al. , 2019 ) adopted evolutionary algorithms , which assumed the possibility of applying genetic operations to force a single architecture or a family evolve towards better performance . Among them , Liu et al . ( Liu et al. , 2018b ) introduced a hierarchical representation for describing a network architecture , and Xie et al . ( Xie & Yuille , 2017 ) decomposed each architecture into a representation of ‘ genes ’ . Real et al . ( Real et al. , 2019 ) proposed aging evolution which improved upon standard tournament selection , and surpassed the best manually designed architecture since then . Another line of heuristics turns to reinforcement learning ( RL ) ( Zoph & Le , 2017 ; Baker et al. , 2017 ; Zoph et al. , 2018 ; Zhong et al. , 2018 ; Liu et al. , 2018a ) , which trained a meta-controller to guide the search process . Zoph et al . ( Zoph & Le , 2017 ) first proposed using a controller-based recurrent neural network to generate hyper-parameters of neural networks . To reduce the computation cost , researchers started to search for blocks or cells ( Zhong et al. , 2018 ; Zoph et al. , 2018 ) instead of the entire network , and consequently , managed to reduce the overall computational costs by a factor of 7 . Other kinds of approximation , such as greedy search ( Liu et al. , 2018a ) , were also applied to further accelerate search . Nevertheless , the computation costs of these approaches , based on either evolution or RL , are still beyond acceptance . In order to accomplish architecture search within a short period of time , researchers considered to reduce the costs of evaluating each searched candidate . Early efforts include sharing weights between searched and newly generated networks ( Cai et al. , 2018 ) , and later these methods were generalized into a more elegant framework named one-shot architecture search ( Brock et al. , 2018 ; Cai et al. , 2019 ; Liu et al. , 2019 ; Pham et al. , 2018 ; Xie et al. , 2019 ) , in which an over-parameterized network or super-network covering all candidate operations was trained only once , from which exponentially many sub-networks can be sampled . As typical examples , SMASH ( Brock et al. , 2018 ) trained the over-parameterized network by a HyperNet ( Ha et al. , 2017 ) , and ENAS ( Pham et al. , 2018 ) shared parameters among child models to avoid retraining each candidate from scratch . This paper is based on DARTS ( Liu et al. , 2018b ) , which introduced a differentiable framework for architecture search , and thus combine the search and evaluation stages into one . A super-network is optimized during the search stage , after which the strongest sub-network is preserved and then retrained . Despite its simplicity , researchers detected some of its drawbacks , such as instability ( Li & Talwalkar , 2019 ; Sciuto et al. , 2019 ) , which led to a few improved approaches beyond DARTS ( Cai et al. , 2019 ; Chen et al. , 2019 ; Mei et al. , 2020 ) . In particular , ProxylessNAS ( Cai et al. , 2019 ) was the first method that searched directly on ImageNet , and P-DARTS ( Chen et al. , 2019 ) designed a progressive search stage to bridge the depth gap between the super-network and the sub-network . 3 THE PROPOSED APPROACH . 3.1 PRELIMINARIES : DIFFERENTIABLE ARCHITECTURE SEARCH ( DARTS ) . We first review the baseline DARTS ( Liu et al. , 2019 ) , and define the notations for the discussion later . Mathematically , DARTS decomposes the searched network into a number ( L ) of cells . Each cell is represented as a directed acyclic graph ( DAG ) with N nodes , where each node defines a network layer . There is a pre-defined space of operations denoted by O , in which each element , o ( · ) , is a fixed operation ( e.g. , identity connection , and 3 × 3 convolution ) performed at a network layer . Within a cell , the goal is to choose one operation from O to connect each pair of nodes . Let a pair of nodes be ( i , j ) , where 0 6 i < j 6 N − 1 , the core idea of DARTS is to formulate the information propagated from i to j as a weighted sum over |O| operations , namely , fi , j ( xi ) = ∑ o∈O exp { αoi , j } ∑ o′∈O exp { αo′i , j } · o ( xi ) , where xi is the output of the i-th node , and αoi , j is a hyper-parameter for weighting operation o ( xi ) . The output of a node is the sum of all input flows , i.e. , xj = ∑ i < jfi , j ( xi ) , and the output of the entire cell is formed by concatenating the output of nodes x2–xN−1 , i.e. , concat ( x2 , x3 , . . . , xN−1 ) . Note that the first two nodes , x0 and x1 , are input nodes to a cell , which are fixed during architecture search . This design makes the entire framework differentiable to both layer weights and hyper-parameters αoi , j , so that it is possible to perform architecture search in an end-to-end fashion . After the search process is finished , on each edge ( i , j ) , the operation o with the largest αoi , j value is preserved , and each node j is connected to two precedents i < j with the largest αoi , j preserved .

Summary: This paper proposes to improve the previously work DARTS in terms of the training efficiency, from the large memory and computing overheads. The authors propose a partially-connected DARTS (PC-DARTS) with two components: 1. Partial channel connection 2. edge normalization. To be detailed, they sample a small part of channels to perform connection and add edge normalization to eliminate the potential optimization problem. The results on CIFAR-10 and IamgeNet show the approach is effective, especially in ImageNet, the approach achieves SOTA results.