PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
Authors: Yuhui Xu
Lingxi Xie
Xiaopeng Zhang
Xin Chen
Guo-Jun Qi
Qi Tian
Hongkai Xiong
Venue:  (2020)
Keywords: Neural Architecture Search
DARTS
Regularization
Normalization

Abstract: Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-net and searching for an optimal architecture. In this paper, we present a novel approach, namely  Partially-Connected DARTS, by sampling a small part of super-net to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance. In particular, we perform operation search in a subset of channels while bypassing the held out part in a shortcut. This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels. We solve it by introducing  edge normalization, which adds a new set of edge-level hyper-parameters to reduce uncertainty in search. Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoy both faster speed and higher training stability. Experiment results demonstrate the effectiveness of the proposed method. Specifically, we achieve an error rate of 2.57% on CIFAR10 within merely 0.1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.2% on ImageNet (under the mobile setting) within 3.8 GPU-days for search. Our code has been made available at https://www.dropbox.com/sh/on9lg3rpx1r6dkf/AABG5mt0sMHjnEJyoRnLEYW4a?dl=0.

Published as a conference paper at ICLR 2020
PC-DARTS: PARTIAL CHANNEL CONNECTIONS FOR SEMORY-EFFICIENT ARCHITECTURE SEARCH
Yuhui Xu
Lingxi Xie² Qi Tian 2
Xiaopeng Zhang*
Xin Chen
Guo-Jun Qi
Hongkai Xiong!
'Shanghai Jiao Tong University Huawei Noah's Ark Lab
*Tongji University "Futurewei Technologies
yuhui xuês jtu. edu. en { {198808xc, exphistory)@gmaill com 14104528tongji guo junq@gmaii com tian. qi @hhuawei..... com xionghongkai@sjjat edu. on
ABSTRACT
Differentiable architecture search (DARTS) provided fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training super-network and searching for an optimal archi- tecture. In this paper, we present novel approach, namely, DARTS, by sampling a small part of super-network to reduce the redundancy exploring the network space, thereby performing more efficient search without comprising the performance. In particular, we perform operation search in a sub- set of channels while bypassing the held out part in shortcut. This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels. We alleviate it using edge normalization, which adds a new set of edge-level parameters to reduce uncertainty in search. Thanks the reduced memory cost, PC-DARTS can be trained with larger batch size and, consequently, enjoys both faster speed and higher training stability. Experi- mental results demonstrate the effectiveness of the proposed method. Specifically, we achieve an error rate of 2.57% on CIFARIO with merely 0.1 GPU-days for architecture search, and state-of-the-ee (under the mobile setting) using 3.8 GPU-days for search. Our code has been made available at https://gtt htt//gttttit thub. com hoom/yuhuiii xu1 993/PC- DARTS.
-rrteely-Connected
top-1 error rate of 24.2% on ImageNet
INTRODUCTION
Neural architecture search (NAS) emerged important branch of automatic machine learning (AutoML), and has been attracting increasing attentions from both academia and industry. The key methodology of NAS to build large space of network architectures, develop an efficient algo- rithm to explore the space, and discover the optimal structure under a combination of training data and constraints (e.g., network size and latency). Different from early approaches that often incur large computation overheads (Zoph & Le, 2017: Zoph et al., 2018; Real et al., 2019), recent one- shot approaches (Pham et al., 2018; Liu et al., 2019) have reduced the search costs by orders of mag- nitudes, which advances its applications to many real-world problems. In particular, DARTS (Liu et al., 2019) converts the operation selection into weighting a fixed set of operations. This makes the entire framework differentiable to architecture hyper-parameters and thus the network search can be efficiently accomplished in an end-to-end fashion. Despite its sophisticated design, DARTS still subject large yet redundant space of network architectures and thus suffers from heavy memory and computation overheads. This prevents the search process from using larger batch sizes for ei- ther speedup or higher stability. Prior work (Chen al., 2019) proposed reduce the search space, which leads to an approximation that may sacrifice the optimality of the discovered architecture.
This work done when Yuhui Xu and Xin Chen interns Huawei Noah's Ark Lab. Qi Tian is the corresponding author.
This work was supported in part by the National NSFC under Grant Nos. 61971285, 61425011, 61529101, 61622112 12, 6172010666 06001, 61932022, and in part by the Program of Shanghai Academic Research Leader under Grant 17XD140190 1000. We thank Longhui Wei and Bowen Shi for instructive discussions.

---

Published as a conference paper at ICLR 2020
In this paper, present simple yet effective approach named Partially-Connecte DARTS (PC- DARTS) to reduce the burdens of memory and computation. The core idea is intuitive: instead of sending all channels into the block of operation selection, we randomly sample subset of them in each step, while bypassing the rest directly in a shortcut. We assume the computation on this subset is a surrogate approximating that on all the channels. Besides the tremendous reduction in memory and computation costs, channel sampling brings another benefit operation search is regularized and less likely to fall into local optima. However, PC-DARTS incurs side effect, where the selection of channel connectivity would become unstable as different subsets of channels are sampled across iterations. Thus, we introduce edge normalization stabilize the search for network connectivity by explicitly learning extra set of edge-selection typer-parametert By sharing these syper-parameters throughout the training process, the sought network architecture is insensitive to the sampled channels across iterations and thus is more stable.
Benefiting from the partial connection strategy, we are able to greatly increase the batch size. Specif- ically, as only 1/K of channels are randomly sampled for an operation selection, it reduces the memory burden by almost K times. This allows us to use a K times larger batch size during search, which not only accelerates the network search but also stabilizes the process particularly for large- scale datasets. Experiments on benchmark datasets demonstrate the effectiveness of PC-DARTS. Specifically, we achieve an error rate of 2.57% in less than 0.1 GPU-days (around 1.5 hours) single Tesla V100 GPU, surpassing the result of 2.76% reported by DARTS that required 1.0 GPU- day. Furthermore, PC-DARTS allows direct search on ImageNet (while DARTS failed due low stability), ), and sets the state-of-the-ar record with a top-1 error of 24.2% (under the mobile setting) in only 3.8 GPU-days (11.5 hours on eight Tesla V100 GPUs).
2 RELATED WORK
Thanks to the rapid development of deep learning, significant gain in performance has been brought to a wide range of computer vision problems, most of which owed to manually desgined network architectures (Krizhevsky al., 2012; Simonyan & Zisserman, 2015; He et al., 2016; Huang al., 2017). Recently, a new research field named neural architecture search (NAS) has been attracting increasing attentions. The goal is to find automatic ways of designing neural architectures to replace conventional bandcrafted ones. According to the heuristics to explore the large architecture space, existing NAS approaches can be roughly divided into three categories, namely, evolution-based approaches, lewroorreeeeee-eennn-bbbbeeeee
approaches and one-shot approaches.
The first type of architecture search methods (Liu et al., 2018b; Xie & Yuille, 2017; Real et al., 2017; Elsken et al., 2019; Real et al., 2019; Miikkulainen et al., 2019) adopted evolutionary algorithms, which assumed the possibility of applying genetic operations to force a single architecture or a family evolve towards better performance. Among them, Liu el al. (Liu et al., 2018b) introduced a hierarchical representation for describing a network architecture, and Xie et al. (Xie Yuille, 2017) decomposed each architecture into representation of 'genes'. Real et al. (Real et al., 2019) proposed aging evolution which improved upon standard tournament selection, and surpassed the best manually designed architecture since then. Another line of heuristics turns to reinforcement learning (RL) (Zoph & Le, 2017; Baker et al., 2017; Zoph et al., 2018: Zhong et al., 2018; Liu al., 2018a), which trained meta-controller guide the search process. Zoph et al. (Zoph & Le, 2017) first proposed using controller-base recurrent neural network to generate hyper-parameters of neural networks. To reduce the computation cost, researchers started to search for blocks or cells (Zhong et al., 2018; 8; Zoph et al., 2018) instead of the entire network, and consequently, managed to reduce the overall computational costs by a factor of 7. Other kinds of approximation, such as greedy search (Liu et al., 2018a). were also applied to further accelerate search. Nevertheless, the computation costs of these approaches, based on either evolution or RI., are still beyond acceptance.
In order to accomplish architecture search within a short period of time, researchers considered to reduce the costs of evaluating each searched candidate. Early efforts include sharing weights between searched and newly generated networks (Cai et al., 2018), and later these methods were generalized into a more elegant framework named one-shot architecture search (Brock et al., 2018; Cai et al., 2019; Liu et al., 2019; Pham et al., 2018; Xie et al., 2019), in which over per-paaramteriieed network or super-network covering all candidate operations was trained only once, from which exponentially many sub-networks can be sampled. As typical examples, SMASH (Brock et al.,
2

---

Published as a conference paper at ICLR 2020
Partial Channel Connection
0
Sample
$
Concet
X
STD
FC0 (K STaE)
exp
explata
Edge Normalization
2
Fo (Kpc Sqa)
Partial Channel Connection
Xo
Partial Connection
X1
3
(N. S
PC
Partial Channel Connection
epplft
Ere explAr
Figure 1: Illustration of the proposed approach (best viewed in color), partially-cone connected DARTS (PC-DARTS). As an example, we investigate how information is propagated to node #3, i.e., 3. There are two sets of hyper-parameters during search, namely, { and (Brg), J. where 0 < j and O. To determine {a{,), we only sample a subset, 1/K, of channels and connect them the next stage, so that the memory consumption is reduced by K times. To minimize the uncertainty incurred by sampling, we add {Br.}} extra edge-level parameters.
2018) trained the over- over-paramen peee-parametei network by a HyperNet (Ha al., 2017), and ENAS (Pham et al., 2018) shared parameters among child models to avoid retraining each candidate from scratch.
This paper is based on DARTS (Liu al., 2018b), which introduced differentiable framework for architecture search, and thus combine the search and evaluation stages into one. A super-network is optimized during the search stage, after which the strongest sub-network preserved and then re- trained. Despite its simplicity, researchers detected some of its drawbacks, such as instability (Li & Talwalkar, 2019; Sciuto et al., 2019), which led to a few improved approaches beyond DARTS (Cai al., 2019; Chen et al., 2019; Mei al., 2020). In particular, ProxylessNAS (Cai et al., 2019) was the first method that searched directly on ImageNet, and P-DARTS (Chen et al., 2019) designed a progressive search stage bridge the depth gap between the super-network and the sub-network.
THE PROPOSED APPROACH
3.1 PRELIMINARIES: DIFFERENTIABLE ARCHITECTURE SEARCH (DARTS)
We first review the baseline DARTS (Liu al., 2019), and define the notations for the discussion later. Mathematically, DARTS decomposes the searched network into number (L) of cells. Each cell is represented as a directed acyclic graph (DAG) with N nodes, where each node defines a network layer. There is pre-defined space of operations denoted O, in which each element, o(.), is a fixed operation (e.g., identity connection, and 3 x 3 convolution) performed at a net- work layer. Within cell, the goal is to choose one operation from O connect each pair of nodes. Let a pair of nodes be (i,j), where < i < j & N - 1, the core idea of DARTS is to formulate the information propagated from to j as a weighted sum over |O] operations, namely, exp(al,)
(x4) EoEo E.'e Expo (x) o(x;), where x; is the output of the i-th node, and at is a paremmprreeeter for weighting operation o(x;). The output of node is the sum of all input flows, ie.. x; = Ei. (x.), and the output the entire cell formed by concatenating the output of nodes X2 -1 i.e., concat(X2. Xg.... 11 Note that the first two nodes, X9 and x are input nodes to cell, which are fixed during architecture search.

---

Published as a conference paper at ICLR 2020
This design makes the entire framework differentiable to both layer weights and a, so that it is possible to perform architecture search in an end-to-end fashion. After the search process is finished, on each edge (i, the operation o with the largest og, value is preserved, and each node j is connected two precedents < j with the largest 0g preserved.
3.2 PARTIAL CHANNEL CONNECTIONS
A drawback of DARTS lies in memory inefficiency. In the main part of the searched architecture, |O| operations and the corresponding outputs need to be stored at each node (i.e., each network layer), leading to |O] memory to use. To fit into a GPU, one must reduce the batch size during search, which inevitably slows down search speed, and may deteriorate search stability and accuracy.
An alternative solution to memory efficiency is the partial channel connection as depicted in Fig- ure 1. Take the connection from x; to x, for example. This involves defining a channel sampling mask S.y. which assigns 1 to selected channels and 0 to masked ones. The selected channels are sent into mixed computation of |O] operations, while the masked ones bypass these operations, i.e. they are directly copied to the output,
exp {a(s} OEO OEO Ed' exp {a]}}
P. (x;;
Σ ]
o(S * xx,) + - (1-Sa,) + X,.
where, Sty + x, and (1-) x; denote the selected and masked channels, respectively. In practice, we set the proportion of selected channels to 1/K by regarding K as hyper-parameter. By varying K, we could trade off between architecture search accuracy (smaller K) and efficiency (larger K) to strike balance (See Section 4.4.1 for more details).
direct benefit brought by the partial channel connection is that the memory overhead of computing f2.j (x;:S..) is reduced by K times. This allows search. There are twofold benefits. First, the computing cost could be reduced by K times during the architecture search. Moreover, the larger batch size implies the possibility of sampling more training data during each iteration. This is particularly important for the stability of architecture search. In most cases, the advantage of one operation over another is not significant, unless more training data are involved in mini-batch to reduce the uncertainty in updating the parameters of network weights and architectures.
larger batch size for architecture
3.3 EDGE NORMALIZATION
Let us look into the impact of sampling channels neural architecture search. There are both posi- tive and negative effects. On the upside, by feeding a small subset of channels for operation mixture while bypassing the remainder, we make it less biased in selecting operations. In other words, for edge (i, given an input x,, the difference from using two sets of hyper-parameters {af,) and {a } is largely reduced, because only small part /)) of input channels would go through the operation mixture while the remaining channels are left intact. This regularizes the preference of weight-free operation (e.g., skip-connect, max-pooling, etc.) over weight-equipped one (e.g., various kinds of convolution) in O. In the early stage, the search algorithm often prefers weight-free operations, because they do not have weights to train and thus produce more consistent outputs, i.e. o(x.). In contrast, the weight-equipped ones, before their weights are well optimized, would propagate inconsistent information across iterations. Consequently, weight-free operations often accumulate larger weights (namely a?,)) at the beginning, and this makes it difficult for the weight- equipped operations to beat them even after they have been well trained thereafter. This phenomenon is especially significant when the proxy dataset (on which architecture search is performed) is diffi- cult, and this could prevent DARTS from performing satisfactory architecture search on ImageNet. In experiments, we will show that PC-DARTS, with partial channel connections, produces more stable and superior performance on ImageNet.
On the downside, in a cell, each output node xj needs to pick up two input nodes from its precedents Xx,, ......, x) which are weighted by max, az,;; max, af.:.... max, a5-1.j respectively, optimized by randomly sampled channels across iterations, and thus the optimal connectivity determined by them could be unstable the sampled channels change over time. This could cause undesired fluctuation in
following the original DARTS. However, these architecture parameters
4

---

Published as a conference paper at ICLR 2020
the resultant network architecture. To mitigate this problem, we introduce edge normalization that weighs on each edge (1,j) explicitly, denoted by 8r.j. so that the computation of x, becomes:
exp (Bi.) 25 exp {Br,a} (x,(
PC EE
Specifically, after the architecture search is done, the connectivity of edge (i,j) is determined by both {ag.5 } and Bi.j. for which we multiply the normalized coefficients together, ie., multiplying exp(Braa
oxpfaf..)
Evey Preeyepp/Ae exp | by E.e {xp) Then the edges are selected by finding the large edge weights
as in DARTS. Since BL. are shared through the training process, the learned network architecture insensitive to the sampled channels across iterations, making the architecture search more stable. In Section 4.4.2, we will show that edge normalization is also effective over the original DARTS.
Finally, the extra computation overhead required for edge normalization is negligible.
3.4 DISCUSSIONS AND RELATIONSHIP TO PRIOR WORK
First of all, there are two major contributions of approach, namely, channel sampling and edge normalization. Channel sampling, as the key technique in this work, has not been studied in NAS for reducing computational overhead (other regularization methods like Dropout (Srivastava al., 2014) and DropPath (Larsson et al., 2017) cannot achieve the same efficiency, memory, as channel sampling). It accelerates and regularizes search and, with the help of edge normalization, improves search stability. Note that both search speed and stability are very important for a search algorithm. Combining channel sampling and edge normalization, we obtain the best accuracy on ImageNet (based on the DARTS search space), and the direct search cost on ImageNet (3.8 GPU-days) is the lowest known. Moreover, these two components are easily transplanted to other search algorithms to improve search accuracy and speed, e.g., edge normalization boosts the accuracy and speed of the original DARTS methods.
both time and
Other researchers also tried to alleviate the large memory consumption of DARTS. Among prior efforts, ProxylessNAS (Cai et al., 2019) binarized the multinomial distribution af, and samples two paths at each time, which significantly reduced memory cost and enabled direct search ImageNet. PARSEC (Casale et al., 2019) also proposed a sampling- pppseing-basdd optimization method to learn a prob- ability distribution. Our solution, by preserving all operations for architecture search, achieves higher accuracy in particular on challenging datasets like ImageNet (+0.7% over ProxylessNAS and +1.8% over PARSEC). Another practical method towards memory efficiency is Progressive- DARTS (Chen et al., 2019), which eliminated a subset of operators in order to provide sufficient memory for deeper architecture search. In comparison, our approach preserves all operators and instead performs sub-sampling on the channel dimension. This strategy works better in particular on large-scale datasets like ImageNet.
EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION DETAILS
We perform experiments on CIFARIO and ImageNet, two most popular datasets for evaluating neu- ral architecture search. CIFARIO 10 (Krizhevsky & Hinton, 2009) consists of 60K images, all of which are of a spatial resolution of 32 x 32. These images are equally distributed over 10 classes, with 50K training and 10K testing images. ImageNet (Deng et al., 2009) contains 1,000 object cate- gories, and 1.3M training images and 50K validation images, all of which are high-resolution and roughly equally distributed over all classes. Following the conventions (Zoph al., 2018; Liu al., 2019), we apply the mobile setting where the input image size is fixed to be 224 224 and the number multi-add operations does not exceed 600M in the testing stage.
Following DARTS (Liu et al., 2019) well as conventional architecture search approaches, use an individual stage for architecture search, and after the optimal architecture is obtained, we conduct another training process from scratch. In the search stage, the goal is to determine the best sets of set is partitioned into two parts, with the first part used for optimizing network parameters, e.g.,
namely {af, and {Big) for each edge (i.j). To this end, the trainnig
5

---

Published as a conference paper at ICLR 2020
Table Comparison with state-of-the-art network architectures on CIFARIO.
Test Err.
Params
Search Cost
Architecture
Search Method
(%)
(M)
(GPU--ays)
DenseNct-Bel BC (Huang ct al., 2017)
3,46
25.6
manual
NASNet-A + cutout (Zoph et al., 2018)
2.65
3.3
1800
RL
AmoehaNet-B + cutout (Real et al., 2019)
2.55++00.05
2.8
3150
evolution
Hireachical Evolution (Liu et al., 2018b)
3.75+-0.12
15.7
300
evolution
PNAS (Liu et al., 2018a)
3.41+00.99
3.2
225
SMBO
ENAS + cutout (Pham al., 2018)
2.89
4.6
0.5
RL
NAONet- WS (Luo et al., 2018)
3.53
3.1
0.4
NAO
DARTS (Ist order) + cutout (Liu al., 2019)
3.00+0.114
3.3
0.4
gradient-basee based
DARTS (2nd order) + cutout (Liu et al., 2019)
2.76:+0.09
3,3
muddient-basee
SNAS (moderate) + cutout (Xie et al., 2019)
2.85.10.02
2.8
1.5
gradient-based
ProxylessNAS + cutout (Cai et al., 2019)
2.08
4.0
gradient-base
P-DARTS cutout (Chen al., 2019)
2.50
3,4
0.3
gradient-based
BayesNAS + cutout (Zhou et al., 2019)
2.81#00004
3.4
0.2
gradient- based
PC DARTS cutout
2.57=00077
3.6
0.1
gradient based
Recorded on single GTX 1080Ti. It can be shortened into 0.06 GPU-days if Tesla V100 is used. We PC-DARTS times and used standalone validation to pick the best from the 5 runs. This process was done by using 45K out of 50K training images for training, and the remaining 5K images for valida- tion. The best one in validation used for testing, which reported test of 2.57%.
convolutional weights, and the second part used for optimizing hyper-parametent The entire search stage is accomplished in an end-to-end manner. For fair comparison, the operation space O remains the same the convention, which contains 8 choices, i.e., 3x3 and 5x5 separable convolution, 3 3 and 5 5x5 dilated separable convolution, 3x3 max-pooling, 3x3 pherage-pooling, skip-connect (a.k.a., identity), and zero (a.k.a., none).
We propose an alternative and more efficient implementation for partial channel connections. For edge (i. j). we do not perform channel sampling at each time of computing oxx)), but instead choose the first 1/ K channels of x; for operation mixture directly. To compensate, after x; is obtained, we shuffle its channels before using it for further computations. This is the same implementation used in ShuffleNet (Zhang et al., 201 8), which is more GPU-friendly and thus runs faster.
4.2 RESULTS ON CIFAR10 10
In the search scenario, the over- over- peee-parameterieed network is constructed by stacking 8 cells (6 normal cells and 2 reduction cells), and each cell consists of N 6 nodes. We train the network for 50 epochs, with the initial number of channels being 16. The 50K training set of CIFARIO is split into two subsets with equal size, with one subset used for training network weights and the other used for architecture paper-parammters.
We set K = for CIFAR1O, ie., only 1/4 features are sampled on each edge, so that the batch size during search is increased from 64 to 256. Besides, following (Chen et al., 2019), freeze the hyper-parameters, { } and (3 and only allow the network parameters to be tuned in the first 15 epochs. This process, called warm-up, is to alleviate the drawback of the parameterized operations. The total memory cost is less than 12GB so that we can train on most modern GPUs. The network weights are optimized by momentum SGD, with an initial learning rate 0.1 (annealed down to zero following a cosine schedule without restart), a momentum of 0.9, and a weight decay 3 10 4 We use an Adam optimizer (Kingma & Ba, 2015) for {a?,} } and [. with fixed learning rate of 6 x 10-4, a momentum of (0.5, 0.999) and weight decay of 10-3, 10,3 Owing to the increased batch size, the entire search process only requires 3 hours
GTX 1080Ti GPU, or 1.5 hours on a Tesla V100 GPU, which is almost 4x faster than the original first-order DARTS.
The evaluation stage simply follows that of DARTS. The network is composed of 20 cells (18 normal cells and 2 reduction cells), and each type of cells share the same architecture. The initial number of channels is 36. The entire 50K training set is used, and the network is trained from seratch for

---

Published as a conference paper at ICLR 2020
Jkip connect
502,000 200_0om__33
[K-2]
100, cowm_ooyyy. 3x-3
Sc0, wet.oooyryrr 333
2pp_ron_3333
Tep
[_
ronnect
[1:1)
3
sep. _0mr_333
dll_coww SX5
2gg pacl
Me corv. 3x3
(a) the normal cell found on CIFAR1O
the normal cell found on ImageNet
21
Sxt Spp canu 3
c [k-2]
cony 5
comi
2
3x5
Sap carry
c.kk]
1k-11
kep cuty 5
Hup, cOny. 3+3
[ko
skp cornicct
sep. coNN 3+3.
200 220_conN331
3x3
(b) the reduction cell found on CIFARIO
(d) the reduction cell on ImageNet
Figure 2: Cells found CIFARIO and ImageNet. Searching on ImageNet makes the normal cell more complex (deeper), although the reduction cell is very similar to that found on CIFARIO.
600 epochs using a batch size of 128. We use the SGD optimizer with an initial learning rate of 0.025 (annealed down zero following cosine schedule without restart), momentum of 0.9, weight decay of x 10 and a norm gradient clipping at 5. Drop-path with a rate of 0.3 as well as cutout Vrevs & Taylor, 2017) is also used for regularization. We visualize the searched normal and reduction cells in the left-hand side of Figure 2.
Results and comparison to recent approaches PC-DARTS achieve an error rate of 2.57%, with both search time and accuracy surpassing the baseline, DARTS, significantly. To the best of our knowledge, our approach is the fastest that achieves an error rate of less than 3%. Our number ranks among the top of recent architecture search results. ProxylessNAS used different protocol to achieve an error rate of 2.08%, and also reported a much longer time for architecture search. P-DARTS (Chen et al., 2019) slightly outperforms our approach by searching over a deeper architecture, which we can integrate our approach into P-DARTS to accelerate it as well as improve its performance (consistent accuracy gain is obtained).
summarized in Table 1. In merely 0.1 GPU-days,
4.3 RESULTS ON IMAGENET
We slightly modify the network architecture used on CIFARIO to fit ImageNet. The over- parameterized network starts with three convolution layers of stride 2 reduce the input image resolution from 224 x 224 to 28 x 28. 8 cells (6 normal cells and 2 reduction cells) are stacked be- yond this point, and each cell consists of N 6 nodes. To reduce search time, randomly sample two subsets from the 1.3M training set of ImageNet, with 10% and 2.5% images, respectively. The former one used for training network weights and the latter for updating typer-paramettrs.
ImageNet is much more difficult than CIFARIO. To preserve more information, we use a sub- sampling rate of 1/2, which doubles that used in CIFARIO. Still, total of 50 epochs are trained and architecture srper-parameters are frozen during the first 35 epochs. For network weights, we use momentum SGD with an initial learning rate of 0.5 (annealed down to zero following a cosine sched- ule without restart), a momentum of 0.9, and weight decay of x 10-5, For hyper-parameters, we use the Adam optimizer (Kingma Ba, 2015) with fixed learning rate of 10-3, a momentum (0.5, 0.999) and a weight decay of 10 3, We eight Tesla V100 GPUs for search, and the total batch size is 1,024. The entire search process takes around 11.5 hours. We visualize the searched normal and reduction cells in the right-hand side of Figure 2.
The evaluation stage follows that of DARTS, which also starts with three convolution layers with a stride of 2 that reduce the input image resolution from 224 x 224 to 28 28. 14 cells (12 normal cells and reduction cells) are stacked beyond this point, with the initial channel number being 48. TTe network is trained from scratch for 250 epochs using batch size of 1,024. We use the SGD
7

---

Published as a conference paper at ICLR 2020
Table Comparison with state-of-the-art architectures on ImageNet (mobile setting).
Architecture
Test Err. (%)
Params
X+
Search Cost
Search Method
top-1
top-5
(M)
(M)
(GPU-days)
Inception-vi (Szegedy et al., 2015)
30.2
10.1
6.6
1448
manual manual
MobileNet (Howard et al., 2017)
29.4
10.5
4.2
569
ShuffleNer 2 (v1) (Zhang et al., 2018)
26.4
10.2
524
manual
ShuffleNet 2x (v2) (Ma el al., 2018)
25.1
~
591
manual
NASNet-A (Zoph et al., 2018)
26.0
8.4
5.3
1800
RL
AmoebaNet-C (Real et al., 2019)
24.3
7.6
6.4
3150
evolution
PNAS (Liu al., 2018a)
25.8
8.1
588
225
SMBO
MnasNeet-22 (Tan al., 2019)
25.2
8.0
4.4
388
RL
DARTS (2nd order) (Liuu al., 2019)
26.7
8.7
4.7
574
4.0
gradient-
SNAS (mild) (Xie al., 2019)
27.3
9.2
4.3
522
1.5
gradient-base based
ProxylessNAS (GPU)$ (Cai et al., 2019)
24.9
gradient-base based
P-DARTS (CIFAR10) (Chen et al., 2019)
24.4
7.4
4.9
gradient-bust based
P-DARTS (CIFAR 100) (Chen al., 2019)
24.7
7.5
577
0.3
gradient-bast based
BayesNAS (Zhou et al., 2019)
26.5
8.9
3.9
0.2
gradient based
PC-DARTS (CIFARIO)
25.1
7.8
5.3
586
0.1
gradient- based
PC-DARTS (ImageNer)*
24.2
7.3
5.3
3.8
gradient based
This architecture was searched on ImageNet directly, otherwise was searched on CIFARIO or CIFAR100.
optimizer with a momentum of 0.9, an initial learning rate of 0.5 (decayed down to zero linearly), and weight decay of 3 10-5, Additional enhancements adopted including label smoothing and an auxiliary loss tower during training. Learning rate warm-up is applied for the first epochs.
Results summarized in Table 2. Note that the architectures searched CIFARIO and ImageNet itself are both evaluated. For the former, reports a top-1/5 error of 25. 1%/78%% which signifi- cantly outperforms 26.7% reported by DARTS. This is impressive given that search time is much shorter. For the latter, we achieve top-1/5 error of 24.2% which is the best known performance date. In comparison, ProxylessNAS (Cai al., 2019), another approach that directly searched on ImageNet, used almost doubled time to produce 24.9% which verifies that our strategy of reducing memory consumption is more efficient yet effective.
CIFARTO
3.2
Test Error (X)
Search Cost (GPU-hours))
PC
EN
Test Error
Search Cost
X
3.00
0.4 GPU-days
X
V
2.82+000054
0.4 GPU-days
V
X
2.67=t0.1 11%
0.1 GPU-days
2.57.10.07%
0.1 GPU-days
ImageNet (ILSVRC2012)
PC
EN
Test Error
Search Cost
X
X
26.8110.1%
7.7 GPU-days
X
V
26.31:011%
7.7 GPU-days
/
X
26.2+t0.%
3.8 GPU-days
25.8+0.1%
3.8 GPU-days
PC-DAMYS_1/1 PC-DANTS_3/2 PC-DARTS_1/A PG-DARTS_1/
Table 3: Ablation study CIFARIO and Ima-
Figure 3: Search cost and accuracy parison between our approach with different sampling rates, namely, 1/1, 1/2, 1/4 and 1 1/8, among which 1/4 makes a nice tradeoff between accuracy and efficiency.
geNet. PC and EN denote partial channel con- nections and edge normalization, respectively. All architectures on ImageNet are re-trained by 100 epochs (the 25.8% error corresponds the best entry, 24.2%, reported in Table 2 (250 epochs).
4.4 ABLATION STUDY
4.4.1 EFFECTIVENESS OF CHANNEL PROPORTION 1 //K
We first evaluate K, the parameer that controls the sampling rate of channels. Note that a tradeoff exists: inereasing the sampling rate (iee. using a smaller K)) allows more accurate infor-
8

---

Published as a conference paper at ICLR 2020
Table 4: Experiments on stability of DARTS and PC-DARTS. Left: Evaluations of searched archi- tectures in five independent search runs. Middle: architectures searched with different numbers of epochs. Right: runs on architectures searched with different numbers of nodes.
Runs #3
Epochs
Nodes
Methods
#1
#2
#4
#5
50
75
100
125
7
DARTS-
2.89
3.15
2.99
3.07
3.27
2.98
2.87
3.32
3.08
3.03
2.98
2.89
DARTS-x2(9)
3.11
2.68
2.77
3.14
3.06
2.76
2.93
3.51
3.18
2.82
2.76
3.02
PC-DARTS(%)
2.72
2.67
2.57
2.75
2.64
2.57
2.67
2.69
2.75
2.63
2.57
2.64
mation to be propagated, while sampling a smaller portion of channels casts heavier regularization and may alleviate over-fitting. To study its impacts, we evaluate the performance produced by four sampling rates, namely 1/1, 1/2, 1/4 and 1/8, on CIFARIO I0, and plot the results into a diagram of search time and accuracy in Figure 3. One can observe that a sampling rate of 1/4 yields superior performance over 1/2 and 1/1 in terms of both time and accruacy. Using 1/8, while being able to further reduce search time, causes a dramatic accuracy drop.
These experiments not only justify the tradeoff between accuracy and efficiency of architecture search, but also reveal the redundancy of super-network optimization in the context of NAS. More essentially, this reflects the gap between search and evaluation, i.c. a better optimized super-network does not guarantee better searched architecture - in other words, differentiable NAS approaches are easily to over-fit on the super-network. From this viewpoint, channel sampling plays the role of regularization, which shrinks the gap between search and evaluation.
4.4.2 CONTRIBUTIONS OF DIFFERENT COMPONENTS OF PC-DARTS
Next, we evaluate the contributions made by two components of PC-DARTS, namely, partial chan- nel connections and edge normalization. The results are summarized in Table 3. is clear that edge normalization brings the effect of regularization even when the channels are fully-connected. Being component with very few extra costs, it can be freely applied to wide range of approaches involv- ing edge selection. In addition, edge normalization cooperates well with partial channel connections to provide further improvement. Without edge normalization, in both the number of network parameters and accuracy. On CIFAR10, we run search without edge normalization for several times, and the testing error ranges from 2.54% 3.01%. %. On the other hand, with edge normalization, the maximal difference among five runs does not exceed 0.15%. Therefore, we justify our motivation in designing edge normalization (see Section 3.3), i.e., a standalone method for stabilizing architecture search, yet it works particularly well under partial channel connection, since the latter introduces randomness and stabilization indeed helps.
approach can suffer low stability
be
4.4.3 STABILITY OF OUR APPROACH
In this part, we demonstrate the stability of our approach from three different perspectives. Results are summarized in Table 4, with detailed analysis below.
First, we evaluate the stability of different approaches by conducting 5 independent search runs. We re-implement DARTS-vI and DARTS-v2 with the proposed code, as well as that of our approach, and perform five individual search processes with the same syper-parameters but different random seeds (0, 1, 2, 3, 4). The architectures found by DARTS in different runs, either vl or v2, suffer much higher standard deviations than that of our approach (DARTS-vI: +0.15%, DARTS-v2: +0.21% PC-DARTS: +0.07%).
Second, we study how the search algorithm is robust to pyper-parameters, e.g., the length of the search stage. We try different numbers of epochs, from 50 to 125, and observe how it impacts the performance of searched architectures. Again, we find that both DARTS-vI and DARTS-v2 are less robust this change.
Third, we go one step further by enlarging the search space, allowing a larger number of nodes to appear in each cell the original DARTS-basee space has 6 nodes, and here we allow 5, 6 and 7 nodes. From to 6 nodes, the performance of all three algorithms goes up. while from 6 to nodes, DARTS-v2 suffers significant accuracy drop, while PC-DARTS mostly preserves it performance.

---

Published as a conference paper at ICLR 2020
Table Detection results, in terms of average precisions, on the MS-COCO dataset (test-dev 2015).
Network
Inpat Size
Backbone
x+
AP
su
AP7o
APs
APM
APL.
SSD300 (Liu et al., 2016)
x30x300
VGG-16
35.2B
23.2
41.2
23.4
5.3
23.2
39.6
SSD512 (Liu et al., 2016)
512x512
VGG-16
99.5B
26.8
46.5
27.8
28.9
41.9
YOLOV2 (Redmon & Farhadi, 2017)
416x416
Darknet-19
17.58
21.6
44.0
19.2
5.0
22.4
35.5
Pelee (Wang et al., 2018)
304x304
PelecNet
1.38
22.4
38.3
22,9
SSDLiteVI (Howard et al.. 2017)
320x320
MobileNetV1
1.3B
22.2
SSDLiteV2 (Sandler
2018)
320x320
MobileNetV2
0.8B
22.1
SSDLiteV3 (Tan et ul., 2019)
320x320
MnasNet-Al
0.8B
23.0
3.8
21.7
42.0
PC-DARTS with SSD
320x320
PC-DARTS
1.2B
28.9
46,9
30.0
7.9
32.0
48.3
The backbone architecture of PC-DARTS was searched on ImageNet (with 24.2% top-1 error).
As side note, all these algorithms fail to gain accuracy in enlarged search spaces, because CIFARIO is relatively simple and the performance of searched architectures seems to saturate.
With all the above experiments, we can conclude that PC-DARTS is indeed more robust than DARTS in different scenarios of evaluation. This largely owes to the regularization mechanism introduced by PC-DARTS, which (i) forces it to adjust to dynamic architectures, and (ii) avoids the large pruning gap after search, brought by the none operator.
4.5 TRANSFERRING TO OBJECT DETECTION
To further validate the performance of the architecture found by PC-DARTS, bone for object detection. We plug the architecture found on ImageNet, as shown in Figure 2, into a popular object detection framework named Single-Shot Detectors (SSD) (Liu et al., 2016). We train the entire model on the MS-COCO (Lin et al., 2014) trainval dataset, which is obtained by standard pipeline that excludes 5K images from the val set, merges the rest data into the 80K train set and evaluates it on the test-dev 2015 set.
it the back-
Results are summarized in Table 5. Results for SSD, YOLO and MobileNets are from (Tan et al., 2019). With the backbone searched by PC-DARTS, we need only 1.2B FLOPs to achieve an AP of 28. 28,,, which 5.7% higher than SSD300 (but with 29x fewer FLOPs), or 2.1% higher than SSD512 (but with 83x fewer FLOPss. Compared to the "Lite" versions of SSD, our result enjoys significant advantages in AP, surpassing the most powerful one (SSDLiteV3) by an AP of 6.9%. All these results suggest that the advantages obtained by PC-DARTS image classification can transfer well to object detection, a more challenging task, and we believe these architectures would benefit even more application scenarios.
CONCLUSIONS
In this paper, we proposed a simple and effective approach named curtially-connectte differentiable architecture search (PC-DARTS). The core idea is to randomly sample a proportion of channels for operation search, so that the framework is more memory efficient and, consequently, a larger hatch size can be used for higher stability. Additional contribution search stability is made by edge nor- malization, light-weighted module that requires merely no extra computation. Our approach can accomplish a complete search within 0.1 GPU-days on CIFARIO, or 3.8 GPU-days on ImageNet, and report -fate-of-the-an classification accuracy in particular on ImageNet.
This research delivers two important messages that are important for future research. First, differen- tiable architecture search seems to suffer even more significant instability compared to conventional neural network training, and so it can largely benefit from both (i) regularization and (ii) larger batch size. This work shows an efficient way to incorporate these two factors in a single pipeline, yet we believe there exist other (possibly more essential) solutions for this purpose. Second, going one step further, our work reveals the redundancy of super-network optimization in NAS, and exper- iments reveal a gap between improving super-network optimization and finding a better architecture, and regularization plays efficient role in shrinking the gap. We believe these insights inspire researchers in this field, and we will also follow this path towards designing stabilized yet efficient algorithms for differentiable architecture search.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. In ICLR, 2017.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. SMASH: one-shot model architecture search through hypernetworks. In ICLR, 2018.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI, 2018.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In ICLR, 2019.
Francesco Paolo Casale, Jonathan Gordon, and Nicolo Fusi. Probabilistic neural architecture search. arXiv preprint 1XXX::9902005166 2019.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In ICCV, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Terrance Deeeie and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint 17Xiv::708.04555 2017.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. In ICLR, 2019.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. In ICLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04800 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convo- lutional neural networks. In NIPS, 2012.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. FractalNet: Ultra-deep neural net- works without residuals. ICLR, 2017.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In UAI, 2019.
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects context. In ECCV, 2014.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018a.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisamtha Fernando, and Koray Kavukcuoglu. Hi- erarchical representations for efficient architecture search. In ICLR, 2018b.

---

Published as a conference paper at ICLR 2020
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019.
Weiwei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng- Yang Fu, and Alexander C. Berg. Ssd: Single shot multibox detector. In ECCV, 2016.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. In NeurlPS, 2018
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet V2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018.
Jieru Mei, Xiaochen Lian, Xiaojie Jin, Linjie Yang, Yingwei Li, Alan Yuille, and Jianchao Yang. AtomNAS: Fine-grained end-to-end neural architecture search. In ICLR, 2020.
Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp- 293-312. Elsevier, 2019.
Evolving deep neural
Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In ICML,
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In AAAI, 2019.
Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger. CVPR, 2017.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. CVPR, 2018.
Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. ArXiv, hiss/192.0811 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICIRR 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. JMLR, 1599)1922911988 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V Le. MnasNet: Platform- aware neural architecture search for mobile. CVPR, 2019.
Robert J. Wang, Xiang Li, Shuang Ao, and Charles X. Ling. Pelee: A real-time object detection system on mobile devices. In NewrlPS, 2018.
Lingxi Xie and Alan Yuille. Genetic CNN. In ICCV, 2017.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: Stochastic neural architecture search. In ICLR, 2019.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018.
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. In CVPR, 2018.
12

---

Published as a conference paper at ICLR 2020
Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. BayesNAS: A Bayesian approach for neural architecture search. In ICML, 2019.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Lc. Learning transferable architectures for scalable image recognition. In CVPR, 2018.
13


Reviews:
[{"review_id": "ryg_OAC6YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---\nrevised score. rebuttal clears my concerns.\n---\nSummary:\n\nThe paper proposes a partially connected differential architecture search (PC-DARTS) technique, that uses a variant of channel dropout for each node's output feature maps, and a weighted summation of concatenating all previous nodes. Searched architecture on CIFAR-10 and ImageNet seems to outperform the one discovered by the original DARTS, however, the results are not directly comparable due to the slight change of search space. \n\nIntroducing this edge normalization is a novel contribution, but it is more like a trick to have a better search space rather than the PC-DARTS itself. My main concerns are about the incremental novelty and experiments are heavily done on one search run, especially the search space is not the same as baseline DARTS. \n\nI do not think the current version is ready for ICLR, but I am looking forward to seeing the authors' rebuttal and I am willing to revise my review accordingly.\n\nMain concerns\n\n- Incremental novelty about channel sampling.\nDoing edge normalization in the PC-DARTS is indeed novel, however, the channel sampling (abbr. PC for partial channel connection) is not. Dropout is widely adopted in all deep learning training since AlexNet. In NAS with parameter sharing, Pham et al. already exploit the channel dropout as shown in ENAS function \"def drop_path\"(https://github.com/melodyguan/enas/blob/master/src/cifar10/image_ops.py). It is true that previous works treated like one hyper-parameters and do not provide deeper insight about this term, but it is not correct to say in Section 3.4 \"Channel sampling ... has never been studied in prior work\". In my perspective, the key difference of channel sampling is the retained channel number is always fixed to K, the non-selected channels are not zeroed, where the dropout usually has only a probability K / total_channel and non-selected feature is multiplied to a zero constant. \n\nThus, I suggest authors provide additional experiments as in Table 3 to compare the original drop-path with proposed channel sampling. Considering the test error drops from 3% to 2.67% while using PC, it will be more convincing to show the original drop path with probability K / total_channel yields a smaller drop to evidence the effectiveness of proposed sampling. \n\n- Proposed edge normalization is not a new sampling policy but a new search space.\nTo my understanding, this edge normalization is effectively a change to the search space rather than the sampling policy, and generalize to many other policies as well, and can be a substantial contribution to the NAS community. However, under current experiments setting, it is hard to isolate the improvement is from this new space or the channel sampling, as detailed later.\n\n- About the motivation.\nThroughout the paper, in abstract, introduction, section 3.2 and section 4.4, the authors claim that the larger batch size is particularly important for the stability of architecture search which is not well-studied and lack of references. From Table 4, it is hard to tell the stability is from the larger batch size or the proposed partial channel sampling.\n\n\n- Questions about experiments\n\n1. Experiments comparing to the baseline is not fair. \nAs in Section 4.2, the CIFAR-10 search is different from the original DARTS and P-DARTS in the following manner. The batch size is changed from 64(in DARTS)/96(in P-DARTS) to 256,  super-net is freezed for the first 15 epochs, and introducing the edge normalization parameter \\beta_{i,j} increase the search space.  With all these changes, it is quite hard to isolate the effectiveness of proposed PC-DARTS. Two possible simple experiments to compare is, using the original DARTS space and training set, 1) do not update the \\beta but use a fixed initialization that all \\beta is the same (to mimic original DARTS concatenation); 2) add \\beta to original DARTS as well and re-run 1). \n\nIt is completely reasonable to me the contribution of this paper is introducing a novel edge-normalization that is simple and effective to improve the DARTS based approach. If so, the authors could revise the conclusion easily. However, in the least scenario, the experiment comparison should be in a fair way.\n\n2. In original DARTS, error drop from 3% for the first-order gradient to 2.76% while using the second-order one, will this trend occurs with PC-DARTS?\n\n3. Robustness \nRecent work about evaluating neural architecture search reveals that NAS algorithms are sensitive to random initialization[1,2] and the search space [3], this in general leads to a notorious reproducibility problem of current NAS and shows it is not reasonable to only compare final performances on proxy tasks over **one** searched architecture. However, in the stability study in Section 4.4.3, multiple runs are still over the same architecture discovered in earlier experiments. In Section 4.4.2, the paper mentioned the search runs multiple times, yet the reported results in Table 3 are against the single run, as indicated by CIFAR-10 no PC- no EN error 3.00 +- 0.14, which is identical to the results in Table 1 DARTS (1st-order). Could the authors report the results with at least 3 different initializations, and possibly release the seeds? It would significantly strengthen the effectiveness of the proposed approach.\n\nMinor comments\n\n- According to Section 4.4.1 and Figure 3, change the K from 1 to 8, the search cost drops significantly. Does this mean the batch size in the ablation study is changing all the time? How could we know if the test-error is reduced due to the sampling ratio or to the batch size? \n\nTypos\n1. Table 2, caption below the table, \\dag is not aligned with the one in the used table.\n\n--- reference ---\n[1] Li and Talwalker, Random search and reproducibility of neural architecture search, UAI’19\n[2] Sciuto et al., Evaluating the search phase of neural architecture search, arxiv’19\n[3] Radosavovic et al., On Network Design Spaces for Visual Recognition, ICCV'19.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJeA7lKatB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Summary ** \nThis paper proposes to improve the previously work DARTS in terms of the training efficiency, from the large memory and computing overheads. The authors propose a partially-connected DARTS (PC-DARTS) with two components: 1. Partial channel connection 2. edge normalization. To be detailed, they sample a small part of channels to perform connection and add edge normalization to eliminate the potential optimization problem. The results on CIFAR-10 and IamgeNet show the approach is effective, especially in ImageNet, the approach achieves SOTA results. \n\n** Strengths **\n1.\tThe research direction of reducing the training/memory effort of Neural Architecture Search is important, which is also very hot in nowadays. \n2.\tThe authors propose two components to perform the efficient search process, which are partial channel connection and edge normalization operations. These methods are reasonable to reduce the training effort. The authors are inspired by ShuffleNet or related research topics. \n3.\tThe approach is easy to follow and implement, the description of the method is also clear.\n4.\tThe experiments also show comparable results in CIFAR-10 and strong performance in ImageNet.\n\n** Weaknesses **\n1.\tThe operation of partial channel connection choses 1/k channels, the remain channels are directly added to the output. This operation somehow feels too straight to be reasonable, since the remain channel has larger weights (1.0) compared to previous weighted combination. Though the second edge normalization can eliminate little, but this modification still suffers from less careful design, for example, another \\alpha weighted combination? Besides, directly bypass is same as perform identity operation, is this right?\n2.\tThe motivation of edge normalization is somehow weak, as the authors are aware of this can also be applied to the original DARTS. From the ablation study, it also shows it works for the original DARTS, which makes the description of 3.3. to be not so convincing. Besides, in the first paragraph of 3.3, what does it mean that “weight-free operations often accumulate larger weights” compared to other operations? I feel the reason is that the weight-free operations are much easier to pass the gradients and easy to be trained. \n3.\tIn imageNet results, it seems P-DARTS significantly outperform PC-DARTS in terms of the search cost, and the accuracy is similar. This makes PC-DARTS approach to be embarrassing. \n4.\tOne general point is what the authors mentioned, indeed, for NAS, more training data involved in the training process is much more important compared to perform operations. Therefore, the advantage benefits from the less memory usage of 1/k selection and the more data in one mini-batch. This makes the design of current research directions to be different. Does it mean more training (longer time) and more GPU memory will significantly outperform current results? Even the SOTA approaches. \n5.\tMinor point: compared to ProxylessNAS which only samples two paths at each time, their method is much more efficient (though the binarization consumes much). What’s the most advantage of PC-DARTS compared to their method? What if combine their approach with edge normalization?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gXOxY3FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose with this paper a simple extension of DARTS, a popular neural architecture search (NAS) method. This extension addresses one of the shortcoming of DARTS: the immense memory cost. This achieved in a simple way. Instead of using all channels only a random subset is used. To account for that, the authors propose a method to normalizes edges.\n\nThe description of the method is very clear. The related work covers many works. I suggest to focus more on the more recent work on NAS and particular work that follows the core idea of DARTS. This deserves more than used two very short sentences since this is the most related work. The experimental section leaves no question unanswered. The setup is clearly described in all cases, ablation studies are conducted wherever it is needed. The method is evaluated on both CIFAR-10 and ImageNet and is even transferred to MS COCO. The search results show some improvements with respect to time budget.\nConcluding, this might not be a ground-breaking paper but it is well made and I see no obvious flaws so I do not see any reason to reject it.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an improvement to the popular DARTS approach, speeding it up by performing the search in a subset of channels. The improvements are robust, and code is available for reproducibility.

The rebuttal cleared up initial concerns, and after the (private) discussion among reviewers now all reviewers give accepting scores. Because the improvements seem somewhat incremental and only applied to DARTS, R3 argued against an oral, and even the most positive reviewer agreed that a poster format would be best for presentation. 

I therefore strongly recommend recommendation, as a poster.
Decision: Accept (Spotlight)