CNAS: Channel-Level Neural Architecture Search
Authors: Heechul Lim
Min-Soo Kim
Jinjun Xiong
Venue:  (2020)
Keywords: Neural architecture search

Abstract: There is growing interest in automating designing good neural network architectures. The NAS methods proposed recently have significantly reduced architecture search cost by sharing parameters, but there is still a challenging problem of designing search space. We consider search space is typically defined with its shape and a set of operations and propose a channel-level architecture search\,(CNAS) method using only a fixed type of operation. The resulting architecture is sparse in terms of channel and has different topology at different cell. The experimental results for CIFAR-10 and ImageNet show that a fine-granular and sparse model searched by CNAS achieves very competitive performance with dense models searched by the existing methods.

Under review as a conference paper at ICLR 2020
CNAS: CHANNEL-LEVEL NEURAL ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
There is growing interest in automating designing good neural network architec- tures. The NAS methods proposed recently have significantly reduced architecture search cost by sharing parameters, but there still challenging problem of design- ing search space. We consider search space is typically defined with its shape and set of operations and propose a channel-level architecture search (CNAS) method using only fixed type of operation. The resulting architecture is sparse in terms of channel and has different topology at different cell. The experimental results for CIFAR-10 and ImageNet show that ine-granular and sparse model searched by CNAS achieves very competitive performance with dense models searched by the existing methods.
INTRODUCTION
Nowadays, deep neural networks (DNNs) used extensively and successfully in many fields and applications such as computer vision, speech recognition, machine translation, and automated vehicles. Designing DNNs often requires significant architecture engineering, large amount of trial and error by experts. Although transfer learning is widely used to save the efforts required for designing good architectures of DNNs from scratch, is not always possible to use.
Recently, there is growing interest in automating designing good neural network architectures 30 31 7 (2 4 [2]
20 24 21 3 28 14 2 (2
1O). Most of them can be categorized into reinforcement learning-based (RL) methods, evolutionary algorithm-based (EV) methods, hypernetwork- (HY)
methods, and gradient-based (GR) methods, in terms of the search algorithm.
use controller model that enumerates a bunch of candidate models, which are trained for fixed number of epochs from scratch, and then, is updated using the validation accuracies of the candidate models evaluated on validation set. To reduce the search space of candidate models, some them (31] 20) assume each model composed of multiple convolutional layers called cells having the same architecture and focuses searching for the best cell architecture. For example, in NASNet (II, a cell is composed of five blocks, and each block composed of two operations, which are selected among a of various convolution and pooling operations by the controller model. To reduce the search space, NASNet also transfers the learned architecture for a small dataset (e.g.. CIFAR- to a large (a.aset(e.g.. ImageNet). To optimize an architecture with less amount of computation, ENAS (20) exploits parameter (weight) sharing, which avoids training each candidate model from scratch by sharing the weights of candidate models. It constructs large computational graph, where each subgraph represents the architecture of a candidate model, and the controller model trained search for a subgraph corresponding a good candidate model.
RL methods 30 31 20 24
EV methods (23) 12 17 26, (5) also have been extensively studied. AmoebaNet (2T) uses the same search space with NASNet, but searches a good cell architecture based on evolutionary algorithm instead of RL controller. The population is initialized with models with random architectures, and some models are sampled from the population. The model with the highest validation fitness within the samples is selected as the parent (i.e., exploitation), and child having mutation in terms of operations and skip connections is constructed from the parent (i.e., exploration). Hierarchical NAS (15) uses hierarchical representation for cell architecture where smaller graph motifs used as building blocks to form larger motifs, instead of flat representation. Unfortunately. most of RL and EV methods, except ENAS, require an enormous amount of computing power for training thousands

---

Under review as a conference paper at ICLR 2020
of child models. They usually need hundreds thousands of GPU days for architecture search, which is almost impossible for a typical machine learning practitioner.
HY methods and GR methods avoid such a large cost of architecture search by sharing parameters as in ENAS (20). HY methods (3) (28) bypass fully training candidate models by instead training an auxiliary model, HyperNet (8) to dynamically and directly generate the weights of candidate model. SMASH (3 generates an architecture of an entire network (i.e., macro search) in terms of the hyperparameters of filters (e.g., number, size) with fixing the type of operation in the HyperNet space, while GHN (28) generates an architecture of a cell (i.e., micro search) in terms of operations in the NAS search space.
GR methods (22 7 27 29 10) do not rely on controllers, evolutionary algorithm, and hypernet- works, but exploit gradient descent on network architectures, which can significantly improve the speed of NAS. They basically relax the search space to be continuous, so that the architecture can be optimized with respect to its validation set performance by gradient descent. Here, search space corresponds to parent network, and child network (subgraph) can be derived from the parent network by gradient descent. Most of GR methods focus on searching a good cell architecture in terms of operations and repeating the same architecture in NASNet. After architecture search, they should usually re- re-train the candidate architecture snapshot from scratch using the training set due to inconsistency between the performance of derived child networks and converged parent networks.
As described above, one of the major trends in NAS is exploiting the concept of parameter sharing through hypernetwork or gradient descent in order to reduce the cost (i.e., GPU days) of NAS. By parameter sharing, HY and GR methods can automatically optimize an architecture that can achieve the state-of-the-an performance on CI and ImageNet just within few days (as summarized in Table p. However, there is still a challenging problem in the above architecture search methods: designing search space. In principle, the search space should be large and expressive enough to capture a diverse set of promising candidate models, and at the same time, should be small enough to train with the limited amount of resources and time (2). Some methods (24 [5] (22) addressed that defining search space is extremely important for the performance of neural architecture search. The problem about designing search space may not be solved at once. The search space of the existing NAS methods is typically defined with shape of the overall network and a set of operations such as identity, normal convolution, separable convolution, average pooling, and max pooling. Many of them follow the NASNet search space for the shape of the network (i.e., stacking cells) and define their own set of operations. Since the number of possible types of operations for search space is limited due to the search cost, the set of operations used itself may have large impact on the performance of architecture search.
In this paper, we investigate the possibility of achieving competitive performance with the state-of-the the art architecture search methods with using a fixed type of operation. To achieve such a performance, we focus on the sparsity of a model. A candidate model in the existing methods has multiple types of operations connected with each other via skip connections, and each operation takes the entire feature maps (called channel) of certain previous nodes or cells as input and returns its entire resulting channels as output. Thus. the candidate model can be regarded as a dense model in term of input and output channels of the operations. We propose channel-level neural architecture search (CNAS) method that regards channels as vertices and a single fixed operation as edges and searches for good architecture by gradient descent. The resulting model is sparse in terms of channels. CNAS uses the existing shape of search space (e.g., NASNet), but performs macro search. Thus, the resulting architecture has different topology at different cells. In CNAS, the final sparse architecture can be searched quickly due to its simplicity, and at the same time, can compensate for the disadvantage of using homogeneous operation due to its sparsity. For CIFAR-10, CNÁS searches for the architecture in 1.1 GPU days, which achieves 2.28% test error with 4.6 million parameters and autoaugment.
The rest of the paper is organized as follows. Section 2 exxlains our method CNAS. Section shows the experimental results, and Section 4 summarizes the characteristies of related methods. Section concludes this paper.
2

---

Under review as a conference paper at ICLR 2020
NCell: Normal
Fixed edges
RCell: Reduction cell
Edges which be removed
[d,,2]] = ( "y ag, 2s 4,D))
Output
Cell
[dg.]]] = (CCd)]]==CCCCooeeeuuCCC by b,, byb.)
(o 4,1, dd)
NCells
X
Node
2, (oncat((4, 4,)
Dv.3%] = (x
RCell
2g
Node
[g,2g] = Xddi(19.. Dz.
NCells
X
Concar([4,, [2, (,)
RCell
Node 1
Node
NCells
X
Input
0. B a.
b. b be
(6)
(b)
(c)
Figure 1: Diagram of search space of CNAS. (a) Shape of search space. We usually set N = 6. (b) Structure of an example cell of the model. The small white squares indicate channels. Both }ai} and } are the input channels of the cell. {c, are the output channels of the cell. The solid edges are fixed operations, and the dotted edges can be removed depending on the existence of the input channels {. The thick edges indicate series of operations, batch normalization (BN), depthwise separable 3x3 convolution (Conv), and Relu, while the thin edges simple operations such as concatenation and add. (c) Structure of an example node in the cell. Each node has two operations (of the same type). { are input channels of the node, while {yr} are intermediate output channels. The red input channels and red thick dotted edges removed, while the blue input channels and blue thick dotted edges remain as a part of the final sparse model.
CNAS METHOD
Since focus investigating the possibility of architecture search relying sparsity instead of the combination of operations in this paper, we mainly use the structure of NASNet for the shape of search space, which is composed of normal cells and reduction cells, and each cell is again composed of submodules called nodes (blocks in NASNet). In Section 3. we will show the result of CNAS using different shape of search space, particular, the structure of DenseNet. Figure [ shows the diagram of the search space of CNAS.
2.1 SEARCH SPACE
The CNAS method consists of the following three steps: (1) Train the one-shot (i.e., full-edges) model in fixed number of epochs to make it predictive of the validation accuracies of sparse models. (2) Search the most promising sparse model satisfying given sparsity based on criteria (e.g., Taylor) by zeroing out less important channels. (3) Re-train the most promising model from scratch (called CNAS-R) or fine-tune (called CNAS-W) and then evaluate the final model on the test dataset.
In CNAS, vertex in a cell or a node is single channel, and an edge is an operation. In Figure lb) and (c), the thick edges are non-trivial operations involving convolution, where the solid thick ones are for pre-processing in other methods, and the dotted solid thick ones the part that can be changed by architecture search. The type of operation used in CNAS is fixed as a specific one, depthwise separable 3x3 convolution since operations contrast, the number of types operations in the existing NAS methods is at least several, and the types of operations are designed differently depending on the method.
not the target of architecture search. In
3

---

Under review as a conference paper at ICLR 2020
In Figure [[c,, zeroing out less important channels of red squares also removes their outgoing edges to the next layer ],,,,.e., i.e., does not apply the operation BN (Con((ReLU() to the red squares. In terms of low-level implementation. CNAS performs partial matrix operations between {yi, y/3 and (#3, J and between (y3, y4) and (#5) Since the red edges do not need to be calculated, the corresponding convolution kernels also not necessary. Thus, the number of weight parameters between { and {yi is reduced by 8 in Node 1. That is, channel-level architecture search makes the model sparse. If {2 J2, 33, F4 all are removed, then {yı, 3/2 are also removed, and only {us, 3 is added to {21, 2-)- In general, the input channels of nodes (i.e., dotted squares) are removed differently depending on whether the cell is close to input data or close to the output layer. Thus, after architecture search, each cell in CNAS has different architecture in terms of the topology of vertices and edges.
2.2 SEARCHING THE MOST PROMISING SPARSE MODEL
We search for the most promising sparse model satisfying given sparsity p by zeroing out less important channels. Here, P indicates TWT one-shot model, and | that of the final sparse model (0.0 < p S 1.0). As the criteria for evaluating the importance of channels, we adopt Taylor expansion (8). Algorithm[sshow the outline of the evaluation. We denote the vector of entire input channels { in the one-shot model as X and the length of X as [ Likewise, we denote the vector of entire gradients of X after a single minibatch as use the current sparse model W', which is initially the same with W, and the parameters of W' are not updated. X' and AX' are the vector of entire input channels and their gradients in the current sparse model, respectively. The saliency vector S has the same length with XX and is initialized with zeros. We consider m. minibatches for the input dataset D. Then, we get X' and A in each minibatch and calculate Taylor expansion using element-wise multiplication between both. The smaller xy O ōx, is, the larger the value x, ou; is. The dimension of the value is reduced to single value, which is again accumulated to the corresponding saliency value in S. Then, we normalize S by applying layer-wise L2-normalization.
_ where |W] is the number of weight parameters of the
AX (dr;,. J. When calculating gradients,
Algorithm 1: Calculation of Taylor expansion for CNAS
for each Dr [D1,... Dm do
X', AX' +- -orvaardAnddaaeppoppaatttnn Dj. W")
S < S + Dimension
X
< Normalization(S)
After calculating the saliency vector S, we gradually zero out the input channels {x;} having the largest values, i.e., least important channels, among the remaining input channels. We let the ratio of zeroing out 7 <Y < 1). We typically use 7 0.1, which means removing 10% input channels of the remaining input channels at each iteration. Thus, the number of input channels becomes X| after the first iteration and 0.81| after the second iteration. We perform fine-tuning of single epoch for the current sparse model W' between iterations. As the iteration goes on, the model becomes sparser and sparser. We stop the iterations when the sparsity of W' reaches the given p. After finding the final sparse model W*, we can initialize the parameters of W* and re-train the model (called CNAS-R), or just fine-tume nue-tuneeee the parameters of W (called CNAS-W).
We incorporate spatial dropout 25) at training the one-shot model or the final sparse model in order to make that the model more robust. We do not use path dropout used in ENAS (20) and One-Shot ( since it is too coarse to incorporate for our channel-level search. We also do not use conventional drouout (9) since is too fine- une-ggrined to apply. Although One-Shot (2) consider the co-adaptation issue in which zeroing out operations from the one-shot model can cause the quality of the model's prediction to degrade severely, we do not need to consider it since the final sparse model is obtained through gradually zeroing out by the ratio 7-
We check the correlation between the one-shot model and the final sparse model in terms of perfor- mance (test error). We generate 27 pairs of one-shot models of three cells and five nodes per cell with different initialization and train them for 150 epochs. Then, we search a single final sparse model from each one-shot model and train them for 310 epochs. Figure a shows strong correlation
4

---

Under review as a conference paper at ICLR 2020
10.0%
DIFFEREN OR TEST TEST ERROR
0.7%
9.5%
0.6%
0.5%
TEST TEST ERROR
9.0%
MNASNet
(OPTIMA))
No-prune
0.3%
8.5%
ENAS micro
0.2%
Random
8.0%
0.1%
CNAS
7.5%
-0.1%
0.15%
0.25%
0.33
0.4%
0.5%
7.0%
-0.2%
DIFFERENCE OF TEST ERROR
0.9
0.3
0.4
0.3
(ONE-SHOT)
SPARSITY p
(a)
(b)
Figure 2: Results of sparse models. (a) Correlation between the one-shot and the final sparse model in terms of test error. (b) Comparison among no- no-pruning, randomly pruning, and our pruning (all have the same number of parameters of 0.15 M).
between the one-shot and the final sparse model in terms of test error. We let E(-) test error. X-axis means E(W1) 2(W2) where W, and W2 are a pair of one-shot models s.t. E(W1) > E(W2). Y-axis means E(Wi) E(W;) where Wi and W are the final sparse models of W1 and W2. respectively. There are 27 points in the figure, and only two points are located below 0.0 at Y-axis. For the remaining 25 points, W1 is better than W2, then WT is also better than W The test error of the one-shot model is computed before architecture search. There is fine-tuning for the one-shot model. The test еrror of the optimal model is computed after fine-tuning. The correlation coefficient between X-axis and Y-axis is about 0.83. It means the way of searching the final sparse model is stable.
2.3 TOPOLOGICAL PROPERTIES OF THE FINAL SPARSE MODEL
We describe the topological properties of the final sparse model W after channel-level architecture search. Table[[shows the statistics of W compared with those of the one-shot model for CIFAR-10 10. Due to the space limit, show only the top three cells, the bottom three cells and two reduction cells among 20 cells. From the statistics, we address two properties. First, the input channels in reduction cells
removed as much in other top and bottom cells. For example, in Cell 18, XX is smaller than one- fifth of [X]. In contrast, in Cell 14, | 5, 233 is almost the same with [X| = 5, 760 reduction cells, the height and width of channel is reduced by half, while the number of channels is increased by two times. As result, the amount of information is reduced by half. Keeping input channels at reduction cells seems to be due to compensating the loss of information to achieve the lower error. Second, |z x|| is extremely low compared with Id + æ in the top cells, whereas |z + z| is similar with (d + x in the bottom cells. The former means that there is almost no edge among the nodes in the top cells and the nodes are located horizontally, each of which doing its own task. The latter means that there are a lot of edges among the nodes in the bottom cells and the nodes are located vertically and horizontally in Figure I with the necessity of aggressive abstraction.
EXPERIMENTS
We use CIFAR-10 10 (33 and ImageNet (6) for our experiments. For training the one-shot model of CIFAR-10, 10, we use 150 epochs with the Nesterov momentum (19) 0.9. We used a cosine learning rate schedule (6) with the initial learning rate (max = 0.05, the minimum learning rate (min = 0.0001, the initial number of epochs To 10, and the multiplication factor Tmaa 2 and & weight decay of 2 x 10-4 We train the final sparse model by using the same setting with the one-shot model, except the number of epochs, which is set 630. For ImageNet.
the same final sparse model for CIFAR- 10 only after adding two more stem convolution layers and modifying the fully connected layer to handle the different number of outputs. We 250 epochs for training the modified sparse
5

---

Under review as a conference paper at ICLR 2020
one-shot model
final sparse model 01.66)
cell
[x]
# params
[a0"]
# params
d x
2 -
20
5,760
1,440
1,088,600
795
328,995
791
4
19
5,760
1,440
1,0 1,088,640
646
306,198
644
2
18
5,760
1,440
1,088,640
1,010
361,890
969
41
14
5,760
1,440
984,960
5,233
904,329
2,537
2,696
7
2,880
720
259,200
2,244
207,
1,258
986
3
1,440
360
77,760
284
25,740
170
114
2 1,440
360
75,168
653
39,753
367
286
1
1,440
360
72,576
905
48,501
531
374
Total
69,120
17,280
9,880,704
26,038
4,548,078
17,322
8,716
Table I: Statistics of the one-shot model and the final sparse model in CNAS for CIFAR-10. The number of cells is 20, and the number of nodes per cell five. |X| and |Y] are the numbers of input channels and output channels in all nodes of the one-shot model, respectively. X is the number of imput channels of the final sparse model. means the output channels of nodes, and the input channels of cells after being preprocessed as in Figure 1(b). Cell 7 and cell 14 are reduction cells, and other cells are normal cells. Id 2| and |z + a| are the numbers of the edges from (d:) to {z,} } and from { to { in Figure respectively.
10%
9%
TEST ERROR
TEST ERROR
8%
8%
0.78
0.76
0.78
7%
7%
6%
5%
5%
3
4
5
10
2
NUMBER OF NODES
NUMBER OPERATIONS
(b)
Figure 3: Test error of CNAS with different settings. (a) Varying the number nodes per cell. The number of parameters of all models is 1.1 M regardless of the number of nodes. (b) Varying the number of operations per node. The number of parameters of all models is 0.15 M regardless of the number of operations.
model with the Nesterov momentum 0.9 and the learning rate 0.05, which decayed by factor 0.98 after each epoch. All test enrors in the result are the mean values of three evaluations.
3.1 EVALUATION OF CNAS VARYING THE NUMBER OF NODES AND OPERATIONS
We check the performance of CNAS models having the same number of parameters while varying the number of nodes per cell varying the number of operations per node. We use the one-shot model of two normal cells and one reduction cell. Figure a) shows the result of CNAS models, which all have I. M parameters,
different number of nodes per cell. The test error tends to be decreased as the number of nodes per cell increases (i.e., model becomes sparser), but slightly increases when the number of nodes ten (i.e., too sparse). Figure Bib) shows the result of CNAS models, which all have 0.15 15 M parameters, but different number of operations (of the same type) per cell. Although use two operations per cell to follow the convention of NASNet, the difference in test error among three settings is quite small. This is mainly due to using single type of operation (edge).

---

Under review as a conference paper at ICLR 2020
3.2 COMPARISON AMONG DIFFERENT PRUNING METHODS FOR CNAS
In this section, we evaluate the performance of no-pruning, randomly pruning, and our pruning (in Section 2.2). Here, no-pruning means making small one-shot model of 0.15 M parameters and training the model without the pruning step. Randomly pruning means pruning each cell randomly with a given sparsity (c.g., 0.9, 0.5, 0.4, 0.3) and making the final model all have the number of parameters of 0.15 M. Thus, this setting does not have different topology at each cell, but rather has similar topology. Our pruning also has the 0.15 M parameters, but different topology at each cell We use CIFAR-10 10 for comparison. Figure b)) shows the results of three settings.
as in Table I Among them, no-pruning shows the worst performance, while our pruning of CNAS shows the best performance. It means that sparse model in terms of channel improves the performance compared with a dense model, and at the same time, different topology at each cell is important to achieve the better performance. Just for reference, the same number of parameters of 0.15 M. Both No-prune and ENAS are dense models in terms of channel, but ENAS shows a better performance than No-prune due to its various operations in search space. MNASNet (24) shows slightly worse performance than No-prune since its architecture is the one optimized for ImageNet.
add the results of MNASNet (24) and ENAS (20) having
3.3 COMPARISON WITH OTHER METHODS
In this section, we present the comparison result with the state-of-the-art methods for CIFAR-10 and ImageNet. Table 2 shows the model size, test errors and GPU days (for architecture search methods) for CIFAR-I0. For CNAS, we measure the first two steps, i.e., training the one-shot model and searching the final sparse model, as the GPU days for architecture search. The CNAS model used in the comparison is the same as the final sparse model in Table Overall, CNAS achieves very competitive performance with only 1.1 GPU days and moderate number of parameters (4.6 M) among the methods compared. CNAS with autoaugment (5) can further improve the performance up to 2.28% test error.
Table shows the comparison result for ImageNet. The model size of CNAS slightly increases to 5.7 M due to adding two stem convolution layers to and modifying the fully connected layer of the final sparse model in Table T We denote this model just as CNAS. Overall, CNAS achieves comparable performance with other methods but does not show very competitive performance as in CIFAR-10. It is probably because we use the same final sparse model obtained from CIFAR-10 10 for ImageNet due to the limit of evaluation time. Searching and training inherent final sparse model from Iet may further improve the performance with spending more GPU days for architecture search.
3.4 USING DENSENET-LIKE SEARCH SPACE FOR CNAS
DenseNet- BC (IT) search space instead of NASNet search space. Figure 4 shows the diagram of the search space for CNAS. In Figure 4 a), each dense block consists of the 19 bottleneck layers, and there are transition layers between dense blocks. In each bottleneck layer in Figure 4b). zeroing out less important channels in red squares also removes their outgoing edges to the next layer {wr). are concatenated to in output to make skip connection. The number of æ increases as the bottleneck layer number increases in DenseNet. In Table CNAS-R (DenseNet-BC) outperforms the original DenseNet-BC with the same number of parameters of 0.8 M. This means our CNAS method is effective not only NASNet search space, but also different shapes of search space.
In this section, we apply CNAS to different shape of search space. In particular,
We note that the performance of CNAS-R (DenseNet-BC) with 4.6 M parameters is worse than that of CNAS-R using NASNet search space in Table 2 This means the shape of the search space of NASNet itself is superior to that of DenseNet.
4 RELATED WORK
We have briefly explained the recently proposed NAS methods according to the search algorithm in Section 1. Table 4 summarizes their characteristics in terms of not only search algorithm, but also search space, search range and how to generate candidate parameters.
7

---

Under review as a conference paper at ICLR 2020
GPU days for architecture search
Methods
params (x 10%)
Test error (%)
SMASH (3)
16.0
1.5
4.03
NAS (30)
37.4
16,800
3.65
Hierarchical NAS (5
61.3
300
3,63
Progressive NAS 14
3.2
150
3.63
One-shot Top (F=64) (2
1..4=1100
3,3+0
0.1
One-shot Small (F=64)
5.0:
3,3+0
4.0
NASNet-A 31
3.3
1,350
3.41
ENAS macro
38.0
0.45
3.87
ENAS micro 20)
4.6
0.45
3.54
GDAS
3,4
0.21
3,87
GDAS (FRC) (7)
2.5
0.17
3.75
DARTS + cutout (22
2.9
1,5
2.94
DARTS cutout (22)
3.4
4
2.83+0.06
AmoebaNet-A + cutout (21)
3.2
3150
0.06
AmoebaNet-BB + cutout (1
2.8
3150
2.06
Petridish cell + cutout (10)
3.2
5
2.75
SNAS (mild constraint) + cutout
2.9
1.5
2.98
GHN Top-Best cutout (28
5.7
0.84
2.8470.07
DSO-NAS -C0-NAS-share-- cutout
3.0
1
2.84
DenseNet-BC (11
0.8
4.51
CNAS-R (DenseNet-Bl BC)
0.8
0.18
4.28
CNAS-R (DenseNettBCC
4.6
0.67
3.97
CNAS-R
4.6
1.1
3.49
CNAS-W
4.6
1.1
3.40
CNAS-R cutout
4.6
1.1
2.94
CNAS-W + cutout
4.6
1.1
2.77
CNAS-R autoaugmented 5
4.6
1.1
2.39
CNAS-W + autoaugmented
4.6
1.1
2.28
Table 2: Comparison results among the state-of-the-art architecture search methods for CIFAR-10.
GPU days for architecture search
Methods
# params (x10%)
Top-1
Top-5
(31
5.3
I,350
74
91.3
DARTS (22)
4.9
73.1
91
MNASNet-92 (24)
4.4
1666
74.8
92.1
One-shot Top (F=24)
(2
6.8
3.3+a
73.8++0.4
One-shot small (F=32) (2
5.1
3.3+a
74.2+0.33
AmoebaNet-A 21
5.1
3150
74.5
92
AmoebaNet-CC (1
6.4
3150
75.7
92.4
GDAS (
5.3
0.21
74
91.5
GDAS (FRC)
4.4
0.17
72.5
90.9
Petridish cell
4.8
73.7
SNAS + mild constraint 27
4.3
1.5
72.7
90.8
GHN Top-Bar Best 228
6.1
0.84
73
91.3
DSO-NAS
29
4.7
73.8
91.4
DSO-NAS-shan share 29
4.8
6
74.6
91.6
CNAS-R
5.7
74
91.9
Table 3: Comparison results among the state-of tfrt architecture search methods for ImageNet.
8

---

Under review as a conference paper at ICLR 2020
Fixed edges
Output
Edges which can removed
BLayer
19
bu.5s Comu(Rall]) (g.
Tlaayer
[ Conv(ReLU (BN(), (.DD)))))
BI ayer: Rottleneck layers
TLayer: Transition layer
BLayer
19
"3. ) Concut , Pg
2. X
Tlayer
BLayer
Blayer
X 19
Input
(a)
(b)
Figure 4: Diagram of DenseNet-like search space for CNAS. The meaning of boxes and edges is the
same in Figure
Search space
Search algorithm
Search
Candidate parameters
Search cost (GPU days)
Model
range
NAS (30)
OS
RL
Macro
Independent
16800
NASNet (31)
OS
RL
Micro
Independent
1350
ENAS (00
OS
RL
Micro,Mach
Shared
0.45,0.32
MNASNet (24)
OS
RL
Macro
Independent
1666*
AmoebaNet (21)
OS
EV
Micro
Independent
3150
Hierarchical NAS 155
OS
EV
Micro
Independent
300
SMASH (3)
HS
HY
Macro
Dynamic
1.5
GHN (28)
OS
HY
Micro
Dynamic
0.84
Progressive NAS
OS
PD
Micro
Independent
225
One-shot 2
OS
RS
Micro
Shared
3.3+a
DARTS (22)
OS
GR
Micro
Shared
1.5
GDAS (
OS
GR
Mícro
Shared
0.17
SNAS (27)
OS
GR
Micro
Shared
1.5
Proxyles.NAS (4)
OS
GR.RL
Macro
Shared
8.3
DSO-NAS 29
OS
GR
Micro,Macro
Shared
1,6
Petridish 110
OS
GR
Macro,Macro
Shared
5,5
CNAS
CS
GR
Macro
Shared
1.1
Table 4: Characteristics of recently proposed NAS methods. Search space: OS (Operations and skip connections), HS (Hyperparameters of filters and skip connections) and CS (Channels and skip connections). Search algorithm: PD (Performance prediction) and RS (Random Search). Search range: Micro (Cell-level search and repeating the structure), Macro (Network-level search), Both (Cell- and Network-level search). Candidate parameters: Independent (generating parameters Independently), Shared (exploiting Shared parameters) and Dynamic (generating parameters dynamically). Search cost (GPU-days): All numbers except (*) are the costs of searching for CIFAR-10, (*) mean the costs for ImageNet due numbers for CIFAR-10 (i.e., no proxy) in the corresponding papers.
CONCLUSIONS
In this paper, we proposed a channel-level neural architecture search (CNAS) method that considers channels instead of operations for search space of NAS. It only uses single fixed type of operation and instead focuses searching for good sparse architecture in terms of channel. The resulting sparse model has different topology at different cell. particular, the nodes in the bottom cells are located vertically and horizontally with the necessity of aggressive abstraction, but the ones in the top cells are located horizontally for doing their own tasks. For CIFAR-10, CNAS achieves 2.28% test error with 4.6 million parameters using the architecture searched for only 1.1 GPU days. We also showed that CNAS is effective in not only NASNet search space, but also different shapes of search space.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Bayer, D. Wierstra, J. Togelius, and J. Schmidhuber. Evolving memory cell structures for sequence
learning. ICANN, 2009.
G. Bender, P.-. Kindermans, B. Zoph, V. Vasudevan, and Q. Le. Understanding and simplifying one-shot
architecture search. JCML, 2018.
[3] A. Brock, Lim, J. M. Ritchie, and N. Weston. Smash: one-shot model architecture search through
hyperettorks.. ICI.R, 2018.
[4] H. Cai, L. Zhu, and S. Han. ProxylessNAS: Direct neural architecture search on target task and hardware.
In ICLR, 2019.
[5] E. D. Cubuk, B. Zoph, D. Mané, V. Vasudevan, and Q. V. Le. Autoaugment: Learning augmentation
policies from data. CoRR, 2018.
[6] J. Deng. W. Dong, R. Socher. L.-J. K. Li, and L. Fei Fei. Imagenet: A large-scale hierarchical image
database. CVPR, 2009. Available at inttp:: /imaaamaaeeaaaaa net org download
[7] X. Dong and Y. Yang. Searching for robust neural architecture in four gpu hours. In CVPR, 2019.
[8] D. Ha, A. M. Dai, and Q. V. Le. Hypernetworks, In ICLR, 2017.
[9] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Improving neural
networks by preventing co-adaptation of feature detectors. arXiv:/207.088 220700580. 2012.
[10] H. Hu, J. Langford. R. Caruana, S. Mukherjee, E. Horvitz, and D. Dey. Efficient forward architecture
search. In NIPS. 2019.
[11] G. Huang, Z. Liu, L. Maaten, and K. Q. Weinberger. Densely connected convolutional networks. CVPR,
2017.
[12] R. Jozefowicz, W. Zaremba, and Sutskever. An empirical exploration of recurrent network architectures.
ICML, 2015.
[13] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009. Available ltttps://www. as wtt.oott..ee//e.ii/iiiiiii..oooe Famml
[14] C. Liu, B. Zoph, J. Shlens, W. Hua, L.-J. L. Fei-Fel, Yuille, Huang, and Murphy. Progressive neural architecture search. ECCV, 2018.
[15] Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukeuoglu. Hierarchical representations for efficient architecture search. ICI.R, 2018.
[16] Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2017.
[17] R. Miikkulainen, J. Liang. E. Meyerson, A. Rawal, D. Fink, Francon, B. Raju, A. Navruzyan, N. Duffy,
and B. Hodjat. Evolving deep neural networks. arXiv preprint
2017.
[18] P. Molchanov, S. Tyree. T. Karras, T. Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient inference. ICLR. 2017.
[19] Y. E. Nesterov. A method for solving the convex programming problem with convergence rate of o(1/k2).
Soviet Mathematics Doklady. 1983.
[20] H. Pham, M. Y. Guan, B. Zoph, Q V. Lc, and J. Dean. Efficient neural architecture search via parameter sharing. ICML, 2018.
[21] E. Real, A. Aggarwal, Y. Huang, and Q. Le.. Regularized evolution for image classifier architecture search. In AAAII 2019.
[22] R. Shin, C. Packer, and D. Song. Differentiable neural network architecture scarch. In ICLR (Workshop),
2018.
[23] K. O. Stanley and R. Mikkuulanen. Evolving neural networks through augmenting topologies. Evolutionary
computation, 2002.
[24] M. Tan, B. Chen, R. Pang, V. Vasudevan, and Q. V. Le. Mnasnet: Platform-aware neural architecture
search for mobile. CoRR, 2018.
[25] Tompson, R. Goroshin, Jain, LeCnn,, and C. Bregler. Efficient object localization using convolutional
networks. CVPR, 2015.
[26]
L. Xic and A. Yuille. Genetic cnn. ICCV, 2017.
[27]
S. Xie, H. Zheng, C. Liu, and L. Lin. SNAS: stochastic neural architecture search. ICLR, 2019.
[28]
C. Zhang, M. Ren, and Urtasun. Graph hypernetwors for neural architecture search. In ICLR, 2019.
[29]
X. Zhang, Z. Huang, and N. Wang. Single shot neural architecture search via direct sparse optimization. In
ICLR, 2019.
[30]
B. Zoph and Le. Neural architecture search with reinforcement learning. ICLR, 2017.
[31] B. Zoph, V. Vasudevan, J. Shlens, and Q V. L.e. Learning transferable architectures for scalable image
recognition. CVPR, 2018.
10


Reviews:
[{"review_id": "ryx2wOgP9H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to search a sparse but competitive architecture with using a single fixed type of operation by proposing a channel-level neural architecture search (CNAS). Different from most previous NAS works, this paper conducts NAS process on channel-level such that different cell has different topology. CNAS provides a heuristic algorithm to calculate the saliency vector and zero out the channels iteratively until satisfying a given sparsity. This paper performs CNAS on Cifar-10 and ImageNet, and analyzes the topological properties of the final model. The results of experiment demonstrate CNAS can reach a competitive model with dense models searched by baselines. \n\nThis paper provides us with a novel insight that searching neural architecture on the channel level instead of operation and connection level. However, it just combines NAS and pruning parts together, which lacks of novelty in the algorithm level.\n\nI lean to reject this paper because: (1) it lacks of novelty, (2) the experiment result is not convincing, (3) some related works are missed, (4) the expression of the paper is not clear.\n\nMain argument\n\nCNAS is a straightforward combination of NAS and pruning. As the author said in the section 2.1, CNAS method can be seen as two separate processes: training a supernet like one-shot NAS methods and then conducting pruning on the found supernet using Taylor expansion criteria. Both parts are the same as previous works almost and there is no innovation and improvements.\n\nMany related works are missed in the paper. One important step in CNAS is pruning, which uses Taylor expansion technic as previous work. However, it only introduces NAS in introduction section and related works section, ignoring the pruning process. From my view, the pruning part is more important than NAS part.\n\nFrom the results of the experiment, the improvement of CNAS is not convincing. I think the main focus of the paper is the sparsity, but in Table 2, the number of parameters of model is still larger (4.6*10^6) compared with some baselines like DARTS (2.9*10^6). Besides that, much space in the experiment section is devoted to the relationship between supernet and the final model, which is not so important. Because in other methods, supernet is just an intermedia. Therefore, The comparison between them is not so meaningful.\n\nThe paper is hard to understand because of unclear writing. For example, in algorithm part, the author doesn't make DimensionReduction function and its inputs clear. The author mentions the first input in the paragraph but how to combine with the second input? Also, the representation in Figure 1(b) is confused. It's hard to figure out \"the thick edges\", \"the solid thick ones\" and \"the dotted solid thick ones\".\n\nQuestions\n\n1. As we all known, operation set is important to the search space. Have you tried more types of operations? From my view, using only one fixed operation is unfair for CNAS compared with other methods.\n\n2. One of your focus is sparsity of the model. Can you explain the reason that you set the number of parameters to a large value (4.6*10^6)? Have you tried to use a larger sparsity value? What's the performance of CNAS when the model is sparser?\n \n3. In Table 2, there are some different tricks (cutout, autoaugmented) applied on some methods. Can you explain how you guarantee the fair comparison between different methods? If we just compare CNAS-R or CNAS-W, they are not better than baselines.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJxkVRfTFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper models the neural architecture search (NAS) as a network pruning problem, and propose a method to sparsify the super-net during the search of architectures.\n\nOverall, the novelty in this paper is not strong and their experimental performance is weak compared with recently published papers. I do not see a need to have such a new algorithm in the NAS literature. Please see the question below:\n\nQ1. \"Bayesnas: a bayesian approach for neural architecture search\". ICML 2019\n- This paper also takes a pruning's perspective for NAS, but it is much more efficient than the proposed one. Would the authors have some discussion and experimental comparison with this paper? Specifically, Bayesnas considers more complex sparse patterns then the submission.\n\nQ2. \"adaptive stochastic natural gradient method for one-shot neural architecture search\". ICML 2019\n- Could the authors have some discussion with this paper? This paper has comparable performance, but it is also much faster.\n\nQ3. What are the benefits of the proposed method? \n- From Tables 2 & 3, the proposed method is not better than STOA on the accuracy or number of parameters. \n- CNAS + autoaugmented can offer better accuracy, but the comparison is not fair as different pre-processing method is used.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlMlAIp5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to propose a novel framework for neural architecture search. Although there have been many solutions in the literature, the authors try to build a NAS model that is sparse in structure while being similarly effective as conventional dense models.  The method is straightforward - they select a single fixed operation as edges, and channels as vertices, and the problem of NAS can be directly solved by a gradient descent method. The sparsity can also be achieved on the level of channels.\n\nI have three major concerns, including a lack of novelty, unconvincing experiments, and poor presentation of the work. First, the proposed method is quite straightforward and can be viewed as a quick extension of existing structures. Simplifying the selection of operations make it easy for computation, while it also constrains the applications of the proposed framework. Second, the reported results do not seem to be promising since the improvement was marginal. It is also very difficult to tell whether the contribution is brought by the proposed sparse structure or the adoption of autoaugment since the baseline methods are not applied with it. Third, the paper has not been well written and there are grammatical mistakes throughout the manuscript. I attached the original abstract of the paper and my corrected version below.\n\nThere is growing interest in automating designing good neural network architectures. The NAS methods proposed recently have significantly reduced architecture search cost by sharing parameters, but there is still a challenging problem of designing search space. We consider search space is typically defined with its shape and a set of operations and propose a channel-level architecture search (CNAS) method using only a fixed type of operation. The resulting architecture is sparse in terms of channel and has different topology at different cell. The experimental results for CIFAR-10 and ImageNet show that a fine-granular and sparse model searched by CNAS achieves very competitive performance with dense models searched by the existing methods.\n\nThere is a growing interest in automating designing good neural network architectures. The NAS methods proposed recently have significantly reduced costs of architecture search by sharing parameters, but there is still a challenging problem of designing search space. \nConsidering that existing search space is typically defined with its shape and a set of operations, we propose a channel-level architecture search (CNAS) method using only a fixed type of operation. \nThe resultant architecture is sparse in terms of channels and it has different topologies at different cells. \nThe experimental results for CIFAR-10 and ImageNet show that a fine-granular and sparse model searched by CNAS achieves very competitive performance with dense models searched by existing methods.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJgFd7gpKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a channel pruning approach based one-shot neural architecture search (NAS). Unlike other NAS works that mostly search for operations/connections and topologies, this paper focuses on pruning channels for a fixed network.\n\nIn general, the idea of channel pruning has been extensively studied in previous works, and the channel pruning search algorithm is very similar previous one-shot NAS framework. The results on CIFAR-10 are reasonably good, but the results on ImageNet are not competitive to other NAS works.\n\nHere are some more comments:\n\n1. This paper is more like a new automated pruning technique rather than a new NAS technique. Therefore, I recommend the authors compare this technique with other pruning techniques, such as NetAdapt (https://arxiv.org/abs/1804.03230 ) and AMC (https://arxiv.org/abs/1802.03494).\n\n2. The baseline model described in Figure 1 is quite limited. It would be helpful if the authors can also apply this pruning technique to other types of models (such as NASNet-A/MNASNet-92 from your Table 3,  or mobilenets used in NetAdapt/AMC papers).\n\n3. Section 2.2 and Algorithm 1 is difficult to follow. It is not clear how Taylor expansion is carried out, and how saliency vector S is used. I recommend the authors expanding Algorithm 1 to include more details.\n\n4. Figure 2(b) shows random pruning leads to better results than no-pruning. This is kind of counter-intuitive, could you give more details about your settings and explanation?\n\n5. There are some minor errors: (1) Figure 1 [y1, y2] should be [z1, z2], and [y3, y4] should be [z3, z4];  (2) At the end of section 2.1, the number of weights in node 1 should be reduced by 4/8 instead of 3/8.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a channel pruning approach based one-shot neural architecture search (NAS). As agreed by all reviewers, it has limited novelty, and the method can be viewed as a straightforward combination of NAS and pruning. Experimental results are not convincing. The proposed method is not better than STOA on the accuracy or number of parameters. The setup is not fair, as the proposed method uses autoaugment while the other baselines do not. The authors should also compare with related methods such as Bayesnas, and other pruning techniques. Finally, the paper is poorly written, and many related works are missing.
Decision: Reject