AtomNAS: Fine-Grained End-to-End Neural Architecture Search
Authors: Jieru Mei
Yingwei Li
Xiaochen Lian
Xiaojie Jin
Linjie Yang
Alan Yuille
Jianchao Yang
Venue:  (2020)
Keywords: Neural Architecture Search
Image Classification

Abstract: Search space design is very critical to neural architecture search (NAS) algorithms. We propose a fine-grained search space comprised of atomic blocks, a minimal search unit that is much smaller than the ones used in recent NAS algorithms. This search space allows a mix of operations by composing different types of atomic blocks, while the search space in previous methods only allows homogeneous operations. Based on this search space, we propose a resource-aware architecture search framework which automatically assigns the computational resources (e.g., output channel numbers) for each operation by jointly considering the performance and the computational cost. In addition, to accelerate the search process, we propose a dynamic network shrinkage technique which prunes the atomic blocks with negligible influence on outputs on the fly.  Instead of a search-and-retrain two-stage paradigm, our method simultaneously searches and trains the target architecture. 
Our method achieves state-of-the-art performance under several FLOPs configurations on ImageNet with a small searching cost.
We open our entire codebase at: https://github.com/meijieru/AtomNAS.

Published as a conference paper at ICLR 2020
ATOMNAS: FINE-GRAINED END-TO-END NEURAL ARCHITECTURE SEARCH
Jieru Mei Yingwei Li Xiaochen Lian², Xiaojie Jin?, Linjie Yang', Alan Yuille' & Jianchao Yang² 1 Johns Hopkins University
ByteDance AI Lab
ned eieu gnail 1, yingwei 1 1ieg.u. (xisochen. jinxiaojie, wandie..angyyoyyeeeee
aen yot www.lleeeeemeeooommmm yemgii.ncccaodbrreeaar...oo
ABSTRACT
Search space design is very critical to neural architecture search (NAS) algo- rithms. We propose fine-gra fineggrinedd search space comprised of atomic blocks, a minimal search unit that is much smaller than the ones used in recent NAS algo- rithms. This search space allows a mix operations by composing different types of atomic blocks, while the search space in previous methods only allows homo- geneous operations. Based on this search space, we propose a resource-aware architecture search framework which automatically assigns the computational re- sources (e.g., output channel numbers) for each operation by jointly considering the performance and the computational cost. In addition, to accelerate the search process, we propose a dynamic network shrinkage technique which prunes the atomic blocks with negligible influence on outputs the fly. Instead of a search- and-retrain two-stage paradigm, our method simultaneously searches and trains the target architecture. Our method achieves state-of-the-as performance under several FLOPs configurations on ImageNet with a small searching cost. We open our entire codebase at: https: //tgtttt..////////////////tttbht com me fieru/ omNAS.
INTRODUCTION
Human-designed neural networks are already surpassed by machine-designed ones. Neural Ar- chitecture Search (NAS) has become the mainstream approach discover efficient and powerful network structures (Zoph & Le (2017); Pham et al. (2018): Tan et al. (2019); Liu et al. (2019a)). Although the tedious searching process is conducted by machines, humans still involve extensively in the design of the NAS algorithms. Designing of search spaces is critical for NAS algorithms and different choices have been explored. Cai et al. (2019) and Wu et al. (2019) utilize supernets with multiple choices in each layer to accommodate a sampled network on the GPU. Chen et al. (2019b) progressively grow the depth of the supernet and remove unnecessary blocks during the search. Tan & Le (2019a) propose to search the scaling factor of image resolution, channel multiplier and layer numbers in scenarios with different computation budgets. Stamoulis et al. (2019a) propose to use different kernel sizes in each layer of the supernet and reuse the weights of larger kernels for small kernels. Howard et al. (2019); Tan & Le (2019b) adopts Inverted Residuals with Linear Bottle- necks (MobileNetV2 block) (Sandler et al., 2018), building block with light-weighted depth-wise convolutions for highly efficient networks in mobile scenarios.
However, the proposed search spaces generally have only small set of choices for each block. DARTS and related methods (Liu et al., 2019a; Chen et al., 2019b; Liang et al., 2019) use around 10 different operations between two network nodes. Howard et al. (2019); Cai et al. (2019): Wu et al. (2019); Stamoulis et al. (2019a) search the expansion ratios the MobileNetV2 block but still limit them few discrete values. We argue that search space of finer granularity is critical find
This work done during the internship program Bytedance.

---

Published as a conference paper at ICLR 2020
optimal neural architectures. Specifically, the searched building block in supernet should be as small as possible to generate the most diversified model structures.
We revisit the architectures of state-of-the-ar networks (Howard et al. (2019); Tan & Le (2019b); He et al. (2016)) and discover a commonly used building structure: convolution channel- wise operation convolution. We reinterpret this building structure as an ensemble of computationally independent blocks, which we call atomic blocks. As the minimum search unit, the atomic block constitutes much larger and more fine-graines search space, within which we are able search for mixed operations (e.g., convolutions with different kernel sizes and their channel numbers).
For the efficient exploration of the new search space, we propose NAS framework named Atom- NAS which applies network pruning techniques to architecture search. Specifically, we start from an initial large supernet and rewrite every convolution channel-wii operation convolution structure of it in the form the weighted sum of atomic blocks; the weights reflect the contribution of the atomic blocks to the network capacity and are called importance factors. For each atomic block, a penalty term in proportion to its FLOPs is enforced on its importance factor; effectively, the penalty makes AtomNAS favor atomic blocks with less FLOPs. By minimizing the combination of the original network loss and the total penalty on the weights, AtomNAS is able to learn both the parameters of the network and the weights of the atomic blocks. At the end of the learning, atomic blocks with very small weights (e.g., < 0.001) which has fewer FL.OPs. Since the pruned atomic blocks have little contribution to the network output due to their negligible weights, the final network does not need be retrained or finetuned.
removed from the network and obtain the final network
Training on the large supernet is computationally demanding. We observe that for many pruned atomic blocks, their weights diminish at the early stage of learning and never "revive" throughout the rest of learning. We propose a dynamic network shrinkage technique which removes those atomic blocks on the fly and greatly reduces the run time of AtomNAS.
In our experiment, our method achieves 75.9% top-1 accuracy on ImageNet dataset around 360M FLOPs, which is 0.9% higher than state-of-the- art model (Stamoulis et al., 2019a). By further in- corporating additional modules, our method achieves 77. 77.6% top-1 accuracy. It outperforms MixNet by 0.6% using 363M FLOPs, which is new tfae-of-theeeeee under the mobile scenario.
In summary, the major contributions of our work are:
1. We design fine-grained search space which includes the exact number of channels and mixed operations (e.g., combination of different convolution kernels).
2. We propose NAS framework, AtomNAS. Within the framework, an efficient end-to-end NAS algorithm is proposed which can simultaneously search the network architecture and train the final model. No finetuning needed after the algorithm finishes.
3. With the proposed search space and AtomNAS, achieve sate-oofthe-ee performance on ImageNet dataset under mobile setting.
2 RELATED WORK
2.1 NEURAL ARCHITECTURE SEARCH
Recently, there is a growing interest in automated neural architecture design. Reinforce learning based NAS methods (Zoph & Le. 2017; Tan et al., 2019; Tan & Le, 2019b;a) are usually compu- tational intensive, thus hampering its usage with limited computational budget. To accelerate the search procedure, ENAS (Pham et al.. 2018) represents the search space using a directed acyelic graph and aims to search the optimal subgraph within the large supergraph. A training strategy of parameter sharing among subgraphs is proposed to significantly increase the searching efficiency. The similar idea of optimizing optimal subgraphs within a supergraph is also adopted by Liu et al. (2019a); Jin et al. (2019); Xu et al. (2020); Wu et al. (2019); Guo et al. (2019); Cai et al. (2019). Stamoulis et al. (2019a); Yu et al. (2020) further share the parameters of different paths within block using super-kerne representation. A prominent disadvantage of the above methods is that their coarse search spaces only support selecting one out of set of choices (e.g.. selecting one ker- nel size from (3, 5, 7)). MixNet tries to benefit from mixed operations by using a predefined set of mixed operations ((3). (3, 5), (3. 5, 7). (3. 5, 7, 9}), where the channels are equally distributed
2

---

Published as a conference paper at ICLR 2020
8D..]]
5
a/2:1
[.22
Figure Illustration the ensemble perspective. Arrow means operators. The structure of two convolutions joined by channel-wise operation is mathematically equivalent to the ensemble of multiple atomic blocks, according to Eq. (2), Colored rectangles represent tensors, with numbers inside indicating their channel numbers; The shaded path the right is one example of atomic block.
among different kernel sizes. Due to this limitation, it is difficult to learn optimal architectures under computational resource constraints. On the contrary, our method takes advantage of the fine-grained search space and is able to search for more flexible network architectures satisfying various resource constraints. The fine-grained search space proposed this paper is exponentially larger than pre- vious search space. For reference, the total number of possible structures within the experiment is around 10 162, compared with 1021 for FBNet. Recently, to improve the final performance of the searched architectures, Yu et al. (2020) utilizes knowledge distillation which is orthogonal method. could be easily integrated into our method by Eq. (5) thanks to the end-to-end learning paradigm of our method.
2.2 NETWORK PRUNING
Assuming that many parameters in the network are unnecessary, network pruning methods start from a computation-intensiee model, identify the unimportant connections and remove them get compact and efficient network. Early method (Han et al., 2016) simultaneously learns the important connections and weights. However, non-regularly removing connections in these works makes hard to achieve theoretical speedup ratio on realistic hardwares due to extra overhead in caching and indexing.
tackle this problem, structured network pruning methods (He et al., 201 Liu et al., 2017; Luo et al., 2017; Ye et al., 2018; Gordon et al., 2018) are proposed to prune structured components in networks, c.g. the entire channel and kernel. In this way, empirical acceleration can be achieved on modern computing devices. Liu et al. (2017); Ye et al. (2018); Gordon et al. (2018) encourage channel-level sparsity by imposing the L-1 regularizer on the channel dimension, which is also used by our method. Recently, Liu et al. (2019b) show that in structured network pruning, the learned weights are unimportant. This suggests structured network pruning is actually a neural architecture search focusing channel numbers. Our method jointly searches the channel numbers and mix of operations, which is a much larger search space.
3 ATOMNAS
We formulate our neural architecture search method in a fine-grained search space with the atomic block used as the basic search unit. An atomic block is comprised of two convolutions connected by a channel-wise operation. By stacking atomic blocks, we obtain larger building blocks (e.g. residual block and MobileNetV2 block proposed in variety of state-of-the-an models including ResNet, MobileNet V2/V3 (He et al., 2016: Howard et al., 2019; Sandler et al., 2018). In Section 3. 1, We first show larger network building blocks (e.g. MobileNetV2 block) can be represented by ensembles of atomic blocks. Based on this view, we propose fine- one-grained search space using atomic blocks. In Section 3.2, propose resource- atomic block selection method for end-to-end architecture search. Finally, we propose a dynamic network shrinkage technique in Section 3.3, which greatly reduces the search cost.

---

Published as a conference paper at ICLR 2020
3.1 GRAINGR SEARCH SPACE
Under the typical block-wise NAS paradigm (Tan et al., 2019; Tan & Le, 2019b), the search space of each block in a neural network represented as the Cartesian product C = II - P., where each P. is the set of all choices of the i-th configuration such as kernel size, number of channels and type of operation. For example, = {conv, depth-wise conv, dilated conv] x f3. 5] x {24, 32, 64, 128) represents search space of three types of convolutions by two kernel sizes and four options of channel number. A block the resulting model can only pick one convolution type from the three and one output channel number from the four values. This paradigm greatly limits the search space due to the few choices of each configuration. Here we present a more fine- fine-graing -raiind search space by decomposing the network into smaller and more basic building blocks.
We denote (X) as a convolution operator, where X is the input tensor and c' are the input and output channel numbers respectively. A wide range of manually-designe and NAS architectures share structure that joins two convolutions by channel-wise operation:
Y = (JT' ogo fo pe
(X)
where g is a channel-wise operator. For example, in VGG (Simonyan & Zisserman, 2015) and Residual Block (He et al., 2016), fo and fi are convolutions and g is one of Maxpool, ReLU and BN-ReLU: in MobileNetV2 block (Sandler et al., 2018), , fo and fi are point-wiss convolutions and is depth-wise convolution with BN-ReLU in the MobileNetV2 block. Eq. (1) can be reformulated as follows:
Y Σ " (.. (X),
(2)
where [ i] is the i-th convolution kernel of for gli,:| is the operator of the i-th channel of g, and {f [i, 1)<' are obtained by splitting the kernel tensor of fi along the the input channel dimension. Each term in the summation can be seen as a computationally independent block, which is called atomic block. Fig. (1) demonstrate this reformulation. By determining whether to keep each atomic block the final model individually, the search of channel number c' is enabled through channel selection, which greatly enlarges the search space.
This formulation also naturally includes the selection of operators. To gain better understanding, first generalize Eq. (2) as:
Y
S gs soii) (X).
Note the array indices are moved to subseripts. In this formulation, we use different types of operators for fos- fii and 9ú in other words, fo. fi and can each be a combination of different operators and each atomic block can use different operators such as convolutions with different kernel sizes.
Formally, the search space is formulated supernet which is built based on the structure in Eq. (1); such structure satisfies Eq. (3) and thus can be represented by atomic blocks; each of fo- f1 and is combination of operators. The new search space includes some state-of-the-art network architec- tures. For example, by allowing g to be a combination of convolutions with different kernel sizes, the MixConv block in MixNet (Tan Le, 2019b) becomes a special case in our search space. In addition, our search space facilitates discarding any number of channels in g, resulting in a more fine-grained channel configuration. In comparison, the channel numbers are determined heuristi- cally in Tan & Le (2019b).
3.2 RESOURCE- -WARR ATOMIC BLOCK SEARCH
In this work, we adopt a differentiable neural architecture search paradigm where the model structure is discovered in a full pass of model training. With the supernet defined above, the final model can be produced by discarding part of the atomic blocks during training. Following DARTS (Liu et al.
4

---

Published as a conference paper at ICLR 2020
(2019a)), we introduce importance factor scale the output of each atomic block in the supernet. Eq. (3) then becomes
Y = Σ as (Fi og fa:)) (X).
(4)
1=1
.2 Here, each a; is tied with an atomic block comprised of three operators f1: :9, and fIc. The importance factors are learned jointly with the network weights. Once the training finishes, the atomic blocks that have negligible effect (i.e., those with factors smaller than a threshold) on the network output are discarded.
We still need to address two issues related to the importance factors aç's The first issue where in the supernet we should put the a? Let's first consider the case when g only contains linear operations, e.g., convolution, batch normalization and linear activation like ReLU. If contains at least one BN layer. The scaling parameters in the BN layers can be directly used as such importance factors (Liu et al. (2017)). If has no BN layers, which is rare, we can place anywhere between So and f1; however, we need to apply regularization terms to the weights of Jo and fi (e.g., weight decays) in order to prevent weights in fo and fi from getting too large and canceling the effect of a. When y contains non-linear operations, e.g.. Swish activation and Sigmoid activation, we can only put behind f1-
The second issue is how to avoid performance deterioration after discarding some of the atomic blocks. For example, DARTS discards operations with small scale factors after iterative training of model parameters and scale factors. Since the scale factors of the discarded operations are not small enough, the performance of the network will be affected which needs re-training to adjust the weights again. In order to maintain the performance of the supernet after dropping some atomics blocks, the importance factors a of those atomic blocks should be sufficiently small. Inspired by the channel pruning work in Liu et al. (2017), we add L1 norm penalty loss on a, which effectively pushes many importance factors near-zero values. At the end of learning, atomic blocks with a close to zero are removed from the supernet. Note that since the BN scales change more dramatically during training due to the regularization term, the running statistics of BNs might be inaccurate and needs to be calculated again using the training set.
With the added regularization term, the training loss
== + Σ Glad,
(5)
iES
G = C/ Ōk
(6)
where is the coefficient of LI penalty term, S is the index set of all atomic blocks, and & is the conventional training loss (e.g., cross-entropy loss combined with the regularization term like weight decay and distillation loss.). Ja,| is weighted by coefficient e; which is proportional to the computation cost of i-th atomic block, i.e. é. By using computation costs aware regularization, we encourage the model to learn network structures that strike good balance between accuracy and efficiency. In this paper, we use FL.OPs as the criteria of computation cost. Other metrics such as latency and energy consumption can be used similarly. As a result, the whole loss function C trades off between accuracy and FLOPs.
3.3 DYNAMIC NETWORK SHRINKAGE
Usually, the supernet is much larger than the final search result. We observe that many atomic blocks become "dead" starting from the early stage of the search, i.e., their importance factors are close to zero till the end of the search. To utilize computational resources more efficiently and speed up the search process, we propose a dynamic network shrinkage algorithm which cuts down the network architecture by removing atomic blocks once they are deemed "dead".
We adopt a conservative strategy to decide whether an atomic block is "dead": for importance factors a, we maintain its momentum à which is updated as
à 3ā + (1 3)a²
5

---

Published as a conference paper at ICLR 2020
Amortized
Klmminated
1600
1400
1200
FLOOSSND)
1000
600
50 100
200
250
300
350
Epoch
Figure 2: FLOPs change of the supernet during the searching and training for AtomNAS-C. The crossed-out region corresponds to the saved computation compared to training the supernet without the dynamic shrinkage. The region in yellow corresponds to the extra cost compared with training the final model from scratch, the cost which is the region below the red dashed line.
Initialize the supernet and the exponential moving average;
while epoch I maxz_pocl do
Update network weights and importance factors a by minimizing the loss function L; Update the à by Eq. (7);
Total FLOPs of dead blocks 2 then
Remove dead blocks from the supernet; end
|
Recalculate BN's statistics by forwarding some training examples; Validate the performance of the current supernet;
end
Algorithm 1: Dynamic network shrinkage
where a' is the importance factors at 1-th iteration and / is the decay term. An atomic block is considered "dead" if both à and a² are smaller than threshold, which is set to 0.001 throughout experiments.
Once the total FLOPs of "dead" blocks reach predefined threshold, we remove those blocks from the supernet. As discussed above, recalculate BN's running statistics before deploying the net- work. The whole training process is presented in Algorithm
We show the FLOPs of sample network during the search process in Fig. 2. supernet with 1521M FLOPs and dynamically discard "dead" atomic blocks to reduce search cost, The overall search and train cost only increases by 17. 17.2% compared that of training the searched model from scratch.
We start from
EXPERIMENT
We first describe the implementation details in Section 4.1 and then compare AtomNAS with pre- vious state-of-the-art methods under various FLOPs constraints in Section 4.2. In Section 4.3, provide more detailed analysis about AtomNAS. Finally, in Section 4.4, we demonstrate the trans- ferability of AtomNAS networks by evaluating them detection and instance segmentation tasks.
4.1 IMPLEMENTATION DETAILS
The architecture of the supernet we use for the experiments is shown in table on the right of Fig. 3. The supernet contains 21 AtomNAS blocks, the searchable block in supernet; the picture on the right of Fig. 3 illustrates the structure of an AtomNAS block, where fo is a x pointwise convolutions that expands the input channel number from to 6C:; g is mix of three depth-wise

---

Published as a conference paper at ICLR 2020
Input Shape
Block
n
stride
224 3
3x3 conv
32(16)
1
2
112a 32(
3x3 MB
1
1x1 pointwise
112 16
searchable
24
4
56 x 24
searchable
40
4
2
3x3 5x5 div
7x7 dw
28 x
searchable
4
142 x 80
searchable
96
4
pointwise
Skip
142 x 96
searchable
192
4
2
7 192
searchable
320
I
7 320
avgpool
1
1280
1000
1
Figure 3: (Left) The searchable block of the supernet. fo and fi are fixed to convolutions; here is mix of three convolutions with kemnel sizes of 3 x 3, 5 x 5 and 7 x 7. fo expands the input channel number from C 18C and fi projects the channel number to the output channel number. If the output dimension stays the same the input dimension, we use skip connection to add the input to the output. (Right) Architecture the supernet. Column-Block denotes the block type; MB denotes MobileNetV2 block; "scarchable" means searchable block shown on the left. Column-f denotes the output channel number of a block. Column-n denotes the number of blocks. Column-s denotes the stride of the first block in stage. The output channel numbers of the first convolution are 16 for AtomNAS-A, 32 for AtomNAS-B and AtomNAS-C.
pointwise
convolutions with kernel sizes of 3 x 3, 5 and 7x and fi is another 1 x pointwise convolutions that projects the channel number to the output channel number. Similar to MobileNetV2 (Sandler et al., 2018), if the output dimension stays the same the input dimension, to add the input to the output. AtomNAS block is effectively an ensemble of 3 x 6C atomic blocks, whose underlying search space covers the MobileNetV2 block (Sandler et al., 2018) and its multi- kernel variant, MixConv (Tan & Le, 2019b). Within AtomNAS block, we are able to optimize the distribution of computation resources (i.e., channel numbers) among the three types of depth-wise convolution.
use skip connection
We the same training configuration (e.g., RMSProp optimizer, EMA on weights and exponential learning rate decay) as Tan et al. (2019); Stamoulis et al. (2019a) and do not use extra data aug- mentation such as MixUp (Zhang al., 2018) and AutoAugment (Cubuk et al., 2018). We find that using this configuration is sufficient for our method to achieve good performance. Our results are shown in Table and Table When training the supernet, use total batch size of 2048 on 32 Tesla V100 GPUs and train for 350 epochs. For our dynamic network shrinkage algorithm, we set the momentum factor / in Eq. (7) to 0.9999. randomly initialized. To avoid removing atomic blocks with high penalties (i.e., FLOPs) pre- maturely, the weight of the penalty term in Eq. (5) is increased from to the target by a linear scheduler during the first 25 epochs. By setting the weight of the LI penalty term À to be 1.8 10 1.2 x 10 and 1.0 x 10 4 respectively, we obtain networks with three different sizes: AtomNAS-A, AtomNAS-B B, and AtomNAS-C. They have the similar FLOPs previous stute-of-the- -h- networks under 400M: MixNet-S (Tan & Le, 2019b), MixNet-M (Tan & Le, 2019b) and SinglePath (Sta- moulis et al., 2019a). In Appendix A, visualize the architecture of AtomNAS-C.
the beginning of the training, all of the weights
4.2 EXPERIMENTS ON IMAGENET
We apply AtomNAS search high performance light-weight model on ImageNet 2012 classifica- tion task (Deng et al., 2009). Table compares our methods with previous fate-of-the-an models, either manually designed or searched.
With models directly produced by AtomNAS, our method achieves the new state-of-the-ar under all FLOPs constraints. Especially, AtomNAS-C achieves 75.9% top-1 accuracy with only 360M FLOPs, and surpasses all other models, including models like PDARTS and DenseNAS which have much higher FLOPs.

---

Published as a conference paper at ICLR 2020
AtomNAS
AtomNAS++
Imagenet Top-1 Accuracy
MobileNetV2
ShuffleNetV2
76
FBNet
Proxyless
75
DARTS
SinglePath
MnasNett
MixNer!
EfficientNet!
72
200
300
400
500
600
FLOPs(M)
Figure 4: FLOPs versus accuracy on ImageNet. means methods use extra techniques like Swish activation and suueezzeead-Exciiiee module.
Techniques like Swish activation function (Ramachandran al., 2018) and suueeze-anddEExiittte (SE) module (Hu et al., 2018) consistently improve the accuracy with marginal FLOPs cost. For a directly modify the searched network by
fair comparison with methods that use these techniques, replacing all ReLU activation with Swish and add SE module with ratio 0.5 to every block and then retrain the network from scratch. Note that unlike other methods, we do not search the configuration of Swish and SE, and therefore the performance might not be optimal. Extra data augmentations such MixUp and Autonugment are still used. We train the models from scratch with a total batch size of 4096 on 32 Tesla V100 GPUs for 250 epochs.
Simply adding these techniques improves the results further. AtomNAS-A+ achieves 76.3% top- accuracy with 260M FLOPs, which outperforms many heavier models including MnasNet-A2. Without extra data augmentations, it performs as well as Efficient-BB (Tan & Le, 2019a) by using 130M less FLOPs. It also outperforms the previous state-of-the-arr MixNet-S by 0.5%. In addition, AtomNAS-C+ + improves the top-1 accuracy on ImageNet to 77.6%, surpassing previous state-of- the-art MixNet-M by 0.6% and becomes the overall best performing model under 400M FLOPs.
Fig. 4 visualizes the top-I accuracy on ImageNet for different models. It's clear that our fine-grained search space and the end-to-end resource-aware search method boost the performance significantly.
4.3 ANALYSIS
4.3.1
RESOURCE-AWARE REGULARIZATION
To demonstrate the effectiveness of the resource-aware regularization in Section 3.2, compare with a baseline without FLOPP--relatd coefficients e, which is widely used in network pruning (Liu et al., 2017: He et al., 2017b). Table 2 shows the results. First, by using the same LI penalty coefficient À = 1.0 x 10 the baseline achieves a network with similar performance but using much more FLOPs; then by increasing
1.5 x 10-4 the baseline obtain a network which has similar FLOPs but inferior performance (i.e., about 1.0% lower). In Fig. 6b we visualized the ratio of different types of atomic blocks of the baseline network obtained by 1.5x 10-4, The baseline network keeps more atomic blocks in the earlier blocks, which have higher computation cost due to higher input resolution. On the contrary, AtomNAS is aware of the resource constraint, thus keeping more atomic blocks in the later blocks and achieving much better performance.
4.3.2 BN RECALIBRATION
As the BN's running statistics might be inaccurate as explained in Section 3.2 and Section 3.3, re-calculate the rumning statistics of BN before inference, by forwarding 131k randomly sampled training images through the network. Table shows the impact of the BN recalibration. The top-1 accuracies of AtomNAS-A, AtomNAS-B, and AtomNAS-CC on ImageNet improve by 1.4%, 1.7%, and 1.2% respectively, which clearly shows the benefit of BN recalibration.
8

---

Published as a conference paper at ICLR 2020
Table I: Comparision with state-of-the-arts on ImageNet under the mobile setting. denotes meth- ods using extra network modules such Swish activation and Squeezz-andd-Exee module. f de- notes using extra data augmentation such as MixUp and AutoAugment. denotes models searched and trained simultaneously.
Model
Parameters
FLOPs
Top-1(%)
Top-5(%)
MobileNetV1 (Howard et al., 2017)
4.2M
575M
70.6
89.5
MobileNetV2 (Sandler al., 2018)
3.4M
300M
72.0
91.0
MobileNetV2 (our impl.)
3.4M
301M
73.6
91.5
MobileNetV2 (1.4)
6.9M
585M
74.7
92.5
ShuffleNetV2 (Ma et al., 2018)
3.5M
299M
72.6
ShuffleNetV2 2x
7.4M
591M
74.9
FBNet-A (Wu et al., 2019)
4.3M
249M
73.0
FBNet-C
5.5M
375M
74.9
Proxyless (mobile) (Cai et al., 2019)
4.1M
320M
74.6
92.2
SinglePath (Stamoulis al., 2019a)
4.4M
334M
75.0
92.2
NASNet-A (Zoph & Le, 2017)
5.3M
564M
74.0
91.6
DARTS (second order) (Liu et al., 2019a)
4.9M
595M
73.1
PDARTS (cifar 10) (Chen et al., 2019b)
4.9M
557M
75.6
92.6
DenseNAS-A (Fang et al., 2019)
7.9M
501M
75.9
92.6
FairNAS-A (Chu et al., 2019b)
4.6M
388M
75.3
92.4
AtomNAS-A
3,9M
258M
74.6
92.1
AtomNAS-B"
4.4M
326M
75.5
92.6
AtomNAS-C*
4.7M
360M
75.9
92.7
SCARLET-AF (Chu et al., 2019a)
6.7M
365M
76.9
93.4
MnasNet-A1 (Tan et al., 2019)
3.9M
312M
75.2
92.5
MnasNet-A2f
4.8M
340M
75.6
92.7
MixNet-St (Tan Le, 2019b)
4.1M
256M
75.8
92.8
MixNet-Mi
5.0M
360M
77.0
93.3
EfficientNet-BOit (Tan & Le, 2019a)
5.3M
390M
76.3
93.2
SE-DARTS+ (Liang et al., 2019)
6.1M
594M
77.5
93.6
AtomNAS-A++
4.7M
260M
76.3
93.0
AtomNAS-B+1
5.5M
329M
77.2
93.5
AtomNAS-C4f
5.9M
363M
77.6
93.6
Table 2: Influence of awareness of resource metric. The upper block uses equal penalties for all atomic blocks. The lower part uses our resource-aware atomic block selection.
FLOPs
Top-
1.0 x 10-4
445M
76.1
1.5 10-4
370M
74.9
1.0 x 10-4
360M
75.9
4.3.3 COST OF DYNAMIC NETWORK SHRINKAGE
Our dynamic network shrinkage algorithm speedups the search and train process significantly. For AtomNAS-C, the total time for search-and-training is 25.5 hours. For reference, training the final architecture from scratch takes 22 hours. Note that as the supernet shrinks, both the GPU memory consumption and forward-backwant time significantly reduced. Thus it's possible to dynamically change the batch size once having sufficient GPU memory, which would further speed up the whole procedure.

---

Published as a conference paper at ICLR 2020
Table Influence of BN recalibration.
Model
w/o Recalibration
w/ Recalibration
AtomNAS-A
73.2
74.6 (+1.4)
AtomNAS-B
73.8
75.5 (+1.7)
AtomNAS-C
74.7
75.9 (+1.2)
4.4 EXPERIMENTS ON COCO DETECTION AND INSTANCE SEGMENTATION
In this section, we assess the performance of AtomNAS models as feature extractors for object detection and instance segmentation on COCO dataset (Lin al., 2014). We first pretrain AtomNAS models (without Swish activation function (Ramachandran et al., 2018) and Squeeze-and-Exciiie (SE) module (Hu al., 2018)) on ImageNet, use them as drop-in replacements for the backbone in the Mask-RCNN model (He et al., 2017a) by building the detection head on top of the last feature map, and finetune the model on COCO dataset.
We use the open-source code MMDetection (Chen et al., 2019a). All the models are trained on COCO train2017 with batch size 16 and evaluated on COCO val2017. Following the schedule used in the open-source implementation of TPU-trained Mask-RCNN the learning rate starts at 0.02 and decreases by a scale 10 at 15-th and 20th epoch respectively. The models are trained for 23 epochs in total.
Table 4 compares the results with other baseline backbone models. The detection results of baseline models are from Stamoulis al. (2019b). We can see that all three AtomNAS models outperform the baselines on object detection task. The results demonstrate that our models have better transferability than the baselines, which may due mixed operations, a.k.a multi-scale here, to object detection and instance segmentation.
important
Table 4: Comparision with baseline backbones on COCO object detection and instance segmenta- tion. Cls denotes the ImageNet top-1 accuracy; detect-mAP and seg-mAP denotes mean average precision for detection and instance segmentation on COCO dataset. The results of baseline models from Stamoulis et al. (2019b). SinglePath+ (Stamoulis al., 2019b) contains SE module.
Model
FLOPs
Cls (%)
detect-mAP (%)
seg-mAP (%)
MobileNetV2 (Sandler et al., 2018)
301M
73.6
30.5
Proxyless (mobile) (Cai al., 2019)
320M
74.6
32.9
Proxyless (mobile) (our impl.)
320M
74.9
32.7
30.0
SinglePath+ (Stamoulis et al., 2019b)
353M
75.6
33.0
SinglePath (our impl.)
334M
75.0
32.0
29.7
AtomAAS-A
258M
74.6
32.7
30.1
AtomNAS-B
326M
75.5
33.6
30.8
AtomNAS-C
360M
75.9
34.1
31.4
5 CONCLUSION
In this paper, we revisit the common structure, i.c., two convolutions joined by a channel-wise op- eration, and reformulate it ensemble of atomic blocks. This perspective enables much larger and more fine-graine ame-ggrined search space. For efficiently exploring the huge fine-grained search space, propose an end-to-end framework named AtomNAS, which conducts architecture search and network training jointly. The searched networks achieve significantly better accuracy than previous state-of-the-arr methods while using small extra cost.
hittps:/9g thub. http:////aaa...ceooooooooooooe/e ronn
/
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. ICLR, 2019.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng. Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint 0ttt::/900000700
2019a.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. CoRR, abs/1904. 2019b.
Xiangxiang Chu, Bo Zhang, Jixiang Li, Qingyuan Li, and Ruijun Xu. Scarletnas: Bridging the gap between scalability and fairness in neural architecture search. CoRR, abs/1908.000 000002 2019a.
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fairnas: Rethinking evaluation faimess of weight sharing neural architecture search. CoRR, abs/1907.01845, 2019b.
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc Le Autoaugment: Learning augmentation policies from data. CoRR, abs/ $85.09501. 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248 248-255, 2009.
Jiemin Fang, Yuzhu Sun, Qian Zhang, Yuan Li, Wenyu Liu, and Xinggang Wang. Densely connected search space for more flexible neural architecture search. CoRR, abs/1906.099m 996609607, 2019.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. Mor- phnet: Fast & simple resource-constrained structure learning of deep networks. In CVPR, pp. 1586-1595, 2018.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. CoRR, his/1994.00420) 2019.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In ICLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshiek. Mask R-CNN. In IEEE ternational Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp- 2980-2988, 2017a
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In ICCV, pp. 1498-1406, 2017b.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chiel Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. Corr, hbs/1905.022444 2019.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861, 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and--eciiation networks. In CVPR, pp. 7132-7141, 2018.
Xiaojie Jin, Jiang Wang, Joshua Slocum, Ming- Hsuan Yang, Shengyang Dai. Shuicheng Yan, and Jiashi Feng. Re-darts: Resource constrained differentiable architecture search. arXiv preprint arXiv: 12..11212..1..4. 2019.
11

---

Published as a conference paper at ICLR 2020
Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping, 2019.
Tsung-Yi Lin, Michael Maire, Serge J. Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Computer Vision ECCV 2014 13th European Conference. Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V. pp. 740-755, 2014.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In ICLR, 2019a.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In ICCV, pp. 2755-2763, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019b.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In ICCV, pp- 5068-5076, 2017.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines for efficient CNN architecture design. In ECCV, pp. 122-138, 2018.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, pp. 4092-4101, 2018.
Prajit Ramachandran, Barret Zoph, and Quoe V. Le. Searching for activation functions. In 6th In- ternational Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 30-May 3, 2018, Workshop Track Proceedings, 2018.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pp. 4510-4520 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. ICLR, 2015.
Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path NAS: designing hardware-efficient convnets in less than hours. CoRR, abs/1904.02877, 2019a.
Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path mobile automl: Efficient convnet design and NAS hyperpa- rameter optimization. CoRR, abs/1907 .0959, 2019b.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, pp. 6105-6114, 2019a
Mingxing Tan and Quoe V. Le. Mixconv: Mixed depthwise convolutional kernels. hsss/190709995... 2019b.
CoRR,
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In CVPR, pp. 2820- 2828, 2019.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR, pp. 10734-10742, 2019.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. PC- DARTS: Partial channel connections for memory-efficient architecture search. In ICLR, 2020.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the smaller-nomee norm norm-less--eeee www.seeeffoomatiiii assumption in channel pruning of convolution layers. In ICLR, 2018
12

---

Published as a conference paper at ICLR 2020
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, and Quoc Le. Scaling up neural architecture search with big single-stage models, 2020. URL
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In ICLR, 2018.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
A
VISUALIZATION
Figure 5: The architecture of AtomNAS-C. Blue, orange, cyan blocks denote atomic blocks with kernel size 3, 5 and respectively; the heights of these blocks are proportional to their expand ratios.
We plot the structure of the searched architecture AtomNAS-C in Fig. 5, from which we see more flexibility of channel number selection, not only among different operators within each block, but also across the network. In Fig. 6a, visualize the ratio between atomic blocks with different kernel sizes in all search blocks. First, we notice that all search blocks have convolutions of all three kernel sizes, showing that AtomNAS learns the importance of using multiple kernel sizes in network architecture. Another observation is that AtomNAS tends to keep more atomic blocks at the later stage of the network. This is because in earlier stage, convolutions of the same kernel size costs more FLOPs; AtomNAS is aware of this (thanks to its resource-aware regularization) and try keep as less as possible computationally costly atomic blocks.
13

---

Published as a conference paper at ICLR 2020
288
432
432
432
432
720
720
720
720
1440
1440
1440
1440
1728
1728
1728
1728
3456
3456
3456
3456
(a) Atom
288
432
432
432
432
720
720
720
720
1440
1440
1440
1440
1728
1728
1728
1728
3456
3456
3456
3456
(b) Baseline
Figure 6: Ratio of different types of atomic blocks in all 21 searchable blocks. The text above each pie tells the total number of atomic blocks of the corresponding block in the original supernet. Grey denotes dead atomic blocks; blue, orange, and cyan represent atomic blocks using depth- wise convolutions with kernel size 3, 5, 7 respectively. Blocks without skip connection are highlighted by bold text. (a) Visualization for AtomNAS-C. (b) Visualization for baseline (i.e., without FL.OPs related coefficients ct).
14


Reviews:
[{"review_id": "SketHivcFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose AtomNAS, a neural architecture search (NAS) algorithm, with a new fine-grained search space and the dynamic network shrinkage method. The searched models achieve a new state-of-the-art result on the ImageNet classification task for mobile setting with restricted FLOPs. \n\n- The proposed method is novel and technically sound. In addition, the experimental results on ImageNet are impressive. However, the experiment section is not solid enough.\n\n- The authors do not include the searching cost and inference latency in Table 1. Different NAS papers have different objectives and searching cost. For instance, ProxylessNAS (Cai et al., 2019) and DenseNAS (Fang et al., 2019) focus on searching cost. They require only 200 and 92 GPU hours (with TITAN XP). However, the proposed AtomNAS takes 32 * 25.5 = 816 GPU hours (with V100). The authors only point out that DenseNAS uses more parameters.  It would be better if the authors can make the comparison more transparent.   \n\n- I wonder when given the same searching budgets as ProxylessNAS and DenseNAS, how well AtomNAS can perform.\n\n- The authors use only one dataset: ImageNet. I would like to see results on some other datasets or tasks. For instance, the authors may apply AtomNAS to other image classification datasets or finetuning the pre-trained models on object detection or semantic segmentation tasks.\n\n- In general, the paper is well written and easy to follow. I would encourage the authors to add legends to Figure 5 and Figure 6. While the meaning of each color is explained in the caption, it is not straight forward.\n\nIn short, the proposed method is interesting and the results on ImageNet are impressive, I weakly accept this paper and hope that the authors can make the experiment section more solid in a revised version.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1xUnP03KS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary] This paper proposes a channel-wise neural architecture search (NAS) approach. The NAS search algorithm is similar to previous one-shot NAS, but the search space is channel-wise: each channel has it’s own kernel size, which is quite novel and interesting. Results are strong in terms of FLOPS and parameters.\n\n[High-level comments]:\n\n1. I like the novel idea of channel-wise search space, which provides great flexibility for NAS.  Although some recent works (e.g., MixNet) have tried to partition channels into groups, this paper goes further and searches for different kernel size for each single channel.  With this channel-wise search space, it naturally enables a combination of per-channel kernel size selection and per-channel pruning, leading to strong results in terms of FLOPS and parameters, as shown in Figure 4 and Table 1.\n\n2. In general, channel-wise NAS is difficult as different channels are often coupled in various ways. However, the authors observe that recent NAS (such as MnasNet/MixNet/SCARLET-A/EfficientNet) are mostly based on a common MB pattern. By targeting to this specific pattern and applying some additional constraints (e.g. fixed input/output channel size and fixed number of layers per stage as shown in Figure 3), the authors successfully make the channel-wise NAS work well.  I appreciate the authors efforts, but I am also a little concerned that the proposed approach might be limited to this specific small search space. \n\n[Questions and suggestions to authors]:\n\n3.  How do you justify the generality of the channel-wise search space?  For example, is it possible to also search for input/output channel size (column f in Figure 3) and #layers per stage (column n in Figure 3)? Adding some discussions for this would be very helpful.\n\n4. The title seems too broad. I recommend the authors including “channel-wise” in the title.\n\n5. Please provide some justifications on how to set λ  and c_i in Equation (5).\n\n6. When you say “expands the input channel number from C to 3 × 6C”, what does “3x6C” mean? Is it 18C? What’s the reason for choosing this specific value?\n\n7. Could you show the accuracy and complexity (either FLOPS or params) of the supernet during the training? This information would be helpful to interpret and justify your algorithm 1.\n\n8. The network architecture in Figure 5 is vague. Could you provide the network source code or frozen graph for this specific model?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xOjiLpKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This basic idea of this paper is to decompose the common building blocks of large network into atomic blocks, which equips NAS with more fine-grained search space. What's more, the authors propose a resource-aware search to reduce the computation and dynamically shrinkage the model to accelerate the learning. Retraining the final network is no longer needed. They achieve state of art on ImageNet under several complexity constraints.\n\nPros:\nNovel idea: the insight of this paper is that \"larger network building blocks can be represented by an ensemble of atomic blocks\". With this in hand, it can search the exact channel number through channel selection (i.e. atomic block selection, according to my understanding).\n\nEfficiency: Resource-aware selection and dynamical shrinkage of the model also make it more efficient in inference and training. \n\nCons:\nIt would be better if the author could provide some comparison on GPU time. Since FLOPs is only an indirect metric for speed evaluation. \n\nThe biggest problem of this paper is that experiment is not enough. It would be more convincing if experiments on other popular datasets (CIFAR10/100 etc.) or tasks (object detection, semantic segmentation, etc.) are implemented.\n\nConclusion:\nThis is an interesting paper with novel idea and efficient implementation. However, more experiments are needed to validate the utility of the proposed method. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewer #1 noted that he wishes to change his review to weak accept post rebuttal, but did not change his score in the system.  Presuming his score is weak accept, then all reviewers are unanimous for acceptance.  I have reviewed the paper and find the results appear to be clear, but the magnitude of the improvement is modest.  I concur with the weak accept recommendation. 
Decision: Accept (Poster)