Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This basic idea of this paper is to decompose the common building blocks of large network into atomic blocks, which equips NAS with more fine-grained search space. What's more, the authors propose a resource-aware search to reduce the computation and dynamically shrinkage the model to accelerate the learning. Retraining the final network is no longer needed. They achieve state of art on ImageNet under several complexity constraints.

Pros:
Novel idea: the insight of this paper is that "larger network building blocks can be represented by an ensemble of atomic blocks". With this in hand, it can search the exact channel number through channel selection (i.e. atomic block selection, according to my understanding).

Efficiency: Resource-aware selection and dynamical shrinkage of the model also make it more efficient in inference and training. 

Cons:
It would be better if the author could provide some comparison on GPU time. Since FLOPs is only an indirect metric for speed evaluation. 

The biggest problem of this paper is that experiment is not enough. It would be more convincing if experiments on other popular datasets (CIFAR10/100 etc.) or tasks (object detection, semantic segmentation, etc.) are implemented.

Conclusion:
This is an interesting paper with novel idea and efficient implementation. However, more experiments are needed to validate the utility of the proposed method. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: [Summary] This paper proposes a channel-wise neural architecture search (NAS) approach. The NAS search algorithm is similar to previous one-shot NAS, but the search space is channel-wise: each channel has it’s own kernel size, which is quite novel and interesting. Results are strong in terms of FLOPS and parameters.

[High-level comments]:

1. I like the novel idea of channel-wise search space, which provides great flexibility for NAS.  Although some recent works (e.g., MixNet) have tried to partition channels into groups, this paper goes further and searches for different kernel size for each single channel.  With this channel-wise search space, it naturally enables a combination of per-channel kernel size selection and per-channel pruning, leading to strong results in terms of FLOPS and parameters, as shown in Figure 4 and Table 1.

2. In general, channel-wise NAS is difficult as different channels are often coupled in various ways. However, the authors observe that recent NAS (such as MnasNet/MixNet/SCARLET-A/EfficientNet) are mostly based on a common MB pattern. By targeting to this specific pattern and applying some additional constraints (e.g. fixed input/output channel size and fixed number of layers per stage as shown in Figure 3), the authors successfully make the channel-wise NAS work well.  I appreciate the authors efforts, but I am also a little concerned that the proposed approach might be limited to this specific small search space. 

[Questions and suggestions to authors]:

3.  How do you justify the generality of the channel-wise search space?  For example, is it possible to also search for input/output channel size (column f in Figure 3) and #layers per stage (column n in Figure 3)? Adding some discussions for this would be very helpful.

4. The title seems too broad. I recommend the authors including “channel-wise” in the title.

5. Please provide some justifications on how to set λ  and c_i in Equation (5).

6. When you say “expands the input channel number from C to 3 × 6C”, what does “3x6C” mean? Is it 18C? What’s the reason for choosing this specific value?

7. Could you show the accuracy and complexity (either FLOPS or params) of the supernet during the training? This information would be helpful to interpret and justify your algorithm 1.

8. The network architecture in Figure 5 is vague. Could you provide the network source code or frozen graph for this specific model?

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The authors propose AtomNAS, a neural architecture search (NAS) algorithm, with a new fine-grained search space and the dynamic network shrinkage method. The searched models achieve a new state-of-the-art result on the ImageNet classification task for mobile setting with restricted FLOPs. 

- The proposed method is novel and technically sound. In addition, the experimental results on ImageNet are impressive. However, the experiment section is not solid enough.

- The authors do not include the searching cost and inference latency in Table 1. Different NAS papers have different objectives and searching cost. For instance, ProxylessNAS (Cai et al., 2019) and DenseNAS (Fang et al., 2019) focus on searching cost. They require only 200 and 92 GPU hours (with TITAN XP). However, the proposed AtomNAS takes 32 * 25.5 = 816 GPU hours (with V100). The authors only point out that DenseNAS uses more parameters.  It would be better if the authors can make the comparison more transparent.   

- I wonder when given the same searching budgets as ProxylessNAS and DenseNAS, how well AtomNAS can perform.

- The authors use only one dataset: ImageNet. I would like to see results on some other datasets or tasks. For instance, the authors may apply AtomNAS to other image classification datasets or finetuning the pre-trained models on object detection or semantic segmentation tasks.

- In general, the paper is well written and easy to follow. I would encourage the authors to add legends to Figure 5 and Figure 6. While the meaning of each color is explained in the caption, it is not straight forward.

In short, the proposed method is interesting and the results on ImageNet are impressive, I weakly accept this paper and hope that the authors can make the experiment section more solid in a revised version.

review_title: Official Blind Review #3

Meta-review: Reviewer #1 noted that he wishes to change his review to weak accept post rebuttal, but did not change his score in the system.  Presuming his score is weak accept, then all reviewers are unanimous for acceptance.  I have reviewed the paper and find the results appear to be clear, but the magnitude of the improvement is modest.  I concur with the weak accept recommendation. 
Decision: Accept (Poster)