Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new method for neural architecture search that searches architectures with not only feature transformational components but also self-calibration and dynamic convolution ones. Although the idea seems to be straightforward and innovating, it is difficult to assess the effectiveness of the proposed approach comparing to other methods just by looking at the experimental results. 

Looking at the CIFAR10 results, the authors claimed that their method achieved second best among all the method compared but with significantly less number of parameters. Although indeed the number of parameters differ by a lot, the error rates are also differed significantly. It might be worth comparing these two baselines using the same number of parameters just to conduct a fair comparison.

In the Imagenet results, the proposed method is behind many of the state-of-the-art methods, casting concerns on the effectiveness of the proposed approach considering its added search space. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a complicated NAS approach, with a lot more ops, a larger search space and an extra teacher network. However, the results on CIFAR10 and ImageNet are both not competitive to other SOTA results.

My major concern is whether the extra complexity introduced by this paper is necessary:

1. The new search space includes a lot more ops and optimizations, which by themselves should improve other models. For example, according to Table1, dynamic conv and attention improve ResNet-18 without any architecture search. What if you simply apply the same optimizations to other SOTA models in Table 2 and 3?
2. The extra ops already make the comparison in Table 2/3 unfair. Despite that, NOS is still not competitive to other SOTA results (prroxylessNAS on CIFAR-10 and MnasNet-A3 on ImageNet).
3. The authors argue that “NOS shows significant compactness advantages than proxylessNAS”, but it is somewhat misleading. By reading this paper, it seems the search algorithm used in this paper aims to find the highest accuracy model WITHOUT resource constraints, so smaller model size should not be related to the search algorithm.

Here are a few other comments and suggestions:

4. Section 3.2 is difficult to follow. I recommend the authors providing some visualization for the search space (see NASNet paper for example).
5. Ablation study is relatively weak: for example, Figure 5 compares w/ and w/o attention-guided search, but it is unclear whether the gain is by the search or by the teacher distillation. A more fair comparison is to w/o attention-guided search but also perform the distillation. It would be also helpful to verify whether the propose attention-guided search can work for existing  search space (such as the NASNet/DARTS search space).
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary:

Often in (neural architecture search) NAS papers on vision datasets, only feature transform operations (e.g. convolution, pooling, etc) are considered while operations like attention and dynamic convolution are left out. These operations have shown increasing performance improvements across many domains and datasets. This paper attempts to incorporate these operations into the search space of NAS.

Specifically they design a new cell (residual block in a resnet backbone) which contains these operations (traditionally used feature transforms, attention, and dynamic convolutions). This new cell contains two-tiers. The first tier separately computes the three kinds of operations and combines the results via elementwise summation to form an 'intermediate calibrated tensor'. This is then fed to the second tier where again the three kinds of operations are performed on them and they net output is again formed via elementwise summation. 

For the search algorithm the authors use the bilevel optimization of DARTS. The one difference to aid in the search is that an 'attention-guided' transfer mechanism is used where a teacher network trained on the larger dataset (like ImageNet) is used to align the intermediate activations of the proxy network found during search, by adding a alignment loss function. 

Results of search with this new search space on cifar10 and transfer to cifar100, imagenet are presented. It is found via manual ablation in resnet backbone that dilated dynamic convolutions dont help and are dropped from the search space. 

The numerical results are near state-of-the-art although as is pervasive in the NAS field actual fair comparisons between methods are hard to get due to differences in search space, hardware space, stochasticity in training etc. But the authors do a best-attempt and have included random search and best and average performances so that is good. 

Comments:

- The paper is generally well-written and easy to understand (thanks!)

- Experiments are generally thorough. 

- The main novelty is the design of the cell such that attention and dynamic convolution operations can be incorporated. I was hoping that the results in terms of error would be much better due to the more expressive search space but they are not at the moment.

- I am curious how the attention guided alignment loss has been ablated. How do the numbers look without alignment loss keeping gpu training time constant? Basically I am trying to figure out how much is the contribution of the new search space vs. the addition of attention guided search. Do we have a negative result (an important one though) that attention and dynamic convolutions don't really help?

review_title: Official Blind Review #1

Meta-review: This paper proposes an extension of the search space of neural architecture search to include dynamic convolutions, teacher nets among others. The method is evaluated on CIFAR-10 and Imagenet with a similar setup as other architecture search methods. The reviewers found that the results did not convincingly show that the proposed improvements were better than other ways of improving neural architecture search such as rroxylessNAS.
Decision: Reject