Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a method for architecture search in deep neural networks in order to identify scaled-down networks that can operate on resource limited hardware. The approach taken in this paper is different from other approaches, which train a single big model then fine tune smaller models for specific hardware, or use distillation to train progressively smaller models. Here, a single model is trained in a manner that allows subsequent slicing to smaller models without additional training required. The authors employ a variety of strategies to get this to work well, including specific initialization techniques, regularization methods, learning schedules, and a coarse-to-fine optimization method to obtain the smaller models. The authors demonstrate SotA performance on ImageNet relative to other techniques.

Overall, this was a well-written paper, and the results appear convincing. I would have liked a little bit more explanation about the implementation details though. As someone knowledgeable about deep ANNs, but not an expert in NAS for efficiency, I was not very clear on a couple of items. Specifically, I think it would be good to clarify the following:

1) The authors say that they use a masking strategy to implement slicing during training. So, do I take it that they effectively do a separate pass through the network with different masks to implement the different sized models? If so, do you then simply accumulate gradients across all masks to do the updates?

2) When the authors say they perform a fine grained grid search by varying the configurations, what is meant by this exactly? Do you mean that you do a search through different slicings of the big model to discover the best smaller architecture? What does it mean to do a grid search through binary masks? Or, is there a continuous hyperparameter that determines the masking which you do the grid search on? Maybe I’m just not understanding well at all, but even still, in that case, it indicates that this could be clarified in the manuscript.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors propose a search for neural architectures with different resource requirements by training a single model only. Furthermore, models found at the end of the search require no additional post-processing and are ready for deployment. A weight-sharing technique is applied to make this happen. The authors argue that there are multiple adaptions required to make it work. This includes the child model sampling strategy, use of model distillation, weight initialization, learning rate schedule, regularization and batch norm calibration.

The work seems to be an incremental extension of Yu & Huang (2019b) and phrased as a NAS algorithm. Many techniques considered vital for the proposed method make use of techniques proposed by Yu & Huang (2019b) (sandwich rule, inplace distillation, batch norm calibration and the way how weights are shared). Other required techniques are either proposed by others (initialization) or very simple (learning rate schedule and regularization). The authors claim that they extend the work by Yu &Huang (2019b) "to handle a much larger space where the architectural dimensions, including kernel sizes, channel numbers, input resolutions, network depths are jointly search". They never clarify why this is a non-trivial step and they might want to point this out in their rebuttal.

Besides this, the authors did a very good job. The paper is well-written, references are given wherever needed and all the closest related work is covered sufficiently. The experimental part conducts several ablation studies which supports their various decisions. Unfortunately, all results reported use heavy data augmentation which makes a comparison to other methods (besides EfficientNet) impossible. 600M is considered the upper bound for mobile architectures by the NAS community. Unfortunately, no such model is considered making it even harder to compare to existing NAS methods. The EfficientNet numbers reported don't match the ones reported in the original paper as far as I see. A red dot in Figure 3 could be added for the BigNASModel with 660M parameters.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Parameter sharing (PS) is an important approach to speed up neural network search (NAS), which further allows the development of differential architecture search (e.g., DARTS) methods. However, PS also deteriorates the performance of learning models on the validation/testing set. 

This paper first changes the search space from a DAG (micro+marco) in e.g., DARTS to a stacked one based on MBConv; and then, propose to use several tricks to train the super-net well. Finally, a search method is constructed for the supernet to find the desired architectures.

Overall, the paper is too experimental. The method is an ensemble of existing approaches, i.e., every single component in the paper has been visited in the literature. Expect for experimental results, I do not see many general lessons we can learn from the paper. Finally, why the proposed method can be better than others is not well-explained and clarified. 

Please see the questions below:

Q1. Is NAS a method only for ImageNet? Can the method generalize to more applications/datasets?
- While ImageNet is a good dataset for CV experiments, I think NAS should be a method for deriving architectures with certain requirements.
- So, with so many tricks proposed in the paper, I wish authors can carry on experiments on other data sets as well, e.g., CIFAR and MNIST, which can still be preferred.

Q2. On motivation, could authors explain more about the difficulties of combining all these techniques? 
- Each method is brought from some other paper, what motivate authors to combine them together? What makes them believe this is possible?

Q3. On presentation, could authors draw a figure of the search space in the main text and give an overall algorithm for Section "3.2 COARSE-TO-FINE ARCHITECTURE SELECTION". It is hard for a reader to see novelties there.

Q4. "We also use the swish activation (Ramachandran et al., 2017) and fixed AutoAugment V0 policy" - are all other compared methods using swish activation and AutoAugment V0 policy?

Q5. How about the search efficiency of the proposed method? Only the accuracy is reported in the paper.

Q6. Could authors give STD (i.e., gray area to represent STD) in Figure 4, 5 and 7? Some curves are too close, I am not sure they are statistically different.

Q7. How is the performance of the super-net? 

Q8. Could the authors add an ablation study on this point? -  "The motivation is to improve all child models in our search space simultaneously, by pushing up both the performance lower bound (the smallest child model) and the performance upper bound (the biggest child model) across all child models." 
- It is important to avoid fine-tune
- From the paper, I am not sure whether the problem is solved by changing the space or the proposed training method (See Q1).
review_title: Official Blind Review #1

Meta-review: This paper presents a NAS method that avoids having to retrain models from scratch and targets a range of model sizes at once. The work builds on Yu & Huang (2019) and studies a combination of many different techniques.
Several baselines use a weaker training method, and no code is made available, raising doubts concerning reproducibility.

The reviewers asked various questions, but for several of these questions (e.g., running experiments on MNIST and CIFAR) the authors did not answer satisfactorily. Therefore, the reviewer asking these questions also refuses to change his/her rating. 

Overall, as AnonReviewer #1 points out, the paper is very empirical. This is not necessarily a bad thing if the experiments yield a lot of insight, but this insight also appears limited. Therefore, I agree with the reviewers and recommend rejection.
Decision: Reject