Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: *UPDATE* I have read the other reviews and authors' responses. All the reviewers agree that improving single-shot NAS is an important problem, and that sampling single-paths can be a plausible approach for it that avoids weight coupling. Consequently, I have updated my rating to weak accept. I think the paper can be substantially stronger though.
The key claim that set this paper apart from other single-path NAS approaches is that they use a fixed distribution (in particular, the uniform distribution) to sample from unlike others like FBnet who use a trainable distribution. They argue that uniform is parameter-free whereas trainable distributions introduce additional parameters that need to be trained. Their findings motivate a natural follow-up question: could we use a different fixed distribution? Reviewer2 has a similar question in their review. Perhaps a distribution that is weighted according to (some proxy of) how much computational resource the networks take to train? Could such a distribution also be parameter-free and give good benefit over uniform distribution without needing to be updated during supernet training? Such an analysis of the prior distribution will make the paper even stronger.


The paper studies a sequential optimization approach to neural architecture search that can provide some benefit over nested or joint approaches. The core challenge in sequential approaches (which first train the weights of a supernetwork; then search through possible architectures which inherit appropriate weights from the supernetwork) is that the weights for a giant network may not be optimal for the weights of a sub-network encountered during subsequent architecture search. The core benefit of such an approach compared to nested approaches is that the subsequent search phase only needs to perform network inference with inherited weights; not train a sub-network from scratch.
The primary contribution is to fix a prior distribution over architectures and sample from them when training the supernetwork. This simple fix helps the weights of the giant network be more useful when inherited into any sub-networks during architecture search.
The paper will be substantially stronger with a careful study of the choice of the prior distribution and how it affects (a) the rejection sampling step needed to ensure the sampled architectures satisfy complexity constraints, and (b) the performance of the eventual neural architecture search procedure.
Another experiment that will be valuable is to rigorously validate the hypothesis that reducing weight coupling in the supernetwork training is crucially linked to improving the downstream architecture search performance.

Minor (writing comments):
Introduction: "Complex optimization techniques are adopted." This statement is awkward. Does this paper specifically adopt more complex methods to address the shortcomings of gradient methods. Or, does the community broadly research more complex methods as a consequence (and you are advocating for a return to simpler gradient methods)?
Pg5: "we randomly choice a range" -> "choose"

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Authors revise the one-shot NAS algorithm in this work. One-shot NAS that employs a supernet to share the weights between subnets is an efficient NAS algorithm. Authors develop a new training paradigm to train the supernet sufficiently. Specifically, they uniformly sample a single path from supernet at each iteration to make the training effective and stable.

Pros:
1.	Improve the one-shot NAS by uniform path sampling.
2.	Experiments demonstrate effectiveness.

Cons:
1.	Authors only report the performance of the best architecture after fine tuning. It is interesting to see the performance of subnets after obtaining the supernet. Is the better subnet in supernet still better than others after fine tuning?
2.	Given the large number of single paths, it is hard to train each one sufficiently within a supernet. Authors may demonstrate how one-shot NAS can address the problem.

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper presents a new one-shot NAS approach. Parameter updating and structure updating are optimized separately. In the process of parameter optimization, different from the previous methods, the network samples the structure according to the uniform probability distribution. This sampling method can avoid the coupling caused by optimizing the structural parameters at the same time. After training the supernet, the network uses the genetic algorithm to search the structure.

Pros:
+ This paper is well-written and easy to follow. 
+ The method proposed in this paper is very efficient in the search stage and saves memory. During the searching stage, the constraints of the model can be restricted by the genetic algorithm. 
+ This method could directly search architectures on the large scale dataset.
+ The results are promising. Experiments are performed on various structures and including quantization layers. 

Still, I have several minor concerns regarding the algorithm and experiments.

1. Why does the author think that the supernet needs to be trained by single path all the time? In the architecture searching space, under the same computational constraints, some models perform well, and some do not. Why do the good models and the worse models use the same probability to sample? Wouldn't it require much more time to optimize the supernet? In the experiment part, the author compares a related work FBNet, which uses the structure parameter \theta and temperature parameter \tau to control the sampling. When the temperature \tau is high, the distribution of the samples degenerates into the single path sampler, and the accuracy is used to optimize the parameter \theta, which makes \theta tend to select the well-performed models. With the decrease of the temperature \tau, the probability of sampling well-performed architectures is higher than the worse-performed models. Therefore, what is the advantage of the single path sampling method over \tau controlled sampler? It seems that gradually pruning the worse-performed search space would accelerate searching time. More analysis is preferred rather than the superior results from the experiments.

2. Is identity in the search space? For at the last of Page4, the author said 'our choice block does not have an identity branch', and the listed architecture on page 13 has the identity.

3. Is the supernet training for 120 epochs necessary? Is the rank of network structures become steady when training for 120 epochs. Because for the subsequent EA-based network structure search, the loss scale is not important, but the sorting (ranking) is important. (This beyond the scope of this paper, but relates to the NAS area)

4. What is the number of parameters for all the listed models in Tab. 3 & Tab. 4? Listing the parameters makes it easier for the followers.

5. What kind of structure do the mixed-precision networks use? The original resnet module, or the bireal resnet module [r1]?

6. While recalculating the statistics of all the BN,  is backpropagation required or just run the inference without gradient backpropagation.

7. The author uses the approximate complete set of Imagenet to train supernet. If we directly inherit the parameter from the supernet, can we accelerate the training of searched good structure? Will Top-1 acc lower than training from scratch?

[r1] Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm

review_title: Official Blind Review #2

Meta-review: This paper introduces a simple NAS method based on sampling single paths of the one-shot model based on a uniform distribution. Next to the private discussion with reviewers, I read the paper in detail. 

During the discussion, first, the reviewer who gave a weak reject upgraded his/her score to a weak accept since all reviewers appreciated the importance of neural architecture search and that the authors' approach is plausible. 
Then, however, it surfaced that the main claim of novelty in the paper, namely the uniform sampling of paths with weight-sharing, is not novel: Li & Talwalkar already introduced a uniform random sampling of paths with weight-sharing in the one-shot model in their paper "Random Search and Reproducibility in NAS" (https://arxiv.org/abs/1902.07638), which was on arXiv since February 2019 and has been published at UAI 2019. This was their method "RandomNAS with weight sharing".

The authors actually cite that paper but do not mention RandomNAS with weight sharing. This may be because their paper also has been on arXiv since March 2019 (6 weeks after the one above), and was therefore likely parallel work. Nevertheless, now, 9 months later, the situation has changed, and the authors should at least point out in their paper that they were not the first to introduce RandomNAS with weight sharing during the search, but that they rather study the benefits of that previously-introduced method.

The only real novelty in terms of NAS methods that the authors provide is to use a genetic algorithm to select the architecture with the best one-shot model performance, rather than random search. This is a relatively minor contribution, discussed literally in a single paragraph in the paper (with missing details about the crossover operator used; please fill these in). Also, this step is very cheap, so one could potentially just run random search longer. Finally, the comparison presented may be unfair: evolution uses a population size of 50, and Figure 2 plots iterations. It is unclear whether each iteration for random search also evaluated 50 samples; if not, then evolution got 50x more samples than random search. The authors should fix this in a new version of the paper.

The paper also appears to make some wrong claims in Section 2. For example, the authors write that gradient-based NAS methods like DARTS inherit the one-shot weights and fine-tune the discretized architectures, but all methods I know of actually retrain from scratch rather than fine-tuning. Also, equation (3) is not what DARTS does; that does a bi-level optimization.
In Section 3, the authors say that their single-path strategy corresponds to a dropout rate of 1. I do not think that this is correct, since a dropout rate of 1 drops every connection (and does not leave one remaining). All of these issues should be rectified.

The paper reports good results on ImageNet. Unfortunately, these may well be due to using a better training pipeline than other works, rather than due to a better NAS method (no code is available, so there is no way to verify this). On the other hand, the application to mixed-precision quantization is novel and interesting.

AnonReviewer2 asked about the correlation of the one-shot performance and the final evaluation performance, and this question was not answered properly by the authors. This question is relevant, because this correlation has been shown to be very low in several works (e.g., Sciuto et al: "Evaluating the search phase of Neural Architecture Search" (https://arxiv.org/abs/1902.08142), on arXiv since February 2019 and a parallel ICLR submission). In those cases, the proposed approach would definitely not work.

The high scores the reviewers gave were based on the understanding that uniform sampling in the one-shot model was a novel contribution of this paper. Adjusting for that, the real score is much lower and right at the acceptance threshold. After a discussion with the PCs, due to limited capacity, the recommendation is to reject the current version. I encourage the authors to address the issues identified by the reviewers and in this meta-review and to submit to a future venue. 
Decision: Reject