Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper relates to automatic neural architecture search techniques. Current methods have certain drawbacks: Some train all network paths to convergence, which wastes computational efforts in unpromising paths, whereas some don't train all the branches uniformly, which can lead to unfair comparisons.

The authors propose a model to balance the two issues mentioned above. Their aim is to produced balanced training while trying to reduce conflicts between the different potential network paths.  So their algorithm Has two phases, one where it randomly builds a block in the network and another one where it discards vertices form the layer that are below a certain threshold. 

The experimental results seem sound to me, and I think this is a reasonable approach. 

Here some general comments that would help with clarity:

1) I think the authors should explain more clearly what "Matthew Effect" in the introduction

2) It's not very clear to me how th_\alpha is computed. Could this please me made more specific. Section 4.4 says that "only the operators with performance much lower than the average of others will be dropped." Is this approach conservative? Did they try different thresholds?

There are several minor typos that the authors might want to correct.

1) There is a couple of spaces missing like between "2018)have shown" in page 1
    - The word "probability" and p_1, p_2 in (1)
    - Eq6 on page 5

2) It is customary to use commas before and after \ldots if one is listing a sequence. The authors don't do this in any of their lists, and this is very strange. 

3) In the description of Algorithm 1, I'd change "S_max is denoted as" for "S_max denotes"

4) In (7) I think there is a "{" and a"}" missing before and after the o_{l,m_i}. It's set notation. 

5) In the discussion, they write the word "differently." Would it be better to write "by contrast"?
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors proposed a new training strategy in achieving better balance between training efficiency and evaluation accuracy with weight sharing-based NAS algorithms. It is consisted of two phrases: in phrase 1, all path are uniformly trained to avoid bias, in phrase 2, less competitive options are pruned to save cost. The proposed method achieved the SOTA on IN mobile setting. 


Overall I found the idea proposed in the paper intuitive and convincing. Especially I appreciate the ablation study that identified one of the option is encouraged too much in the early stage will can lead to worse final performance. From the methodology perspective, I think this is a solid incremental contribution. However, the highlight of this paper, in my opinion, is the SOTA results on IN. My main concern is that the authors did not indicate that the code/model will be open sourced, which will help verification as well as reproducibility. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper introduces a better searching strategy in the context of automatic neural architecture search (NAS). Especially, they focus on improving the search strategy for previously proposed computationally effective weight sharing methods for NAS. Current search strategies for the weight sharing NAS methods either focus on uniformly training all the network paths or selectively train different network paths with different frequency, where both have their own issues like wasting resources for unpromising candidates and unfair comparison among network paths. To this end, this paper proposes a balanced training strategy with “selective drop mechanism”. Further, they validate their approach by showing leading performance on ImageNet under mobile settings.

Overall, I appreciate the effort on exploring new strategies for better search algorithm for NAS in the context of weight sharing methods. However, the analysis of the current approach is limited and some relevant papers are missing. More details below. 

Arguments:
1) This paper’s main focus is on developing a better search strategy for NAS based weight sharing methods. The validation of their approach is supported via showing improvement in the accuracy on ImageNet task. However, my main argument is that you have to validate the proposed approach with better analysis on the search space along with improvement on the tasks. For example, “Evaluating the Search Phase of Neural Architecture Search” paper which studies the effectiveness of current search strategies is not referenced in this paper. I would suggest to do analysis of you search strategy by following the analysis experiments conducted in the above paper. 

2) The related work doesn’t discuss the latest papers which suggest why and in what ways the current search strategies are based for weight-sharing based NAS approaches. As this paper is trying to address this problem, I would naturally assume that this is well discussed, but it’s missing!

review_title: Official Blind Review #1

Meta-review: This paper proposes a neural architecture search method that uses balanced sampling of architectures from the one-shot model and drops operators whose importance drops below a certain weight.

The reviewers agreed that the paper's approach is intuitive, but main points of criticism were:
- Lack of good baselines
- Potentially unfair comparison, not using the same training pipeline
- Lack of available code and thus of reproducibility. (The authors promised code in response, which is much appreciated. If the open-sourcing process has completed in time for the next version of the paper, I encourage the authors to include an anonymized version of the code in the submission to avoid this criticism.)

The reviewers appreciated the authors' rebuttal, but it did not suffice for them to change their ratings.
I agree with the reviewers that this work may be a solid contribution, but that additional evaluation is needed to demonstrate this. I therefore recommend rejection and encourage resubmission to a different venue after addressing the issues pointed out by the reviewers.
Decision: Reject