Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: ---
revised score. rebuttal clears my concerns.
---
Summary:

The paper proposes a partially connected differential architecture search (PC-DARTS) technique, that uses a variant of channel dropout for each node's output feature maps, and a weighted summation of concatenating all previous nodes. Searched architecture on CIFAR-10 and ImageNet seems to outperform the one discovered by the original DARTS, however, the results are not directly comparable due to the slight change of search space. 

Introducing this edge normalization is a novel contribution, but it is more like a trick to have a better search space rather than the PC-DARTS itself. My main concerns are about the incremental novelty and experiments are heavily done on one search run, especially the search space is not the same as baseline DARTS. 

I do not think the current version is ready for ICLR, but I am looking forward to seeing the authors' rebuttal and I am willing to revise my review accordingly.

Main concerns

- Incremental novelty about channel sampling.
Doing edge normalization in the PC-DARTS is indeed novel, however, the channel sampling (abbr. PC for partial channel connection) is not. Dropout is widely adopted in all deep learning training since AlexNet. In NAS with parameter sharing, Pham et al. already exploit the channel dropout as shown in ENAS function "def drop_path"(https://github.com/melodyguan/enas/blob/master/src/cifar10/image_ops.py). It is true that previous works treated like one hyper-parameters and do not provide deeper insight about this term, but it is not correct to say in Section 3.4 "Channel sampling ... has never been studied in prior work". In my perspective, the key difference of channel sampling is the retained channel number is always fixed to K, the non-selected channels are not zeroed, where the dropout usually has only a probability K / total_channel and non-selected feature is multiplied to a zero constant. 

Thus, I suggest authors provide additional experiments as in Table 3 to compare the original drop-path with proposed channel sampling. Considering the test error drops from 3% to 2.67% while using PC, it will be more convincing to show the original drop path with probability K / total_channel yields a smaller drop to evidence the effectiveness of proposed sampling. 

- Proposed edge normalization is not a new sampling policy but a new search space.
To my understanding, this edge normalization is effectively a change to the search space rather than the sampling policy, and generalize to many other policies as well, and can be a substantial contribution to the NAS community. However, under current experiments setting, it is hard to isolate the improvement is from this new space or the channel sampling, as detailed later.

- About the motivation.
Throughout the paper, in abstract, introduction, section 3.2 and section 4.4, the authors claim that the larger batch size is particularly important for the stability of architecture search which is not well-studied and lack of references. From Table 4, it is hard to tell the stability is from the larger batch size or the proposed partial channel sampling.


- Questions about experiments

1. Experiments comparing to the baseline is not fair. 
As in Section 4.2, the CIFAR-10 search is different from the original DARTS and P-DARTS in the following manner. The batch size is changed from 64(in DARTS)/96(in P-DARTS) to 256,  super-net is freezed for the first 15 epochs, and introducing the edge normalization parameter \beta_{i,j} increase the search space.  With all these changes, it is quite hard to isolate the effectiveness of proposed PC-DARTS. Two possible simple experiments to compare is, using the original DARTS space and training set, 1) do not update the \beta but use a fixed initialization that all \beta is the same (to mimic original DARTS concatenation); 2) add \beta to original DARTS as well and re-run 1). 

It is completely reasonable to me the contribution of this paper is introducing a novel edge-normalization that is simple and effective to improve the DARTS based approach. If so, the authors could revise the conclusion easily. However, in the least scenario, the experiment comparison should be in a fair way.

2. In original DARTS, error drop from 3% for the first-order gradient to 2.76% while using the second-order one, will this trend occurs with PC-DARTS?

3. Robustness 
Recent work about evaluating neural architecture search reveals that NAS algorithms are sensitive to random initialization[1,2] and the search space [3], this in general leads to a notorious reproducibility problem of current NAS and shows it is not reasonable to only compare final performances on proxy tasks over **one** searched architecture. However, in the stability study in Section 4.4.3, multiple runs are still over the same architecture discovered in earlier experiments. In Section 4.4.2, the paper mentioned the search runs multiple times, yet the reported results in Table 3 are against the single run, as indicated by CIFAR-10 no PC- no EN error 3.00 +- 0.14, which is identical to the results in Table 1 DARTS (1st-order). Could the authors report the results with at least 3 different initializations, and possibly release the seeds? It would significantly strengthen the effectiveness of the proposed approach.

Minor comments

- According to Section 4.4.1 and Figure 3, change the K from 1 to 8, the search cost drops significantly. Does this mean the batch size in the ablation study is changing all the time? How could we know if the test-error is reduced due to the sampling ratio or to the batch size? 

Typos
1. Table 2, caption below the table, \dag is not aligned with the one in the used table.

--- reference ---
[1] Li and Talwalker, Random search and reproducibility of neural architecture search, UAI’19
[2] Sciuto et al., Evaluating the search phase of neural architecture search, arxiv’19
[3] Radosavovic et al., On Network Design Spaces for Visual Recognition, ICCV'19.


review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: ** Summary ** 
This paper proposes to improve the previously work DARTS in terms of the training efficiency, from the large memory and computing overheads. The authors propose a partially-connected DARTS (PC-DARTS) with two components: 1. Partial channel connection 2. edge normalization. To be detailed, they sample a small part of channels to perform connection and add edge normalization to eliminate the potential optimization problem. The results on CIFAR-10 and IamgeNet show the approach is effective, especially in ImageNet, the approach achieves SOTA results. 

** Strengths **
1.	The research direction of reducing the training/memory effort of Neural Architecture Search is important, which is also very hot in nowadays. 
2.	The authors propose two components to perform the efficient search process, which are partial channel connection and edge normalization operations. These methods are reasonable to reduce the training effort. The authors are inspired by ShuffleNet or related research topics. 
3.	The approach is easy to follow and implement, the description of the method is also clear.
4.	The experiments also show comparable results in CIFAR-10 and strong performance in ImageNet.

** Weaknesses **
1.	The operation of partial channel connection choses 1/k channels, the remain channels are directly added to the output. This operation somehow feels too straight to be reasonable, since the remain channel has larger weights (1.0) compared to previous weighted combination. Though the second edge normalization can eliminate little, but this modification still suffers from less careful design, for example, another \alpha weighted combination? Besides, directly bypass is same as perform identity operation, is this right?
2.	The motivation of edge normalization is somehow weak, as the authors are aware of this can also be applied to the original DARTS. From the ablation study, it also shows it works for the original DARTS, which makes the description of 3.3. to be not so convincing. Besides, in the first paragraph of 3.3, what does it mean that “weight-free operations often accumulate larger weights” compared to other operations? I feel the reason is that the weight-free operations are much easier to pass the gradients and easy to be trained. 
3.	In imageNet results, it seems P-DARTS significantly outperform PC-DARTS in terms of the search cost, and the accuracy is similar. This makes PC-DARTS approach to be embarrassing. 
4.	One general point is what the authors mentioned, indeed, for NAS, more training data involved in the training process is much more important compared to perform operations. Therefore, the advantage benefits from the less memory usage of 1/k selection and the more data in one mini-batch. This makes the design of current research directions to be different. Does it mean more training (longer time) and more GPU memory will significantly outperform current results? Even the SOTA approaches. 
5.	Minor point: compared to ProxylessNAS which only samples two paths at each time, their method is much more efficient (though the binarization consumes much). What’s the most advantage of PC-DARTS compared to their method? What if combine their approach with edge normalization?

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The authors propose with this paper a simple extension of DARTS, a popular neural architecture search (NAS) method. This extension addresses one of the shortcoming of DARTS: the immense memory cost. This achieved in a simple way. Instead of using all channels only a random subset is used. To account for that, the authors propose a method to normalizes edges.

The description of the method is very clear. The related work covers many works. I suggest to focus more on the more recent work on NAS and particular work that follows the core idea of DARTS. This deserves more than used two very short sentences since this is the most related work. The experimental section leaves no question unanswered. The setup is clearly described in all cases, ablation studies are conducted wherever it is needed. The method is evaluated on both CIFAR-10 and ImageNet and is even transferred to MS COCO. The search results show some improvements with respect to time budget.
Concluding, this might not be a ground-breaking paper but it is well made and I see no obvious flaws so I do not see any reason to reject it.
review_title: Official Blind Review #1

Meta-review: This paper proposes an improvement to the popular DARTS approach, speeding it up by performing the search in a subset of channels. The improvements are robust, and code is available for reproducibility.

The rebuttal cleared up initial concerns, and after the (private) discussion among reviewers now all reviewers give accepting scores. Because the improvements seem somewhat incremental and only applied to DARTS, R3 argued against an oral, and even the most positive reviewer agreed that a poster format would be best for presentation. 

I therefore strongly recommend recommendation, as a poster.
Decision: Accept (Spotlight)