Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes NASB-CMP, which overlaps sub-tasks of forward and backward phases to reduce idle time across GPUs and utilize binary architecture parameters to reduce GPU utilization for heavy supernets. Experiments on CIFAR-10 show NASB- CMP runs 1.2× faster with a large batch size of 896 than other model parallel approaches in 4 GPUs. The large memory usage for NAS algorithms is a critical issue.  The Consecutive model parallel (CMP) overlaps the two forward sub-tasks and two backward sub-tasks is novel, while the path-level binarization is not new. I wish to see a more solid evaluation: the accuracy on the cifar dataset doesn't show advantage over baseline algorithms, and the number of parameters is larger. Although the search cost is significantly reduced, people care about the quality of the search. It's hard to convince people that this approach is better. Given Cifar has a lot of randomness, most NAS algorithms need to demonstrate the effectiveness on ImageNet. The paper needs a more convincing evaluation.
review_title: needs stronger evaluation 

final_score: rating: 5: Marginally below acceptance threshold
review_content: ####Summary:
The paper proposed a binary neural architecture search with consecutive model parallelism to tackle the OOM problem for NAS. The method divides forward/backward phases into several sub-tasks and executes thee same type of the sub-tasks together to reduces idle hardware cycles. This approach effectively improves hardware utilization and saves GPU memory.

####Strengths:
The idea of consecutive model parallel is cute, effectively overlapping the pipeline from two models. 

The paper is well written.

####Weakness:
-Binary NAS is not new. The paper is not clear about how much gain is from binary NAS and how much gain is from the consecutive model parallelism.  

-Intuitively, model parallelism is used to support larger models and larger search spaces. However, the paper does not provide strong empirical results on larger models constructed using larger supernets. It can be true that larger search spaces make the search and optimization more challenging. But assuming spending more search and training time, the method should be converging to better models. 

-The two-cell based search strategy is quite limited. More recent work on layer-wise search space such as TuNAS and EfficientNet yield better results, more particularly, impressive results on ImageNet. Layer-wise search space can be much larger than the cell-based search space. If the paper does not observe significant performance gain via a larger model from a larger search space using model parallelism, it is not very convincing to adopt such a method. 

####Detailed feedback:
-The existing results on better hardware utilization and shorter search time are good but not strong enough. 

-The reviewer strongly believe the paper can make a bigger impact by enabling a larger search space and demonstrating SoTA performance on more impactful workloads, like ImageNet. A very good baseline to use is EfficientNet, where the model can scale up easily. The goal of this paper should not be targeting reducing search time for toy problems, but aim for improving search quality over larger and more impactful problems. 

-There are many related approaches to reduce search time and improve search efficient, such as a latent space search via NAO, or using a surrogate cost function, or using search space pruning via MCTS. There is very few comparisons or explanations why this approach is better. 

review_title: Model parallelism for NAS is new but there is no evidence showing it enables a large search space.

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##############################################################

Summary:

This paper provides the interesting method that leverages GPU memory resources more efficiently for supernet (meta-graph) of differentiable NAS. For this, this paper proposes binary neural architecture search and consecutive model parallel (CMP). CMP parallelizes one supernet with multiple GPUs, which allows NAS model to use larger batch size and search space. Additionally, this paper improves neural architecture search speed and hardware utilization with waiting cycles reduction by dividing forward/backward phases into several sub-tasks and executing the same type of sub-tasks. The proposed method shows 1.2x faster search time compared with other model parallel methods and the highest performance among differentiable NAS methods in the experiment section.

##############################################################

Reasons for score:

I vote for weak rejecting. I believe that the subject this paper handled is important and I like the idea which parallelizes the model (supernet) which has high potential of usefulness. My main concerns are the unclear experimental results for baseline models which hinders fair comparison and the lack of experiments for demonstrating the effectiveness of the proposed methods. During rebuttal period, I hope the authors can address my concerns.

##############################################################

Strong Points:

1. This paper tackles the main problems of the neural architecture search field: 1) search speed 2) resource efficiency 3) scalability of search space. I think the subject handled in this paper is timely and important.
2. This paper proposes a more improved binary neural architecture search than that which was proposed by the previous NAS method (NASP) in terms of both the neural architecture search performance and speed.
3. The proposed CMP has a lot of potentials to be useful and practical. The CMP is suitable to be applied to large-scale datasets and large-scale models. Also, since the CMP target differentiable NAS which is one of the most popular approaches in the NAS field, it has a probability to be developed as universally applicable tools for several differentiable NAS methods. 

##############################################################

Weak Points:

1. Although this paper shows many experiments, I have some concerns about the reliability of the experimental results for the baseline models as follows.

(1) [PC-DARTS] I understand the search space in this paper is different from that of PC-DARTS, but all results components (test error, params, GPU hours) on CIFAR-10 of PC-DARTS in this paper are worse than those of PC-DARTS in the original paper as below. It is difficult to understand for me that the performance of PC-DARTS becomes worse even using more parameters. Could the authors report the search results on CIFAR-10 of the proposed method under the same search space of PC-DARTS?

Model                                      Test error(%)        Params(M)     GPU hours 

PC-DARTS in the original paper: 2.57                      3.6                      2.4

PC-DARTS in this paper:               2.60                      5.5                     4.10

(2) [NASP] As I understand, this paper follows the search space of NASP. Then I think it is better to use the published results of NASP for a fair comparison.   

Model                                                 Test error(%)        Params(M)    GPU hours 

NASP (12 operation) in original paper: 2.44                         7.4                   4.8

NASP in this paper:                                   2.76                         5.5                  6.44

2. Since the batch sizes between NASP and NASB are different in Table 1, the effectiveness of each proposed binary neural architecture search is unclear for me. To clearly show the performance (e.g. the use of GPU memory, the speed gains, FLOPS) of the proposed binary neural architecture search (without CMP), could the authors show comparison between binary neural network search methods (The proposed model, NASP, ProxylessNAS) under the same batch size and a single GPU? (For ProxylessNAS, by referring CIFAR10 conditions in their paper with their official code) 
3. I guess the proposed model will be effective for large-scale dataset such as ImageNet-1K. Could the authors validate the proposed method on ImageNet-1K?
4. Could the author give more information with specific values for the communication overhead and uneven model problems for better practical use of the proposed method?  
5. Some expressions such as ‘harnesses’ give me the impression that this paper just use the binary neural architecture search of NASP. It would be better to focus to clarify the representation denoting the different points between them in the overall paper. 
6. I believe if CMP is generalized to other differentiable NAS, CMP is very useful as a search time reduction tool for differentiable NAS even it does not achieve SOTA. Could the authors explain the direction of development of CMP to be integrated to other differentiable NAS such as DARTS, PC-DARTS? 
7. Instead of model parallelism, if we parallelize the learning of architecture parameters and network weights as below concept, it seems to be reducing the search time more than CMP. (50% reduction with 2 GPUs)

Dev 1. F_A, F_A,  B_A, B_A    (Save?)

Dev 2. F_W,F_W, B_W, B_W<--------->

==> 4 phase

CMP is as follows:

Dev 1. F_A, F_W,<-------->B_W, B_A

Dev 2.      , F_A, F_W, B_W,B_A

==> 6 phase 

(normal: 8phase)

Could you address my question by comparing those two parallelism methods?

8. I think it would be better to average multiple results with different seeds instead of picking top 1 from two results with two seeds.

##############################################################

Questions during rebuttal period:
 
Please address and clarify the weak points above. 

##############################################################

Some typos: 

1.2X --> 1.2$\times$

---
=====POST-REBUTTAL COMMENTS======== 

I thank the authors for the response and the efforts in the updated draft. Some of my queries were clarified. However, unfortunately, I still think more needs to be done to demonstrate the effectiveness of the proposed model on large dataset and analyze the effectiveness of each module (binary NAS and CMP). I keep my original decision for these reasons.
review_title: I believe the subject is important and I like the idea, but more experiments to demonstrate the effectiveness of the proposed approaches are needed.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################
Summary: 

The paper presents a method to run large batch size with supernets method for Neural Architecture Search (NAS). 
It also shows model parallelization method CMP that allows 1.2x search speed improvement
Using a large batch improves the test accuracy because it allows the model to train on wider variety of data in the backpropagation. 

##########################################################################

Reasons for score: 

NASB-CMP algorithm that can use supernet with large batch and run faster
The paper shows model parallelization method CMP that allows 1.2x search speed improvement
The paper also shows that it improves the quality of the models by using this technique
The paper is clear and coherent

Testing the approach on other NAS Benchmarks (NAS-bench201) would be interesting.
Provide GPU system details that the experiments where ran on for the comparison with other methods.

##########################################################################
Pros: 

- NASB-CMP algorithm that can use supernet with large batch and run faster

##########################################################################

Cons: 

- Need to provide  GPU system details that the experiments where ran on for the comparison with other methods.
- Need to test on other NAS benchmarks

##########################################################################

Questions during rebuttal period: 

 
Please address and clarify the cons above 


review_title: running batch for NAS supernet

Meta-review: This paper proposes a model parallelism scheme (CMP) for training differentiable NAS with large supernets, which performs the forward and backward passes for multiple tasks at the same time, to increase hardware utilization. Moreover, since CMP consumes large GPU memory due to having multiple computational graphs in memory at the same time, the authors further propose binary neural architecture search (NASB), which binarizes the parameters and gradients to reduce memory footprints and computations. The experimental validation shows that the proposed parallelization technique is more efficient than prior parallelization techniques and can significantly reduce the search cost of differentiable NAS methods. Specifically, the proposed NASB-CMP yields architectures with competitive performance on CIFAR-10, at lower search cost compared with baselines that have memory reduction mechanisms such as PC-DARTS.

This paper received split reviews, with the majority of the reviewers leaning toward rejection (three 5’s) and one leaning positive (6). The reviewers in general agreed that the problem of achieving resource and time-efficiency for NAS is important, and that the proposed idea of consecutive model parallelism is novel and may have some practical impact. The reviewers also found the paper to be mostly clear and well-written.

However, reviewers had a common concern that the experimental validation is weak, since 1) the improvement in search cost seems less meaningful with small workloads such as CIFAR-10, 2) some important baselines are missing, and 3) unclear contribution of CMP and NASB due to a missing ablation study. During the interactive discussion period, the authors provided results on additional baselines (NAO and AlphaX), and intermediate results of ImageNet experiments which shows that the proposed method is faster than the baseline. Yet, the reviewers kept their original ratings even after the internal discussion, as they found the incomplete experiments and missing ablation study unsatisfactory.

In summary, this is a well-written paper proposing a novel idea to tackle the resource and time-efficiency of differentiable NAS, which is a practically important problem. Yet, the experimental validation is too weak to validate the effectiveness and practicality of the proposed method, and thus it seems like a preliminary work not yet ready for publication. However, the work is well-motivated and promising, and addressing the reviewers’ common concerns on missing large-scale experiments and ablation study will make the paper stronger and significantly increase its chance of getting accepted in the next submission.

Decision: Reject