Efficient Differentiable Neural Architecture Search with Model Parallelism
Authors: Yi-Wei Chen
Qingquan Song
Xia Hu
Venue:  (2021)
Keywords: Neural Architecture Search
Model Parallel

Abstract: Neural architecture search (NAS) automatically designs effective network architectures. Differentiable NAS with supernets that encompass all potential architectures in a large graph cuts down search overhead to few GPU days or less. However, these algorithms consume massive GPU memory, which will restrain NAS from large batch sizes and large search spaces (e.g., more candidate operations, diverse cell structures, and large depth of supernets). In this paper, we present binary neural architecture search (NASB) with consecutive model parallel (CMP) to tackle the problem of insufficient GPU memory. CMP aggregates memory from multiple GPUs for supernets. It divides forward/backward phases into several sub-tasks and executes the same type of sub-tasks together to reduce waiting cycles. This approach improves the hardware utilization of model parallel, but it utilizes large GPU memory. NASB is proposed to reduce memory footprint, which excludes inactive operations from computation graphs and computes those operations on the fly for inactive architectural gradients in backward phases. Experiments show that NASB-CMP runs 1.2× faster than other model parallel approaches and outperforms state-of-the-art differentiable NAS. NASB can also save twice GPU memory more than PC-DARTS. Finally, we apply NASB-CMP to complicated supernet architectures. Although deep supernets with diverse cell structures do not improve NAS performance, NASB-CMP shows its potential to explore supernet architecture design in large search space.

Under review as a conference paper at ICLR 2021
EFFICIENT DIFFERENTIABLE NEURAL ARCHITECTURE SEARCH WITH MODEL PARALLELISM
Anonymous authors
Paper under double-blind review
ABSTRACT
Neural architecture search (NAS) automatically designs effective network archi- tectures. Differentiable NAS with supernets that encompass all potential archi- tectures in a large graph cuts down search overhead to few GPU days or less. However, these algorithms consume massive GPU memory, which will restrain NAS from large batch sizes and large search spaces (c.g., more candidate oper- ations, diverse cell structures, and large depth of supernets).
this paper, we present binary neural architecture search (NASB) with consecutive model parallel (CMP) to tackle the problem of insufficient GPU memory. CMP aggregates mem- ory from multiple GPUs for supernets. It divides forward/backward phases into several sub-tasks and executes the same type sub-tasks together to reduce wait- ing cycles. This approach improves the hardware utilization of model parallel, but it utilizes large GPU memory. NASB is proposed to reduce memory foot- print, which excludes inactive operations from computation graphs and computes those operations on the fly for inactive architectural gradients in backward phases. Experiments show that NASB-CMP runs 1.2x faster than other model parallel approaches and outperforms state-of-the- differentiable NAS. NASB can also save twice GPU memory more than PC-DARTS. Finally, we apply NASB-CMP to complicated supernet architectures. Although deep supernets with diverse cell structures do not improve NAS performance, NASB-CMP shows its potential explore supernet architecture design in large search space
INTRODUCTION
Neural architecture search (NAS) has revolutionized architecture designs of deep learning from manually to automatically in various applications, such as image classification (Zoph & Te 2016)
and semantic segmentation (Liu al. 2019a). Reinforcement learning (Zoph L. 2016 Zoph
al. 2018 Pham et al. 2018), evolutionary algorithms (Real et al. 2017 2019), and differentiable algorithms (Liu et al. 20 Cai et al., (019) have been applied to discover the optimal architecture from a large search space of candidate network structures. Supernets (Zoph et al. 2018 Pham al.)
2018) comprising all possible networks reduce search spaces from complete network architectures to cell structures. Recent acceleration techniques of differentiable NAS (Xie al. 2019 Yao al.)
2020 Chen et al.] 2019 Xu et al., 2020) further diminish search costs to affordable computation overheads (e.g., half GPU day). Prior work (Xu al. 2020) randomly samples partial channels of intermediate feature maps in the mixed operations.
However, supernets of differentiable NAS consume gigantic GPU memory, which constrains NAS from using large batch sizes and imposes restrictions on supernet architectures' complexity. For example, NAS determines networks in shallow supernets (e.g., layers) for deep compact networks (e.g., 20 layers). The cell structures are also required to remain identical for the same type of cells. Data parallelism can increase the search efficiency of NAS by using large batch sizes, such as SNAS (Xie et al.) 2019)), but it requires supernet complexity low enough to fit in single GPU. In contrast, model parallelism can parallelize complex supernets, which distributes partial models to multiple devices. Nevertheless, model parallelism suffers from low hardware utilization. Only one device exccutes its model partition, while other devices stay idle. How to take advantage of multiple GPUs for large supernets efficiently is an open problem.
Search and evaluation code are released link

---

Under review as a conference paper at ICLR 2021
Images
Dev
Model Parallel
Device
Fa
Ba
For
Bw
Reduoe
Device
Fa
Ba
Fwr
Bi
Forwerd
Fowarr
Baokward
Backward
Devi
Time
Normal
Device
Fx
Fw
By
BA
A
W
Reduse
WA
Device
Fw
Bug
BA
Save
Consecutive Model Parallel
Large
Large Cell Divemity
Large Batch Size
Layer Size
Figure 1: Consecutive model parallel (CMP) overlaps the two forward sub-tasks (Fa and Fw) and two backward sub-tasks (Bw and BA)- This new execution order empowers neural architecture search (NAS) to search faster than using model parallel (MP). The right figure shows that CMP can save two cycles from vanilla MP. Furthermore, CMP inherits MP's advantages, like using large batch sizes in the supernet, enlarging layer numbers of the supernet, and even diversifying cell architecture across different layers.
In this paper, we propose a simple and efficient solution, binary neural architecture search (NASB) using consecutive model parallel (CMP), tackle the above limitations. Specifically, supernets have two forward and two backward phases to learn architecture parameters and network weights. CMP distributes several sub-tasks split from the four phases in multiple GPUs and executes the sub-tasks of all forward/hackwwad phases together. Figure illustrates that sub-tasks of forward/hackward phases will be overlapped reduce waiting cycles. Nevertheless, CMP consumes large GPU mem- ory due to two computation graphs existing at the same time. Thus, we introduce NASB to declines GPU memory occupation. NASB utilizes binary and sparse architecture parameters (1 or 0) for mixed operations. It excludes inactive operations in the computation graph and computes feature maps of inactive operations for architecture gradients during the prop-propagation. In this way, NASB-CMP can increase hardware utilization of model parallelism with efficient GPU memory in differentiable NAS.
In experiments CIFAR-10, NASB-CN runs 1.2x faster than using model parallel and can achieve the
pipeline parallel, TorchGPipe (Kim et al. 2020) in server with 4 GPUs ? test error of 2.53 + 0.06% by searching for only 1.48 hours. Our contribution be summarized
as follows:
NASB-CMP is the first NAS algorithm that can parallelize large supernets with large batch sizes. We analyze the acceleration ratio between CMP and traditional model parallelism. Even though complex supernets (c.g., large layers and different cell structures) will not boost NAS performance, NASB-CMP paves the way to explore the supernet architecture design in the future.
NASB utilizes binary architecture parameters and extra architecture gradients computation to reduce GPU usage. It can save memory consumption by accepting twice batch sizes larger than the other memory saving algorithm, PC-DARTS (Xu al. 2020).
We fairly compare NASB-CMP with state-of-the-art differentiable NAS in the same hard- ware and search space. Extensive experiments show that NASB-CMI can achieve compet- itive test error in short search time.
NVIIIA GTX 1080 Ti.
2

---

Under review as a conference paper at ICLR 2021
Image
Risaniee
Normal
Ano
Aou1
GoD
Ga.1
Reduce
Normal
mo(A)))
2 Ans on (mn)
m (mb))
2
Gop ou (m9)
0.3"
0.7
10.0
1.0
Reduce
Sum
0.6
0.4
node
1.0
0.0
node 2
0.6
0.4
Normal
0.8
0.2
node
1.0
0.0
node 3
1. (a)
0.5
0.5
...0
0.0
(c)
Figure 2: Illustration of Binary Neural Architecture Search (NASB). (a) is a supernet made up of normal and reduce cells. (b) portrays the directed-acyclic-graan embodies the mixed operation the solid red lines of the middle figure. NASB builds its supernet with binary mixed operations m which replace architectural matrix A with binary matrix G. The symbol n and o stand for nodes in DAG and candidate operations. Among rows associated with a and the rest elements to O. Only partial
(DAG) used for cell structures. (c)
node (blue bracket) in A, the largest two values operations are active during the search procedure of NASB.
METHODOLOOGY
We first describe the fundamental concepts of one-shot neural architecture search (NAS) in Sec- tion 2. We then portray the consecutive model parallel to enhance NAS search efficiency in mul- tiple devices in Section 2.2 Finally, we explain how we binarize the architectural weights and compute their gradients to cut down the GPU memory consumption in Section 2.3
2.1 ONE- SHOT NEURAL ARCHITECTURE SEARCH
One-shot neural NAS Zoph et al. 2018 is built supernet (a.k.a. meta graph) in which stack normal cells and reduce cells sequentially in Figure (a). Normal cells are analogous to convolutional layers to extract images features. Reduce cells equivalent pooling layers to reduce the spatial dimension of feature maps. All normal cells share the same structure, but each cell still has its network weights. So do all reduce cells. One-shot approaches required to design two cell structures instead of complete neural networks. Figure (b) illustrates one popular cell structure (Pham et al.) 2018). counting the "concat" node. In the h-th cell, the first two nodes are the (h 2)) and (h - 1)-th cells having no inbound edges. The other nodes accept previous nodes whose index lower than the current index. Total edges E (red lines of Figure 2
--node directed-acycllc-graa
(DAG) with total edges E, not
(b)) is (N + +)NN 2))2.. We denote the h-th and ny is DAG node signified Eq.
cell's output as f/4 = con) where 2 S <
Uh-2n
if 0,
n2
Uh-1,
j )=1, 1,
Eien (o))),
2 S S N 1.
A mixed operation mo is the edge between node i and j in the DAG. Let O be set of candidate operations (e.g., convolution, pooling, identity, zero) and A I [x|O] be matrix of architecture
parameters. Eq. formulates the mixed operation mo from node i to j as the weighted sum of all operations og (Liu et al. 2019b).
10
m (m.) = Σ Ao k=1
(2)
where e = (j+1)(3 - 2)/2+ is the edge index. The mixed operations transform the cell structure search to the problem of learning matrices, An and A n. for the normal and reduce cell.
3

---

Under review as a conference paper at ICLR 2021
Given that Let and Ltrain is the loss function L beyond training and validation dataset, respec- tively. Let A comprise AN and Ar- Mathematically, one-shot NAS can be formulated in the following optimization problem,
Lval (w*, ; A) s.t. w arg min Lirain (w, A).
min A
NAS leverages the validation performance to choose well-trained networks that outperform others. After training A, we derive the compact network by pruning unused operations in the supernet. Since the whole paper follows the image classification setting (Liu et al. 2019b] Cai et al.) 2019), assume each node is assigned two inputs and two operations. And we prune node inputs of cells of the supernet by the largest two values of A associated with that node. For simplicity, we use A in replace of A in the following discussion.
2.2 CONSECUTIVE MODEL PARALLEL
Data parallelism can scale up supernets with large batch sizes, but it cannot handle large supernets (e.g., deep supernets with different cell structures). Model parallelism (MP) is able to amortize such large supernets across multiple GPUs, but its hardware utilization is low. MP would generate unwanted waiting cycles across devices. Figure [ displays that the first device becomes idle until the second device finishes its forward and backward phases. The parallelization gets worse use large available GPUs.
Motivated by pipeline parallelism (Huang et al. 2019), we propose consecutive model parallel (CMP) to decrease GPU idle time. Let FA and BA signify the forward and backward phase to update A, and Fw and B be two phases update w. CMP divides the four phases into several sub-tasks and performs sub-tasks of FA and Fwr consecutively, followed by sub-tasks of B and BA. Figure[iillusraaee
that the execution order change by CMP overlaps sub-tasks without waiting for others to finish. Given the number of available GPUs M, Eq. 4 reveals the ratio of execution time between CMP and MP in theory.
Time of CMP
AM 2(M - 11]] = M - NV
1
(4)
Time of MP
2M
We assume FA. BA. Fw, and B. take the same time unit. MP will complete an iteration in 4 units. For CMP, the total sub-tasks is 4M, and 2(M - 1) sub-tasks can be overlapped. If sub-task takes 1/M ideally, CMP will finish iteration in 1/M (4M - 2(M - 1)) units. According to Eq. 4 CMP with two devices could reduce (2)25% time from MP. In practice, Experiment 3.1 3.1 demon- strates that NASB BM runs 1.2x faster than model parallelism without sacrificing test error. The theoretical value for 4 GPU is 1.6 (or reduce 37.5% time). We believe communication overhead and uneven model balance cause the deviation. Communication overhead comes from the intermediate tensors transfer from one to another GPU when models are split into different GPUs. Morcover, the main thread responsible for loading data and backward propagation. The GPU with the main thread always consumes the most GPU memory, which causes uneven model balance.
CMP is a general model parallel approach for any existing differentiable NAS algorithm. However, running BA and B., consecutively asks for two computation graphs, which doubles GPU utilization and deteriorates CMP efficiency. To address the problem of great GPU consumption, we introduce a memory-efficient NAS to CMP, called binary neural architecture search (NASB).
2.3 BINARY NEURAL ARCHITECTURE SEARCH
Binary neural architecture search (NASB) harnesses binary mixed operations m Yao et al. 2020 that convert the real-valued A into sparse binary matrix G. illustrated in Figure Among rows associate node j, m enforces the two largest elements to 1 (active) and the rest elements to 0 (inactive). The row indexes of active elements indicate selected edges node j. while column indexes indicate chosen operations. Notice that NASB does not directly multiply G with candidate operations in Eq. Instead, NASB constructs a set of active operations (active) based on active elements in G. Only those active operations o, E (Oactivr) are included in the forward phase. This technique could stop inactive operations being stored in the computation graph and decrease roughly
A
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1: - Model Parallel
Initialize architecture weights and network weights
2: while not stopped do
3: G2 binarize(A,)
4: Create m using G2 and Eq.
5: Compute (oa,i((We, G5) and Ltrain (Wt, G+) consecutively model parallel
6:
Compute Vwhtrain (We, GG) and V Caaaiaa (we, G+) consecutively // model parallel
7:
Update Wi+1 by descending V Strain (W), G1)
8: Update Ac+1 by descending V Aaaald (W, G1)
9: end while
[ times GPU memory compared to using the multiplication by G.
101
m
(m.)
Σ Gexox(m)) = on()).
(5)
k=1
NASB computes gradients of network weights w using standard back-propagation in the supernet. For the gradients A, NASB estimates 0C/0A approximately by aC/0G:
OC
OL âmo
OC 0mg OL
x ox(n) 8G
3mo dAc,k
dml öGe,k
dmg
Eq. 6 states that gradients of elements in A come from aL/0mg on (n). However, inactive operations are not in the computation graph. NASB saves inputs of inactive operations n in Pa PyTorch Context that used for backward computation. During the backward phase. NASB will compute inactive operations og (n) on the fly and multiply the results with the Oc/0m
Apart from saving unneeded GPU FLOPS and memory, m can avoid performance bias between supernets and compact networks. Supernets using mo assume that the performance of supernets can represent derived compact networks, but non-linear operations (e.g.. Rel.U-Conv-NN) break the representation that causes performance bias (Xie et al., 2019). Instead, the sparse matrix of m activates one operation. The performance of supernets during the search is only for one compact network. Thus, NASB can mitigate the bias caused by non-linear operations.
Algorithm describes how CMP works with NASB. Note that NASB-CMP does not update any parameter (including A and w) until FA. BA, Fw, and B complete. Lerain will use the current binary architecture matrix G, rather than updated G which is the major difference from the alternate algorithm (See Appendix A). Experiment 3.2 demonstrates NASB could save substantial GPU memory than PC-DARTS (Xu et al. 2020), which reduces GPU memory by partial channels of feature maps in mixed operations.
Comparison with other methods. NASP (ao et al. 2020) binarizes A based on A itself, while ProxylessNAS (Cai et al.) (219)) binarizes based on the softmax results A. The two bina- rization approaches are equivalent, but how they handle binary mixed operations (Eq. is different. NASP multiplies G with all operations (i.e., saving active and inactive operations in the computation graph). ProxylessNAS selects two sampled operations (paths) in the computation graph according to multinomial distribution. NASB utilizes the same binarization as NASP but only keeps one active operation in the computation graph according to G.
EXPERIMENTS
We compare WASB-CMP with other parallelisms the CIFAR-IO in Section 3.1 We then inspect the quality of NASB and compare NASB-CMP with state-of-the-art NAS in Section 3.2 Finally, investigate the design of supernet architectures using large layers and different cell structures in which cannot be conducted without saving GPU consumption or model parallel.
Section 3.3
Dataset. CIFAR-10 Krizhevsky & Hinton (009)) is color-image dataset for image classification, composed of 50,000 training images and 10,000 test images for 10 classes. The dataset preparation can be found in Append
5

---

Under review as a conference paper at ICLR 2021
Search Space. The DAG (See Section 2.1 has N 6 intermediate nodes and E 14 total edges. The set of candidate operations follows NASP Yao et al. 2020), where normal operations |On =8 and reduce operations [ n Notice that our baselines also use the same operation sets rather than
their original one ([ON| = O rzl = 8). All operations are included in Appendix
Search Cost
Test Error
6PU
Model Paratiel
aPiie
Consecutive Model Paraliel
Data Parallel
Figure 3: Performance comparison between different parallel approaches in NAS (best viewed in color). GPipe (Huang et al., 2019) an approach of pipeline model parallel. Among the three model parallel approaches (blue, red, yellow), consecutive model parallel (CMP) outperforms them in terms of search cost and test error (lower is better). While the data parallel (green) is the fastest parallel method, but its test errors are not as low as CMP.
3.1 PARALLELISM COMPARISON ON CIFAR-10
The performance of NASB-CMP is compared with other parallel approaches on CIFAR-10, includ- ing data parallelism, model parallelism, and GPipe (Huang et al. 2019), the state-of-the- th------rtt--ttt- art model parallel that pipelines chunks of data into several model partitions. The implementation of Parallel NASB is written in Appendix B and the search and evaluation settings are in Appendix and
Figure B compares the performance of different parallelizations in NASB in varied GPUs CMP runs 1.2x faster than model parallel (MP) and GPipe especially running in 3 and 4 GPUs. According to Eq. 4 four GPUs should run 1.6X faster (or reduce 37.5% search time) than MP. In practice, com- munication overhead and uneven model partitions reduce the ideal speedup ratio. Compared with all parallel approaches, CMP's exccution order change does not degrade the test error. Data parallel takes the lowest search cost, but it does not generate as low test eтгor other model parallel ap- proaches. The reason might be that model replicas in data parallel utilize partial batches to compute architectural gradients, while model parallel can make use of the whole batches. Therefore, CMP is an efficient model parallel approach that helps NAS to utilize large batches.
Despite the competitive performance, the scalability of CMP is inferior. CMP disallows batch sizes from linearly scaling up as large GPUs are involved. For example, 2 GPUs should use 448 (we used 416 instead) if GPU uses 224. Besides, 1-GPU NASB can utilize batch size 448, but NASB-CMP needs 4 GPUs to double batch sizes. The main reason is that CMP keeps two computation graphs (for BA and B.,) simultaneously for overlapping computations, resulting in twice GPU consumption. We believe that mixed parallel combining CMP and data parallel can mitigate the drawback by merging two advantages, accuracy of CMP and scalability of data parallel.
3.2 OHA NAS COMPARISON ON CIFAR-10
Following experiment settings in Appendix C and D we compare NASB and NASB- CMP with sev- eral NAS algorithms on CIFAR-I0. DARTS (Liu et al. 2019b), SNAS (Xie et al. 2019), NASP (Yao) et al. 2020), and PC-DARTS (Xu et al. 2020) are selected as our baselines. DARTS is the pioneer of differentiable NAS. SNAS points out the performance bias between a supernet and derived net- works in DARTS. Both NASP and PC-DARTS reduce GPU memory, which overlaps the scope of this paper. We should select ProxylessNAS (Cai et al. 2019) as a baseline, but their search code on CIFAR-10 is not released. We prefer not to ruin their performance with improper implementation.

---

Under review as a conference paper at ICLR 2021
Table I: Comparison with state-of-the-art NAS on CIFAR-10
Test Error
Params
Search Cost
Search
Model
(%)
(M)
(GPU hours)
Batch Size
DenseNet-BH BC Huang et al.
22017
3,46
25.6
NASNet-A + clo (Zoph
2018)
2.65
3.3
43200
AmoehaNet-B + [Real
2019
2.55 0.05
75600
ENAS + c/o (Pham et al. 2018
2.89
4.6
12
ProxylesSNAS-G c/o (Cai al. 2019
2.08
5.7
NAONet-WS c/o (Luo (0188
2.93
7.2
AlphaX + c/o [Wang cl al.
2019
2.06
9,36
360
DARTS (2nd + (L.u
et al.
2019b
2.83 0.06
3,4
96
64
SNAS-moderate c/o (Xie
2019
2.85 0.02
2.8
36
64
NASP + c/o Yao al. 2020
2.44 0.04
64
PC-DARTS + c/o Xu et al. 2020
2.57 0.07
3.6
2.4
256
DARTS (2nd order) c/o
2019b
7.25 4.20
1.8 0.6
53.62 5.06
60
SNAS +
Xie et al.
2.58 0.08
8.8
11.06 + 0.2
60
NASP + c/o
Yao et al.
2020
2.76 0.35
5.8
6,44 +.4+0.00 0.12
60
PC-DART'S c/o Xu
2020
2.59 0.05
6.9
8,96 + 0.08
60
NASB c/o
2.64 0.09
5.2
3.92 0.38
60
PC-DARTS c/o Xu et al.
2020
2.60 0.17
1.2
4,10 = 0.03
224
NASB c/o
2.49 0.07
6.5
1.64 + 0.06
448
NASB CMP + c/o
2.53 0.06
6.7
(1.48 + 0.02)
896
Instead of directly using their reported results, re-run the baselines from scratch to ensure their hardware and search space are the same. So, we can fairly compare them terms of test error and search cost.
The test error and search cost on CIFAR-10 are stated in Table where "c/o" signifies Cutout (De- Vries & Taylor, 2017) used in the evaluation phase. The first (human designed networks) and the second group of rows are extracted from their papers. The third group compares NASB with differentiable NAS baselines. The fourth group compares NAS algorithms using large batch sizes. Notably, ProxylessNAS attains the outstanding test error, but its supernet structure and search space different from what we use, which might bias the comparison.
In the third group, NASB significantly takes the cheapest search cost, roughly hours, to reach comparable eиrоr of 2.64% with SNAS 2.58% and PC-DARTS 2.59%, not mention the search cost is smaller than the second group. NASB and NASP use similar mixed binary operations, but NASB outperforms NASP in both search cost (3.92 versus 6.44) and error (2.64 versus 2.76). The GPU memory utilization of NASB and NASP is 2,117 MB and 9,587 MB, respectively. These three comparisons indicate that the additional gradient computation for inactive operations useful technique. Notice that DARTS, SNAS, and PC-DARTS use different search space (See Appendix F so the original test errors (second group) differ from what we report in the third group of Table 1. Especially, DARTS tends to overfit the validation set by selecting search space. Its results are not as good as their original search space. Even though NASP uses the same search space, the different batch sizes and random seeds for the search and retrain setting still lead to different results.
connect"" in
The fourth group points out that NASB considerably reduce GPU memory by using twice batch sizes larger than PC-DARTS within 1.64 hours to attain a test error of 2.49. PC-DARTS, how- ever, becomes worse when using large batch sizes. We consider that the Hessian approximation in PC-DARTS fluctuates greatly with large batch sizes, which misleads PC-DARTS to easily select "skip_connect" connectt... NASB-
using four GPUs enables twice batch sizes for NASB finish its search in 1.48 hours without severe test eиror degradation. Its test eтror 2.53% also performs better than other differentiable NAS. The empirical results in the third and fourth groups demonstrate the high efficient NASB with significant memory saving and the strong performance of NASB-CMP.
3.3 LARGE SUPERNETS ON CIFAR-10
One-shot NAS embraces two limitations, (1) searching 8-layer supernets for 20-layer compact net- works and (2) same cell structures (Liu al. 2019h: Pham al. 2018). We hypothesize that

---

Under review as a conference paper at ICLR 2021
Table 2: Compare test error with different supernets on CIFAR-10
Cell Structure
Same
Different
Layers
20
20
NASB 2.59
2.78
2.52
2.82
NASB-CMP
2.58
2.78
2.76
2.68
20-layer supernets with different cell structures can build suitable compact networks. Thanks to NASB and NASB-CMP that reduce GPU utilization and exploit multiple GPUs, we can examine how supermet architectures affect NAS. The search and retrain setting follow Appendix C and D Table [ shows test errors on CIFAR-I0 with various supernet architectures, where the 1st and 2nd rows indicate cell diversity and layers (cells) numbers, Since 8-layer supernets could have six vary- ing normal cells, we magnify each normal cell three times to construct compact networks.
First, supemeet with large layers do not benefit NAS to discover high-quality compact networks. Test errors in NASB (3rd row) and NASB-CMP (4th row) show that most 8-layer supernets can generate lower test errors than 20-layer supernets. The reason is 20-layer supernets have numerous architecture parameters and network weights, and they should ask for more search epochs than 8- layer supernets to train. Insufficient search epochs for deep supernets do not help NAS reach strong compact networks. Furthermore, supernets with different cell structures are not beneficial for NAS as well. When we compare results in 2nd and 4th columns (or 3rd and 5th columns), most super- nets using the same cell structures can generate similar or lower test errors than using different cell structures. The reason is close to the previous one. Different cell structures demand extra search epochs to train high-dimensional architecture parameters compared to homogeneous cell structures. Not enough epochs for different cell structures do not produce low test error. Although the results contradict the hypothesis. NASB-CMP shows its potential to explore complicated supernet architec- tures, which paves the way for designing supernet architectures.
RELATED WORK
Parallelism has been applied to NAS for acceleration (Zoph Le 2016 Xie et al. 2019 Cai al. 2019 Mei et al. 2019). Parameter servers in NAS (Zoph & Le 2016) train several child networks in parallel speed up the learning process of the controller. ProxylessNAS Cai et al. 2019) speed up its retrain phase by distributed framework, Horovod (Sergeev & Del Balso 2018). SNAS (Xie al. 2019) and AtomNAS (Mei et al. 2019) have accelerated the search phase by data parallelism. Data parallelism runs data partitions simultaneen simultancously across multiple devices, but it cannot parallelize large models exceeding the memory of a single device, especially complicated supernets with large batch sizes. In contrast, model parallelism (Lee al. 2014 Harlap et al.,
2018: Huang al. 2019 Kim et al. 2020) excels at parallelizing large models. GPipe (Huang et al. 2019) splits mini-batches mitco- and execute micro-batches in the pipeline of model partitions. The pipeline manner mitigates low hardware utilization in model parallelism. Consecutive model parallel is motivated by pipeline parallelism to overlap sub-tasks of forward/backward phases. We found that batch splitting and re-materializatio (Chen et al., 2016) of GPipe increase NAS search time because frequently updating A and intensifies extra computation. To the best of our knowledge, CMP is the most efficient model parallelism for NAS.
Reducing GPU utilization to enlarge search batch sizes is another acceleration techniques (Xu al.
2020 Chen et al. 2019 Xie et al.) 2019. Yao et al. 2020) Cai et al., 2019). PC-DARTS (Xu et al.)
2020 samples channels feature maps in mixed operations. P-DARTS (Chen et al. 2019) reduce
search space it progressively increases layers of supernets in the search phase. Proxyless lessslesNAS (Cai et al. 2019) and NASP (Yao et al. 2020) binarize A to reduce all operations saved in GPU. NASB uses the same binarization as NASP but saves active operation in the mixed operations. Thus, NASB can reduce GPU consumption substantially and give CMP more space to keep two computa- tion graphs in GPUs.
8

---

Under review as a conference paper at ICLR 2021
5 CONCLUSION
We proposed a simple and efficient model parallel approach, NASB-CMP, which overlaps sub-tasks of forward and backward phases to reduce idle time across GPUs and utilize binary architecture parameters to reduce GPU utilization for heavy supernets. Experiments on CIFAR- show NASB- CMP runs 1.2x faster with large batch size of 896 than other model parallel approaches in 4 GPUs and only took 1.48 hours to attain a test enror of 2.53, surpassing state-of-the-art differentiable NAS. Moreover, NASB-CMP able to accommodate high complicated supernets for search, which paves the way for supernet network architecture design. In the future, will combine the data parallel with NASB-CMP to overcome its inferior scalability, investigate effective and complicated supernet architectures, and analyze the communication overhead of NASB-CMP in multi-node GPU cluster.
REFERENCES
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference Learning Representations, 2019.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint 160.00660000774 2016.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE International Confer- ence on Computer Vision, pp. 1294-1303, 2019.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint / 04552, 2017.
Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv 2018.
preprint
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pp- 103-112, 2019.
Chiheon Kim, Heungsub Lee, Myungryong Jeong, Woonhyuk Back, Boogeon Yoon, Ildoo Kim, Sungbin Lim, and Sungwoong Kim. torchgpipe: On-the-fly pipeline parallelism for training giant models. arXiv preprint arXiv:2004.090 099100, 2020.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Seunghak Lee. Jin Kyu Kim, Xun Zheng, Qirong Ho, Garth A Gibson, and Eric Xing. On model parallelization and scheduling strategies for distributed machine learning. In Advances in neural information processing systems, pp. 2834-2842, 2014.
Chenxi Liu, Liang-Chies Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 82-92, 2019a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference Learning Representations, 2019b.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimization. In Advances in neural information processing systems, pp. 7816-7827, 2018.

---

Under review as a conference paper at ICLR 2021
Jieru Mei, Yingwei Li, Xiaochen Lian, Xiaojie Jin, Linjie Yang. Alan Yuille, and Jianchao Yang. Atomnas: Fine-graine end-to-end neural architecture search. arXiv preprint arXiv:1912.09440, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Proceedings of the 35th International Conference Machine Learning, 2018.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alex Kurakin. Large-scale evolution of image classifiers. arXiv preprint arXiv:1003.01041, 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensor- flow. arXiv preprint arXiv:1802. 802995999, 2018.
Linnan Wang, Yiyang Zhao, Yuu Jinnai, Yuandong Tian, and Rodrigo Fonseca. Alphax: explor- ing neural architectures with deep neural networks and monte carlo search. arXiv preprint orXiv:19033 11059, 2019.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In International Conference on Learning Representations, 2019.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. PC- DARTS: Partial channel connections for memory-efficient architecture search. In International Conference on Learning Representations, 2020.
Quanming Yao, Ju Xu, Wei-Wei Tu, and Zhanxing Zhu. Efficient neural architecture search via proximal iterations. In AAAI, pp. 6664-6671, 2020.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint 1 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
10

---

Under review as a conference paper at ICLR 2021
A ALTERNATE ALGORITHM OF BINARY NEURAL ARCHITECTURE SEARCH
Algorithm displays the alternate fashion to update A and w in the NAS: updating A with fixed and then updating with fixed A. Note that Line of Algorithm V Chrain with updated G which is different from the consecutive algorithm (Algorithm
_ computes the gradients
D
Algorithm 2: NASB
1: Initialize architecture weights A and network weights w
2: while not stopped do
3: Gr binarize
4: Create m using G. and Eq.
5: Compute V VaCuoliaWu, G1) using Eq. // handle the gradients of inactive elements
6:
Update A++1 by descending Vaaial W/, Gt)
7:
G++1 hinarize(A) (A(t)
8:
Create m using G++1 and Eq
9:
Compute Wwrrrain (wy, G(+.1) 1 standard pack-propagation
10: Update w++1 by descending V.. Ctrain (Wy, G(+1)
end while
B IMPLEMENTATION OF PARALLEL NAS
The data parallel leverages PyTorch Paszke et al. 2017) distributed module providing commu- nication interfaces to update parameter tensors between multiple processes. Model parallel and CMP are implemented in multi-threading Each GPU has specialized thread responsible for its model partition. Those threads enable different model partitions to run simultaneously. Without multi-threading, only assigning model partitions specific devices do not automatically overlap sub-tasks. For GPipe, we adopt the corresponding PyTorch package, torchgpipe (Kim et al.) (2020). in replace of GPipe, since GPipe is written in Tensorflow. The chunk setting split mini-batch size to micro-batch size is disabled in the experiment, because enabling the setting increases the search cost.
C SEARCH DETAILS ON CIFAR-10
Our platform is server with 4 GPUs of NVIDIA GTX 1080 Ti, which all search experiments are executed. Supernets consist of 8 cells in which the 3rd and 6th cells are reduce cells, and others are normal cells with initial channels 16. The optimizer for network weights w is momentum SGD with moment 0.9, L.2 penalty 3e 4, and cosine anneal learning rate initialized by 0.025 and minimal 0.001. The optimizer for architecture parameter is Adam with learning rate 3e 4. L2 penalty le 3,,, 3, and (31,92) = (0.5, 0.999). PC-DARTS with large batch sizes (Xu et al. 2020 has unique configurations: initial learning rate 0.1 and minimal 0.0 for SGD optimizer and learning rate 6e for Adam optimizer.
All NAS algorithms will search networks for 50 epochs with varied batch sizes and random seeds. In Experiment 3.I. NABS-CMP is specified search batch size 224, 416, 512, 896 for 1, 2, 3, 4 GPUs, respectively It random seed is 2. In Experiment 3.2 The batch size is 60 determined by DARTS because DARTS consumes the largest GPU memory. We want all NAS algorithms to the same batch size in order to compare each other fairly. Since PC-DARTS is proposed to reduce GPU memory consumption, we also compare the performance of PC-DARTS using a large batch size 224 with NASB and NASB-CMP. NASB is specified with its allowable maximal batch size 448 single GPU, and NASB- CMP uses batch size of 896 4 GPUs. All NAS baselines and NASB use 2, 3, 4, 5, 6 as random seeds, and NASP-CM NASB and NASB-CMP exploit batch size 160 and 256, respectively, for 50 epochs. We ran search experiments twice using random seed 2 and 3 and reported the average test error among the two searches Table 2
uses 2, 3, 9, 11, 18 instead. In Experiment 3.3
11

---

Under review as a conference paper at ICLR 2021
D EVALUATION DETAILS ON CIFAR-10
The compact networks used in the retrain (evaluation) phase have 20 cells (layers), where the one- third and two-thirds of the depth are reduce cells and others are normal cells. We retrain the compact networks from scratch for 600 epochs with the batch size 96, dropout path of probability 0.2, and initial channels of 36. We also add the auxiliary layer in the network with loss weight 0.4. During the evaluation phase, the cutout length 16 is additionally applied for image transformation. The optimizer setting for network weights w is the same as the searching setting. The retrain random seed is assigned to O, which is different from the search seeds.
E PRE-PROCESSING SUMMARY ON CIFAR-10
We preprocess the training images in the following techniques: padding 32 x images with pixels, and then randomly cropping them back direction; normalizing image pixels by the channel mean and standard deviation. The processed training set is split evenly: the first half serves as the final training set, and the other serves as the validation set. SNAS merely relies the training set to search, its training set is not split.
32; randomly flipping images the horizontal
F CANDIDATE OPERATIONS ON CIFAR-10
Table 3 summarizes candidate operations for mixed operations used in NAS papers. Experiment makes use of the first row of Table B as its search space on CIFRAR-10. "skip connect" sym- bolizes identity operation if stride size is or Rel.U-Con--Conv-NN operation. "conv", "sep", and "dil conv" signifies convolution, depthwise-sepaaabe convolutions, and dilated depthwise-separalee convolutions, respectively. "none" means the operation. Note that differentiable NAS baselines (DARTS, SNAS, PC-DARTS) also utilize the first row of Table 3 as their search space.
Table 3: Candidate operations for normal and reduce cells
Normal Cell
Reduce Cell
skip_connee connect (identity)
conv_3x1_1xs dil_conv_3s
skip_connect (identity) avg-pool_3x3
NASB-CMP NASB
conv 1xl
max pool 3x3
3x33
NASP Yao et al. 2020
max max_pool__ pool_5x5
sep_3x3
max max_pool_7xx _x7
sep_5x sep_5x5
sep_7x7
none (zero)
max_pool_3x3
avg-pool_3g
DARTS Liu al.
2019b
skip_connect (identity)
SNAS Xie et all
2019
sep_3x3
PC-DARTS (Xu al.
2020
sep sep-5x5
call_cov_333 _3x3
dil conv_5x5
12


Reviews:
[{"review_id": "ydF9NDBTKa", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##############################################################\n\nSummary:\n\nThis paper provides the interesting method that leverages GPU memory resources more efficiently for supernet (meta-graph) of differentiable NAS. For this, this paper proposes binary neural architecture search and consecutive model parallel (CMP). CMP parallelizes one supernet with multiple GPUs, which allows NAS model to use larger batch size and search space. Additionally, this paper improves neural architecture search speed and hardware utilization with waiting cycles reduction by dividing forward/backward phases into several sub-tasks and executing the same type of sub-tasks. The proposed method shows 1.2x faster search time compared with other model parallel methods and the highest performance among differentiable NAS methods in the experiment section.\n\n##############################################################\n\nReasons for score:\n\nI vote for weak rejecting. I believe that the subject this paper handled is important and I like the idea which parallelizes the model (supernet) which has high potential of usefulness. My main concerns are the unclear experimental results for baseline models which hinders fair comparison and the lack of experiments for demonstrating the effectiveness of the proposed methods. During rebuttal period, I hope the authors can address my concerns.\n\n##############################################################\n\nStrong Points:\n\n1. This paper tackles the main problems of the neural architecture search field: 1) search speed 2) resource efficiency 3) scalability of search space. I think the subject handled in this paper is timely and important.\n2. This paper proposes a more improved binary neural architecture search than that which was proposed by the previous NAS method (NASP) in terms of both the neural architecture search performance and speed.\n3. The proposed CMP has a lot of potentials to be useful and practical. The CMP is suitable to be applied to large-scale datasets and large-scale models. Also, since the CMP target differentiable NAS which is one of the most popular approaches in the NAS field, it has a probability to be developed as universally applicable tools for several differentiable NAS methods. \n\n##############################################################\n\nWeak Points:\n\n1. Although this paper shows many experiments, I have some concerns about the reliability of the experimental results for the baseline models as follows.\n\n(1) [PC-DARTS] I understand the search space in this paper is different from that of PC-DARTS, but all results components (test error, params, GPU hours) on CIFAR-10 of PC-DARTS in this paper are worse than those of PC-DARTS in the original paper as below. It is difficult to understand for me that the performance of PC-DARTS becomes worse even using more parameters. Could the authors report the search results on CIFAR-10 of the proposed method under the same search space of PC-DARTS?\n\nModel                                      Test error(%)        Params(M)     GPU hours \n\nPC-DARTS in the original paper: 2.57                      3.6                      2.4\n\nPC-DARTS in this paper:               2.60                      5.5                     4.10\n\n(2) [NASP] As I understand, this paper follows the search space of NASP. Then I think it is better to use the published results of NASP for a fair comparison.   \n\nModel                                                 Test error(%)        Params(M)    GPU hours \n\nNASP (12 operation) in original paper: 2.44                         7.4                   4.8\n\nNASP in this paper:                                   2.76                         5.5                  6.44\n\n2. Since the batch sizes between NASP and NASB are different in Table 1, the effectiveness of each proposed binary neural architecture search is unclear for me. To clearly show the performance (e.g. the use of GPU memory, the speed gains, FLOPS) of the proposed binary neural architecture search (without CMP), could the authors show comparison between binary neural network search methods (The proposed model, NASP, ProxylessNAS) under the same batch size and a single GPU? (For ProxylessNAS, by referring CIFAR10 conditions in their paper with their official code) \n3. I guess the proposed model will be effective for large-scale dataset such as ImageNet-1K. Could the authors validate the proposed method on ImageNet-1K?\n4. Could the author give more information with specific values for the communication overhead and uneven model problems for better practical use of the proposed method?  \n5. Some expressions such as ‘harnesses’ give me the impression that this paper just use the binary neural architecture search of NASP. It would be better to focus to clarify the representation denoting the different points between them in the overall paper. \n6. I believe if CMP is generalized to other differentiable NAS, CMP is very useful as a search time reduction tool for differentiable NAS even it does not achieve SOTA. Could the authors explain the direction of development of CMP to be integrated to other differentiable NAS such as DARTS, PC-DARTS? \n7. Instead of model parallelism, if we parallelize the learning of architecture parameters and network weights as below concept, it seems to be reducing the search time more than CMP. (50% reduction with 2 GPUs)\n\nDev 1. F_A, F_A,  B_A, B_A    (Save?)\n\nDev 2. F_W,F_W, B_W, B_W<--------->\n\n==> 4 phase\n\nCMP is as follows:\n\nDev 1. F_A, F_W,<-------->B_W, B_A\n\nDev 2.      , F_A, F_W, B_W,B_A\n\n==> 6 phase \n\n(normal: 8phase)\n\nCould you address my question by comparing those two parallelism methods?\n\n8. I think it would be better to average multiple results with different seeds instead of picking top 1 from two results with two seeds.\n\n##############################################################\n\nQuestions during rebuttal period:\n \nPlease address and clarify the weak points above. \n\n##############################################################\n\nSome typos: \n\n1.2X --> 1.2$\\times$\n\n---\n=====POST-REBUTTAL COMMENTS======== \n\nI thank the authors for the response and the efforts in the updated draft. Some of my queries were clarified. However, unfortunately, I still think more needs to be done to demonstrate the effectiveness of the proposed model on large dataset and analyze the effectiveness of each module (binary NAS and CMP). I keep my original decision for these reasons.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cd-yGMcQvDt", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary: \n\nThe paper presents a method to run large batch size with supernets method for Neural Architecture Search (NAS). \nIt also shows model parallelization method CMP that allows 1.2x search speed improvement\nUsing a large batch improves the test accuracy because it allows the model to train on wider variety of data in the backpropagation. \n\n##########################################################################\n\nReasons for score: \n\nNASB-CMP algorithm that can use supernet with large batch and run faster\nThe paper shows model parallelization method CMP that allows 1.2x search speed improvement\nThe paper also shows that it improves the quality of the models by using this technique\nThe paper is clear and coherent\n\nTesting the approach on other NAS Benchmarks (NAS-bench201) would be interesting.\nProvide GPU system details that the experiments where ran on for the comparison with other methods.\n\n##########################################################################\nPros: \n\n- NASB-CMP algorithm that can use supernet with large batch and run faster\n\n##########################################################################\n\nCons: \n\n- Need to provide  GPU system details that the experiments where ran on for the comparison with other methods.\n- Need to test on other NAS benchmarks\n\n##########################################################################\n\nQuestions during rebuttal period: \n\n \nPlease address and clarify the cons above \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CvAGqwWqjCE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes NASB-CMP, which overlaps sub-tasks of forward and backward phases to reduce idle time across GPUs and utilize binary architecture parameters to reduce GPU utilization for heavy supernets. Experiments on CIFAR-10 show NASB- CMP runs 1.2× faster with a large batch size of 896 than other model parallel approaches in 4 GPUs. The large memory usage for NAS algorithms is a critical issue.  The Consecutive model parallel (CMP) overlaps the two forward sub-tasks and two backward sub-tasks is novel, while the path-level binarization is not new. I wish to see a more solid evaluation: the accuracy on the cifar dataset doesn't show advantage over baseline algorithms, and the number of parameters is larger. Although the search cost is significantly reduced, people care about the quality of the search. It's hard to convince people that this approach is better. Given Cifar has a lot of randomness, most NAS algorithms need to demonstrate the effectiveness on ImageNet. The paper needs a more convincing evaluation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2i_mLDY5WKs", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "####Summary:\nThe paper proposed a binary neural architecture search with consecutive model parallelism to tackle the OOM problem for NAS. The method divides forward/backward phases into several sub-tasks and executes thee same type of the sub-tasks together to reduces idle hardware cycles. This approach effectively improves hardware utilization and saves GPU memory.\n\n####Strengths:\nThe idea of consecutive model parallel is cute, effectively overlapping the pipeline from two models. \n\nThe paper is well written.\n\n####Weakness:\n-Binary NAS is not new. The paper is not clear about how much gain is from binary NAS and how much gain is from the consecutive model parallelism.  \n\n-Intuitively, model parallelism is used to support larger models and larger search spaces. However, the paper does not provide strong empirical results on larger models constructed using larger supernets. It can be true that larger search spaces make the search and optimization more challenging. But assuming spending more search and training time, the method should be converging to better models. \n\n-The two-cell based search strategy is quite limited. More recent work on layer-wise search space such as TuNAS and EfficientNet yield better results, more particularly, impressive results on ImageNet. Layer-wise search space can be much larger than the cell-based search space. If the paper does not observe significant performance gain via a larger model from a larger search space using model parallelism, it is not very convincing to adopt such a method. \n\n####Detailed feedback:\n-The existing results on better hardware utilization and shorter search time are good but not strong enough. \n\n-The reviewer strongly believe the paper can make a bigger impact by enabling a larger search space and demonstrating SoTA performance on more impactful workloads, like ImageNet. A very good baseline to use is EfficientNet, where the model can scale up easily. The goal of this paper should not be targeting reducing search time for toy problems, but aim for improving search quality over larger and more impactful problems. \n\n-There are many related approaches to reduce search time and improve search efficient, such as a latent space search via NAO, or using a surrogate cost function, or using search space pruning via MCTS. There is very few comparisons or explanations why this approach is better. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a model parallelism scheme (CMP) for training differentiable NAS with large supernets, which performs the forward and backward passes for multiple tasks at the same time, to increase hardware utilization. Moreover, since CMP consumes large GPU memory due to having multiple computational graphs in memory at the same time, the authors further propose binary neural architecture search (NASB), which binarizes the parameters and gradients to reduce memory footprints and computations. The experimental validation shows that the proposed parallelization technique is more efficient than prior parallelization techniques and can significantly reduce the search cost of differentiable NAS methods. Specifically, the proposed NASB-CMP yields architectures with competitive performance on CIFAR-10, at lower search cost compared with baselines that have memory reduction mechanisms such as PC-DARTS.

This paper received split reviews, with the majority of the reviewers leaning toward rejection (three 5’s) and one leaning positive (6). The reviewers in general agreed that the problem of achieving resource and time-efficiency for NAS is important, and that the proposed idea of consecutive model parallelism is novel and may have some practical impact. The reviewers also found the paper to be mostly clear and well-written.

However, reviewers had a common concern that the experimental validation is weak, since 1) the improvement in search cost seems less meaningful with small workloads such as CIFAR-10, 2) some important baselines are missing, and 3) unclear contribution of CMP and NASB due to a missing ablation study. During the interactive discussion period, the authors provided results on additional baselines (NAO and AlphaX), and intermediate results of ImageNet experiments which shows that the proposed method is faster than the baseline. Yet, the reviewers kept their original ratings even after the internal discussion, as they found the incomplete experiments and missing ablation study unsatisfactory.

In summary, this is a well-written paper proposing a novel idea to tackle the resource and time-efficiency of differentiable NAS, which is a practically important problem. Yet, the experimental validation is too weak to validate the effectiveness and practicality of the proposed method, and thus it seems like a preliminary work not yet ready for publication. However, the work is well-motivated and promising, and addressing the reviewers’ common concerns on missing large-scale experiments and ablation study will make the paper stronger and significantly increase its chance of getting accepted in the next submission.

Decision: Reject