BETANAS: Balanced Training and selective drop for Neural Architecture Search
Authors: Muyuan Fang
Qiang Wang
Jian Zhang
Zhao Zhong
Venue:  (2020)
Keywords: neural architecture search
weight sharing
auto machine learning
deep learning
CNN

Abstract: Automatic neural architecture search techniques are becoming increasingly important in machine learning area recently.  Especially, weight sharing methods have shown remarkable potentials on searching good network architectures with few computational resources. However, existing weight sharing methods mainly suffer limitations on searching strategies: these methods either uniformly train all network paths to convergence which introduces conflicts between branches and wastes a large amount of computation on unpromising candidates, or selectively train branches with different frequency which leads to unfair evaluation and comparison among paths. To address these issues, we propose a novel neural architecture search method  with balanced training strategy to ensure fair comparisons and a selective drop mechanism to reduces conflicts among candidate paths. The experimental results show that our proposed method can achieve a leading performance of 79.0% on ImageNet under mobile settings, which outperforms other state-of-the-art methods in both accuracy and efficiency.

Under review as a conference paper at ICLR 2020
BETANAS: BALANCED TRAINING AND SELECTIVE DROP FOR NEURAL ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Automatic neural architecture search techniques are becoming increasingly im- portant in machine learning area. Especially, weight sharing methods have shown remarkable potentials on searching good network architectures with few com- putational resources. However, existing weight sharing methods mainly suffer limitations on searching strategies: these methods either uniformly train all net- work paths convergence which introduces conflicts between branches and wastes a large amount of computation on unpromising candidates, selectively train branches with different frequency which leads unfair evaluation and comparison among paths. To address these issues, propose novel neural architecture search method with balanced training strategy to ensure fair comparisons and a selective drop mechanism reduce conflicts among candidate paths. The experi- mental results show that our proposed method can achieve a leading performance 79.0% ImageNet under mobile settings, which outperforms other state-of-the-art methods in both accuracy and efficiency.
INTRODUCTION
The fast developing of artificial intelligence has raised the demand design powerful neural networks. Automatic neural architecture search methods (Zoph & Le. 2016 /Zhog et al. 2018 Pham et al. 2018) have shown great effectiveness in recent years. Among them, methods based on weight sharing (Pham et al.) 2018 Liu et al., 2018 Cai et al., 2018 Guo et al.
2019) show great potentials on searching architectures with limited computational resources. These methods are divided into categories: alternatively training ones (Pham et al.) 2018 Liu et al.
2018 Cai et al. (018) and one- shot based ones (Brock et al. 2017 Bender et |l.] 2018). As shown Fig both categories construct super-net to reduce computational complexity. Methods in the first category parameterize the structure of architectures with trainable parameters and alternatieeyy optimize architecture parameters and network parameters. In contrast, one-shot based methods train network parameters to convergence beforehand and then select architectures with fixed parameters. Both categories achieve better performance with significant efficiency improvement than direct search.
Despite of these remarkable achievements, methods in both categories are limited in their searching strategies. In alternatively training methods, network parameters in different branches are applied with different training frequency or updating strength according to searching strategies, which makes different sub-network convergent different extent. Therefore the performance of sub-networks extracted from super-net can not reflect the actual ability of that trained independently without weight sharing. Moreover, some paths might achieve better performance at early steps while perform not well when actually trained to convergence. In alternatively training methods, these operators will get more training opportunity than other candidates at early steps due to their well performance. Sufficient training in turn makes them perform better and further obtain more training opportunities, forming the Matthew Effect. In contrast, other candidates will be always trained insufficiently and never show their real ability.
Differently, One-shot methods train paths with roughly equal frequency or strength to avoid the Matthew Effect between parameters training and architectures selection. However, training all paths to convergence costs multiple time. Besides, the operators are shared by plenty of sub-networks, making the backward gradients from different training steps heavily conflict. To address this issue,

---

Under review as a conference paper at ICLR 2020
Capr3
Capr
Clarn
Mid
3il
Sii
Hunky
Figure 1: Balanced training and selective drop strategy. The evaluation of each candidate only influence architecture search strategy, and paths with low performance will be gradually dropped to reduce conflicts among paths. In addition, paths still remaining are all trained with comparable frequency insure fair comparison among candidate operators.
Eayal
Cooel
Goack
Crach
Hiaatiyy
Canrch
CousT
Edeutity
Cras?
Miallly
Cears
Clarn
Cous?
Divelly
(a) alternatively training methods
(b) one-shot methods
Figure 2: Weight sharing methods. In alternatively training methods, candidate paths with better performance get more training opportunities or higher fusion weight. one-shot methods, sub- networks are uniformly randomly trained convergence before architecture selection.
we follows the balanced training strategy to avoid the Matthew Effect, and propose a drop paths approach reduce mutual interference among paths, as shown in Fig
Experiments are conducted on ImageNet classification task. The searching process costs less compu- tational resources than competing methods and searched architecture achieves outstanding accuracy of 79.0%, which outperforms state-of-the-es methods under mobile settings. The proposed method is compared with other competing algorithms with visualized analysis, which demonstrates its the effectiveness. Moreover, we also conduct experiments to analysis the mutual interference in weight sharing and demonstrate the rationality of the gradually drop paths strategy.
2 RELATED WORK
Automatic neural architecture search techniques has attracted much attention in recent years. NAS- Net (Zoph & Le 2016 Zoph et al.) 2018) proposes a framework to search for architectures with reinforcement learning, and evaluates each of the searched architectures by training it from scratch. BlockQNN (Zhong et al., 2018 Guo et al., 2018 Zhong et al., (018)) expands the search space to the entire DAG and selects nets with Qelarniing Network pruning methods (Li al., 2019 Noy ct al. 2019) prune redundant architectures to reduces search spaces. Considering the searching policy, of these methods depend on reinforcement learning, evolutionary algorithms and gradient based
algorithms (Bello et al., 2017 Liu et al. 2018 Cai et al. 2018).
The most related works to our method are the ones based on weight sharing proposed by (Pham et al. 2018). from which two streams are derived: Alternatively training methods (Cai et al. 2018
Liu et al.) 2018) and one-shot methods (Brock al., 2017: Bender al. 2018 Guo al. 2019).
Methods in the first stream alternatively train architecture parameters and network parameters. During search process, operators in the super-net are selectively trained and evaluated with a certain policy and the policy is updated dynamically according to the evaluations. Among them, ENAS (Pham) et al. 2018) introduces RL to select paths. DARTS (Liu et al.] 2018)) improves the accuracy and efficiency of paths selection policy by considering the importance of each path trainable parameters. ProxyLessNAs (Cai et al., 2018) proposes to directly search on target datasets with single paths and makes the latency term differentiable. Single-Path NAS (Stamoulis et al. 2019) directly shares
2

---

Under review as a conference paper at ICLR 2020
weights via super-kemel. By contrast, one-shot based methods Guo et al. (2019) Brock al. (201 Beenee et al. (2018) firstly train each path the super-net with equal frequency to convergence, then selected from the super-net and evaluated with fixed parameters. Darts+ Darts+Liang et al. 2019) improves Darts with early stop. Progressive-Noe NNS Chen et al. 2019) gradually increases blocks while searching. HM-NAS Yan et al. (2019) uses mask to select paths. (Sciuto al. 2019)
all the architectures
Our work benefits from the advantages of both categories: On one hand, the importance factors are
evaluated with a gradient based approach but has no influence on training shared parameters. On the other hand, the shared parameters are updated uniformly as those in one-shot.
3 APPROACH
ProxYLesNNAS(Cai et al., 2018) and Single Path One-shot et al. 2019) proposed to train the super-net with only one path on in each step make the performance trained with weight sharing more close to that trained alone. Both of them enhance the performance of weight sharing to higher level. ProxyLessNAS updates architecture parameters and network parameters alternatively. Paths are selectively trained according to their performance, and paths with higher performance get more training opportunities. Single Path One-shot first proposed balanced train all paths until convergence and then use evolution algorithms to select network structures. The equivalent functions of the choice blocks in two methods are described as PL and m OS in Eq
(1((r)
with probability P1,
O1 (1(2)
with probability 1/N,
m (x)
m OS (x))=
(1)
with probability P2- (N(
(N(2)
with probability 1/N.
Our method follows the alternatively training ones, which architecture parameters and network parameters are optimized alternatively in each step. To give a better solution to the problems discussed above, we train each candidate path with equal frequency to avoid the "Matthew effect" and gradually dropped least promising paths during searching process to reduce conflicts among candidate paths.
3.1 PIPELINE
The pipeline of our method is shown in Algorithm choice blocks O2, O2,... OL, as shown in Fig Each choice block O₂ composed of M candidate paths and corresponding operators 01 01,2, .... O1,M- The importance factor of ot,m is denoted as ar,m and 04,m are converted to probability factor Pt.m using softmax normalization.
I First of all, a super-net is constructed with
Secondly, the parameters of ot,m and their importance factors Ct,m are trained alternatively in Phase 1 and Phase 2. When training 0t,m. latency term introduced balance accuracy and complexity. Paths with ai.m lower than the will be dropped and no more trained.
Algorithm 1 Searching Process
Initialization: Denote O, as the choice block for layer with M candidate operators 04,1, ai,,, are the corresponding importance factors of candidate operators and initialized with identical value. Smar denotes the max number of optimization steps.
{or, o,, OI,M
1: while t < Smax do
2: Phasel: Randomly select ol,m, E O; for block O₁ with uniform probability, then fix all an. and train the super-net constructed with the selected 01,my 02. ......... OL,m, for some steps.
3: Phase2: Fix all the parameters in otm and measure their flops/latency. Then evaluate each operator or.m with both cross-entropy loss and flops/latency loss. Update atm according to the losses feedback.
4: for Ol,m E O do
5:
at. < the then O, = O; \ {orm
t=t+1
6: for O, Oj do mi argmarm (a
7: refurn 01 02, OLmm.
3

---

Under review as a conference paper at ICLR 2020
Finally, after alternatively training Oi.m and at,m for given steps, paths with the highest importance factor in each choice block are selected to compose a neural architecture as the searching result.
3.2 BALANCED TRAINING
Alternatively training methods focus computational resources on most promising candidates to reduce the interference from redundant branches. However, some operators that perform well early phases might not perform as well when they are trained to convergence. These operators might get much more training opportunities than others due their better performance the beginning steps. Higher training frequency in turn maintains their dominant position in the following searching process regardless their actual ability, forming the Matthew Effect. In contrast, the operators with high performance when convergent might never get opportunities to trained sufficiently. Therefore, accuracy of alternatively training methods might degrade due to inaccurate evaluations and comparison among candidate operators.
Our method follows the alternatively optimizing strategy. Differently, we only adopt gradient to archi- tectures optimization while randomly sample paths with uniformly probability when training network parameters to avoid the Matthew Effect. More specifically, when updating network parameters of Ol,m in Phase and architecture parameters in Phase 2, the equivalent output of choice block O is given as Opath Eq 2 and Opreh in Eq
{
with probability M if ai,m > tha with probability 0 else.
Opath (x) = 0l.((x)
(2)
with probability pt. ai, > tha with probability 0
Oprch((a (x) = =0
(3)
else.
Where M' is the number of remaining operators O₂ currently, and Pt,m is the softmax form of ar.-- The ar.m of dropped paths are not taken into account when calculating Pt.m The parameters both phases are optimized with Stochastic Gradient Descent (SGD). In Phase 1, the outputs in network parameters, thus gradients can be calculated with the Chain Rule. In Phase 2, the outputs not only depend on the fixed network parameters but also architecture parameters
Eq 2 only depends
O1.-- Note that Ogrch (ar) is not differentiable with respect dj.m- thus we introduce the manually defined derivatives proposed by Cai et al. (2018) to deal with this issue: Eq 3 can be expressed as
Oarch (x) = 9l,m or.m (x). where 9t,0 9t,0 .... AI, one-hot vector with only one element equals to I while others equal to O. Assuming 0g..s/ /0p.. Re 1 according to Cai et al. (2018), the
derivatives of Oprck (x)
w.r.f. 01,m are defined as
aOprch(x)
M' (0preh(z) 0mn. 0pr.j
M' a0preh(xc) 0pr.j
dar,m
Ögi,5 Öpi.s dou,m
Ögi,
Don,m
M' aOprch (x)
Pý (ōmi Pm)
Ögi.j
From now on, Opath (x) and Ogrch( (x) are differentiable w.r.t. network parameters and architecture parameters respectively. Both parameters can be optimized alternatively in Phase and Phase 2.
3.3 SELECTIVELY DROP PATHIS
One-shot based methods, such as Single Path One-shot Guo et al. (2019) also uniformly train paths. These methods train network parameters of each path uniformly to convergence, after that searching policy is applied to explore best structure with fixed network parameters. However, the optimizations of candidate operators in a same choice block actually conflict. Considering N candidate operators in a same choice block and their equivalent functions fi, f2,...... ......f.. fN. given F.,, and Fo.. as the equivalent functions of the sub-supernet before and after the current choice block, 2; and yi input
4

---

Under review as a conference paper at ICLR 2020
data and labels from the training dataset, and L as the loss metric, the optimization of network parameters can be described as:
min Clyis Four (Jn(Fin (144,, wn))), , M4n
(5)
When the super-net is trained to convergence, Fin and Fout is comparatively stable. In this situation, fi (w1), f2(w2)..... Jy (w))) are actually trained to fit a same function. However when operators are trained independently without weight sharing, different operators are unlikely to output same feature maps. Take the super-net in Fig (b) as an example, the four operators in choice block I are likely be optimized to fit each other. Intuitively. Conv3 and Identity are unlikely to fit same function when trained without weight sharing. On the other hand, the operators in the second choice block are trained to be compatible with various input features from different operators the first choice block. In contrast, each operator process data from only one input when networks are trained independently. Both problems widen the gap between network trained with and without weight sharing.
Fewer candidate paths help reduce the conflicts among operators, which will explained in the experiments section. Therefore. drop paths strategy applied in our method to reduce mutual interference among candidate operators during searching process. The paths with performance lower than a threshold will be permanently dropped to reduce its influence on remaining candidates.
When updating at,m in phase 2, follow the strategy in ProxyLessNAS (Cai et al. 2018) sample path in each choice block with probability Pt,m, and optimize at,m by minmizing the expectation joint loss L:
C
M
C= + CC
= LCE + +B
Σ 1=1 m=1
Pl. LL
(6)
where Lce and LLA are the expectation cross-entropy loss and latency loss, LLA. is the flops or latency of operator 0t,m and / is importance factor 01,m and its softmax form Pt,m as sampling probability and use 6 optimize ar.m- The derivatives of Lee and LLA w.r.t. Oi,m can be get from Eq @ that 0,,m m is only applied to evaluate the importance of paths and have no inlluece on the balanced training strategy in Phase 1. After each step of evaluation, paths with low an are dropped and will not be trained anymore:
to balance two loss terms. We regard the
Note
O₂ Oz {or, if mm < that Vmi}
The limitations of alternatively training methods and one-shot based ones are relieved by the proposed balanced training and drop paths strategies respectively. The two phases are trained alternatively until meeting the stop condition, then paths with highest 0z,m from each block selected compose an architecture as the searching result.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets The target architecture is directly searched on ImageNet (Deng et al. (209) for classification task. 50000 images are extracted from training set to train architecture parameters Ot,m and the rest of training dataset used to train network weights.
Search Space We follow the search space in MNASNet (Tan et al. 2018), where each choice block includes an identity operator and 6 Mob2leNetV2 blocks which have kernel size 3,5,7 and expand ratio 3, 6 respectively. SE-Layer (Hu et al. 2018) is not applied to the search space.
Training Detail We search on V100 GPUs for 160 GPU hours. The shared parameters are trained with 1024 batch size and 0.1 learning rate. alt, is trained with Adam optimizer and le-3 initial learning rate. Finally, the searched architecture trained from scratch according to the setting of MobileNetV2 (Sandler et al.) 2018). The searched networks are trained from scratch training dataset with prper-parameters as follows: batch size 2048, learning rate 1.4, weight decay 2e-5, cosine learing rate for 350 epochs, dropout 0.2, label smoothing rate 0.1.
5

---

Under review as a conference paper at ICLR 2020
MB5 CS
MB3 C8
MBS C8
MBS C6
MBS O7
MBS C5
MB3 C5
MB3 C5
MB8 C7
MD3 C7
MB6 C7
mmm
Conys
MBI
MBS
MIS
MBS
MB3
MB3
MB3
MB6
PC
BetaNet-A
C8
MB4 C3
CH
Cf
C7
C6
C5
CT
C5
C6
MB3 C5
C7
C7
M136 C7
Luw
Cou
MB
MTR
MB0
Mil
MBO
MB6
MB0
MB6
MBC
MB6
MB6
MiN
FC
(b) BetaNet-B
Figure 3: The searched architectures with different complexity limitations: BetaNet-A is searched with flops limitation. BetaNet-B is searched with latency limitaion.
4.2
EXPERIMENTAL RESULTS
Table 1: BetaNet-A compared with the state-of ofne-offthhe the-hhhaad art methods under comparable flops.
Method
Top1
Top5
Flops
params
cost (GPU*h)
(%)
(%)
(M)
(M)
MobileNet V2 (Sandler et al.
2018
72.0
91.0
300
3.4
DARTS (Liu et al. 2018
73.1
91.0
595
4.9
96
FBNet-C (wu al. 2018
74.9
375
ProxyLessNAS (Car et al. 2018)
74.6
92.2
320
200
MNASNet+SE [Tan al.
2018
75.4
92.5
317
4.2
40,000
MobileV3+SE++wwiif (Howard et al.
2019)
75.2
219
5.4
BetaNet-A
75.1
92.3
315
3.6
160
BetaNet-A + SE
75.9
92.8
333
4.1
160
MobileNet V2 (Sandler et al.
2018)
1.4
74.7
92.5
585
6.9
BINNeBBO-SEEwiiSS--aauuauuuu
lan Le
2019
76.3
93.2
390
5.3
+ Tan Le (019
78.8
94.4
700
7.8
BetaNet-A x 1.4
77.1
93.5
596
6.2
160
MNASNetX 1.4 + SE (Tan et 2018
77.2
93.2
600
40000
BetaNet-Ax +.4 SE
77.7
93.7
631
7.2
160
BetaNet-Ax 1.4 SE auto-aug swish
79.0
94.2
631
7.2
160
Table 2: BetaNet-B compared with the state-of-the-ar methods under comparable GPU latency.
Method
Topl (%)
Top5
GPU Latency
search cost (GPU hours)
(%)
(ms)
MobileNet V2 (Snndler et al. 2018)
72.0
91.0
6.1
ShuffleNetV2 (1.5) Ma et 2018
72.6
7.3
ProxYLesNNAS-gng gpu CCa et al.
2018
75.1
92.5
5.1
200
MNASNet (Tan et al. 2018
74.0
91.8
6.1
40,000
BetaNet--
75.8
92.8
6.2
160
Searched Archtectures As shown in Fig BetaNet-A and BetaNet-B are searched with flops and GPU latency limitation respectively. BetaNet-A tends to select operators with lower flops at front layers where feature maps are large and operators with higher flops elsewhere to enhance the ability. BetaNet-B tends select large kernels and fewer layers, since GPU performs better with parallelism.
Performance on ImageNet Experiments results compared with state-of-the ofie-----hhhrtttt methods under comparable flops and gpu latency shown in Table and 2 respectivly and architectures

---

Under review as a conference paper at ICLR 2020
77
77
76
76
73
73
72
72
71
MobileNetV2
71
MobileNetv2
MNAASNet
MNASNet
ProxyLessNAS
ProxyLesSNAS
500
100
Bopss
epu latendy
(a) accuracy($)) (6) (M)
(b) accuracy( %) > opu (atency(ms)
Figure 4: BetaNet-A is compared with MobileNetV2 Sandler et al. 2018) ProxyLessNAS Cai et al. (2018) and MNASNet (Tan et al. 2018) with various depth multiplier (0.75, 1.0, 1.4).
Conv3 exp3
Cunvb mxps
Conv
Conv3 exp6
Coav5
Come7 exp6
Adentity
150
20k
150
206
POb
3.2
23k
20k
6.00
256
536
10:9
235
500
75
10th
3256
6.00
255
52c
756
10:0
254
750
1006
125%
3710
(a) ProxyLessNAS
2.22
1.75
1.2
1.00
upiies
0.22
0.2
0.25
2000
(b) Our Method
Figure 5: The variation of and training frequency of 2 choice blocks from each method are shown in (a) and (b). From left to right each line: a in block paths training frequency in block 1, a in block 2, paths training frequency in block 2.
achieve the best performance. SENet (Hu et al. 2018 with ratio 0.0625 applied in table as BetaNet-A SF. BetaNet-A performs better than MobileNerV2 with comparable flops by 3. %. BetaNet-B performs better with comparable latency by A.to.Auto-uuu Auto-augggee (Cubuk et al.) 20188 and SWISH activation (Ramachandran al.) (017)) also applied to the searched BetaNet-A and the performance is further enhanced to 79.0%. As shown Fig BetaNet-A outperforms MobileNetV2, ProxyLessNAS (Cai et al. 2018) and MNASNet (Tan al. 2018) with various depth multiplier.
4.3
ABLATION STUDIES
4.3.1
COMPARED WITH GRADIENT BASED METHODS
Experiments are conducted this sub-section analyze the contribution of the proposed searching approach: balanced training and selective drop. The searching process of BetaNet-A is compared with that of ProxyLessNAS (Cai et al. 2018 8). Fig 5(a) and (b) show the variation of architecture parameters and updating frequency of choice blocks for each method respectively. In the result of ProxyLessNAS shown in F(a) (top left), the conv3 ___pppp6 operator performs better at early phases and get much more training opportunity than other operators and thus trained more sufficiently, which might lead to degraded performance due to the above mentioned "Matthew Effect". Differently, our strategy ensures that all remaining paths trained with roughly equal frequency. In addition,
7

---

Under review as a conference paper at ICLR 2020
70
accuracy
20
10
BetaNetA
Random
100
150
200
250
300
350
epoch
Figure 6: Comparison between architectures searched with different random seeds and randomly sampled ones. Architectures searched with our method and with random policy are shown with solid lines and dotted lines respectively.
methods train remaining with much more frequency than ProxyLessNAS, while ProxyLessNAS spend much steps redundant paths with little promising.
4.3.2 EXPERIMENTS WITH DIFFERENT RANDOM SEEDS
(Sciuto et al. 2019) suggests evaluate weight sharing methods with different random seeds. To examine the robustness of the proposed method, experiments are conducted with our approach with random seeds. architectures from the same search space and super-net. Constraint is applied to insure both searched and randomly sampled networks have similar flops with BetaNetA. All the 16 networks are trained with the same setting. As shown in Fig 6 all the searched architectures have similar performance on ImageNet. Our method outperforms the random sampled ones and other competing methods with robust performance.
searched independently. We also randomly sample architectures
4.3.3 ANALYSIS OF DROP PATH
Besides the policy selecting architecture parameters, optimization of network parameters of our method is different from that of one-shot methods mainly in the drop paths strategy. To give better understanding of its effectiiveness, we conduct experiments on network architectures with 3 different training policies to explore the influence of the number of candidate branches weight sharing. As shown in Figure7] networks in 3 groups are trained with none weight sharing(NS), 2-branch weight sharing branches(B2) and 4-branch weight sharing branches(B4), respectively. Networks in NS, B2, B4 are trained on cifar10 for 30, 60, 120 epochs respectively to insure network parameters are trained with comparable steps in each group.
The accuracy of the 4 networks trained with NS, B2 and B4 are shown in Fig respectively. The experiments indicate 2 phenomenons: 1. The accuracy trained via B2 with 60 epochs is much higher than B4 with 120 epochs, which indicates that less branches in weight sharing helps network parameters converge better. The relative rank of the 4 networks trained with B2 is more similar to those with NS than those with B4, which indicates that less branch can give a better instruction in network selection and demonstrate the rationality of our drop paths strategy.
4.3.4 COMPARISON WITH OTHER WEIGHT SHARING METHODS ON SEARCH STRATEGIES
Our searching and optimizing strategy compared to those other weight sharing approaches, shown in Tab 3 Alternatively training methods tend adopt similar strategies to decide which path to train in optimizing phases and which path to evaluate in search phases. However searching strategies designed select paths and inevitably treat candidates discriminatively. Therefore,
8

---

Under review as a conference paper at ICLR 2020
Ded
Dest
Ded Ded
Dopa
Dopal
(a) network structures and 3 training strategies
(b) accuracy
Figure 7: 4 Networks are trained in 3 groups with different training policies. 00, 01, 10, represent
architectures convl-conv' 1-covv, conv3-convl, conv3-conv3 respectively.
Table 3: Comparison with the other weight sharing methods on their strategies.
Method
supernet optimization
search
sample sub-net
mutual interference
policy
ENAS Pham et al. (2018)
imbalanced imbalanced
RL
all candidates all candidates all candidates all candidates fewer candidates all candidates all candidates fewer candidates
Darts Luu et al. (2018)
gradient
SinglePath Guo ef al.
(2019
balanced
EA
V
ProxyLess Car
(2018)
imbalanced
gradient/RL gradient
Progressive Chen et al.
2019
imbalanced
AXXXX
X
Darts+ Liang al. (019
imbalanced
gradient
HM-NAS Yan et al.
(2019
imbalanced
gradient+mask
X
Ours
balanced
gradient
searching strategies are usually not suitable for optimizing phases. Our method applies different policies in optimizing phases and searching phases to deal with this problem.
Some of the methods sample a single sub-net in each step while training super-net instead of summing up features from all possible paths to bridge the gap between training super-net and evaluating sub- nets. Our methods follow this strategy to improve performance and reduce GPU memory consumption as well. To deal with mutual interference among candidates same choice block. Progressive- Darts Chen et al. (2119) and our method make effort to drop redundant paths to reduce the interference from them.
4.4 DISCUSSION ON TWO STREAMS OF WEIGHTT SHARING METHOOS
There is always a compromise between accuracy and efficiency. Weight sharing shows remarkable improvements in reducing searching cost though introduce inevitable bias accuracy. Alternatively training approaches are actually greedy methods, since they are talented at searching for next optimal solution. Meanwhile, architectures with less competition early phases abandoned. In contrast, one-shot methods attempt offer equal opportunities to all candidates which leads to a more global solution. However operators weight sharing should deal with outputs from multiple former paths, which is challenging for operators with less flops. Therefore these operators suffer more from the mutual interference than those with larger flops.
Our approach tries to balance the advantages and disadvantages of both streams. On one hand, we tries insure the accuracy of most promising operators which similar to the strategies in alternatively training ones. By contrast, only the operators with performance much lower than the average others will dropped. On the other hand, we train paths balanced follows the strategies one-shot based ones. Unlikely, paths are gradually dropped to a lower amount in our method to reduce conflicts.
5 CONCLUSION
This work proposes novel neural architecture search method via balanced training and selective drop strategies. The proposed methods benefits from both streams of weight sharing approaches and relieve their limitations in optimizing the parameters in super-net. Moreover, our method achieves

---

Under review as a conference paper at ICLR 2020
new state-of- the-att result of 79.0% on ImageNet under mobile settings with even less searching cost, which demonstrates its effectiveness.
REFERENCES
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V Le. Neural optimizer search with reinforce- ment learning. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 459-468. JMLR. org, 2017.
Gabriel Bender, Picter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understanding and simplifying one-shot architecture search. In International Conference on Machine Learning. pp. 549-549555 558, 2018.
Andrew Brock. Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model architecture search through hypernetworks. arXiv preprint arXivv/708..o.m 053444444, 2017.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332. 2018.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv: /005:008000000000 2018
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 255. leee, 2009.
Minghao Guo, Zhao Zhong, Wei Wu. Dahua Lin, and Junjie Yan. Irlas: Inverse reinforcement learning for architecture search. arXiv preprint arXiv:1812.05285 2018.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904.00420, 2019.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. arXiv preprint
2019.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excinatin networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Xin Li, Yiming Zhou, Zheng Pan, and Jiashi Feng. Partial order pruning: for best speed/accuracy trade-off in neural architecture search. arXiv preprint 1003:v:9903.0300
2019.
Hanwen Liang. Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping. 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.090) 2018.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference Computer Vision (ECCV), pp. 116-131, 2018
Asaf Noy, Niv Nayman, Tal Ridnik, Nadav Zamir, Sivan Doveh, Itamar Friedman, Raja Giryes, and Lihi Manni.-Manor. Asap: Architecture search, anneal and prune. arXiv preprint arXiv: 194,00/344 2019.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv: 802.032033 2018.
10

---

Under review as a conference paper at ICLR 2020
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. arXiv preprint arXiv: /t////020000/ 04242, 2019.
Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-pate path nas: Designing hardware-efficient convnets in less than hours. arXiv preprint arXiv: (.xii::9044.0 /90::020007777 2019.
Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint 1905. 2019.
Mingxing Tan, Bo Chen, Ruoming Pang. Vijay Vasudevan, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. arXiv preprint arXiv: /807. 1626, 2018.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Fbnet: Hardware aware efficient convnet design via differentiable neural architecture search. arXiv preprint arXiv: 1812.038 2018.
Shen Yan, Biyi Fang, Faen Zhang, Yu Zheng, Xiao Zeng, Hui Xu, and Mi Zhang, Hm-nas: Efficient neural architecture search via hierarchical masking. 2019.
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wir wise neural network architecture generation. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2423-2432, 2018.
Zhao Zhong, Zichen Yang, Boyang Deng. Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Block- qnn: Efficient block-wise neural network architecture generation. arXiv preprint arxiv: 1808.08884444 2018.
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.


Reviews:
[{"review_id": "rkg_5UjRYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a better searching strategy in the context of automatic neural architecture search (NAS). Especially, they focus on improving the search strategy for previously proposed computationally effective weight sharing methods for NAS. Current search strategies for the weight sharing NAS methods either focus on uniformly training all the network paths or selectively train different network paths with different frequency, where both have their own issues like wasting resources for unpromising candidates and unfair comparison among network paths. To this end, this paper proposes a balanced training strategy with “selective drop mechanism”. Further, they validate their approach by showing leading performance on ImageNet under mobile settings.\n\nOverall, I appreciate the effort on exploring new strategies for better search algorithm for NAS in the context of weight sharing methods. However, the analysis of the current approach is limited and some relevant papers are missing. More details below. \n\nArguments:\n1) This paper’s main focus is on developing a better search strategy for NAS based weight sharing methods. The validation of their approach is supported via showing improvement in the accuracy on ImageNet task. However, my main argument is that you have to validate the proposed approach with better analysis on the search space along with improvement on the tasks. For example, “Evaluating the Search Phase of Neural Architecture Search” paper which studies the effectiveness of current search strategies is not referenced in this paper. I would suggest to do analysis of you search strategy by following the analysis experiments conducted in the above paper. \n\n2) The related work doesn’t discuss the latest papers which suggest why and in what ways the current search strategies are based for weight-sharing based NAS approaches. As this paper is trying to address this problem, I would naturally assume that this is well discussed, but it’s missing!\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxVImW65r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper relates to automatic neural architecture search techniques. Current methods have certain drawbacks: Some train all network paths to convergence, which wastes computational efforts in unpromising paths, whereas some don't train all the branches uniformly, which can lead to unfair comparisons.\n\nThe authors propose a model to balance the two issues mentioned above. Their aim is to produced balanced training while trying to reduce conflicts between the different potential network paths.  So their algorithm Has two phases, one where it randomly builds a block in the network and another one where it discards vertices form the layer that are below a certain threshold. \n\nThe experimental results seem sound to me, and I think this is a reasonable approach. \n\nHere some general comments that would help with clarity:\n\n1) I think the authors should explain more clearly what \"Matthew Effect\" in the introduction\n\n2) It's not very clear to me how th_\\alpha is computed. Could this please me made more specific. Section 4.4 says that \"only the operators with performance much lower than the average of others will be dropped.\" Is this approach conservative? Did they try different thresholds?\n\nThere are several minor typos that the authors might want to correct.\n\n1) There is a couple of spaces missing like between \"2018)have shown\" in page 1\n    - The word \"probability\" and p_1, p_2 in (1)\n    - Eq6 on page 5\n\n2) It is customary to use commas before and after \\ldots if one is listing a sequence. The authors don't do this in any of their lists, and this is very strange. \n\n3) In the description of Algorithm 1, I'd change \"S_max is denoted as\" for \"S_max denotes\"\n\n4) In (7) I think there is a \"{\" and a\"}\" missing before and after the o_{l,m_i}. It's set notation. \n\n5) In the discussion, they write the word \"differently.\" Would it be better to write \"by contrast\"?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkxCrcjmcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a new training strategy in achieving better balance between training efficiency and evaluation accuracy with weight sharing-based NAS algorithms. It is consisted of two phrases: in phrase 1, all path are uniformly trained to avoid bias, in phrase 2, less competitive options are pruned to save cost. The proposed method achieved the SOTA on IN mobile setting. \n\n\nOverall I found the idea proposed in the paper intuitive and convincing. Especially I appreciate the ablation study that identified one of the option is encouraged too much in the early stage will can lead to worse final performance. From the methodology perspective, I think this is a solid incremental contribution. However, the highlight of this paper, in my opinion, is the SOTA results on IN. My main concern is that the authors did not indicate that the code/model will be open sourced, which will help verification as well as reproducibility. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a neural architecture search method that uses balanced sampling of architectures from the one-shot model and drops operators whose importance drops below a certain weight.

The reviewers agreed that the paper's approach is intuitive, but main points of criticism were:
- Lack of good baselines
- Potentially unfair comparison, not using the same training pipeline
- Lack of available code and thus of reproducibility. (The authors promised code in response, which is much appreciated. If the open-sourcing process has completed in time for the next version of the paper, I encourage the authors to include an anonymized version of the code in the submission to avoid this criticism.)

The reviewers appreciated the authors' rebuttal, but it did not suffice for them to change their ratings.
I agree with the reviewers that this work may be a solid contribution, but that additional evaluation is needed to demonstrate this. I therefore recommend rejection and encourage resubmission to a different venue after addressing the issues pointed out by the reviewers.
Decision: Reject