Deeper Insights into Weight Sharing in Neural Architecture Search
Authors: Yuge Zhang
Quanlu Zhang
Junyang Jiang
Zejun Lin
Yujing Wang
Venue:  (2020)
Keywords: Neural Architecture Search
NAS
AutoML
AutoDL
Deep Learning
Machine Learning

Abstract: With the success of deep neural networks, Neural Architecture Search (NAS) as a way of automatic model design has attracted wide attention. As training every child model from scratch is very time-consuming, recent works leverage weight-sharing to speed up the model evaluation procedure. These approaches greatly reduce computation by maintaining a single copy of weights on the super-net and share the weights among every child model. However, weight-sharing has no theoretical guarantee and its impact has not been well studied before. In this paper, we conduct comprehensive experiments to reveal the impact of weight-sharing: (1) The best-performing models from different runs or even from consecutive epochs within the same run have significant variance; (2) Even with high variance, we can extract valuable information from training the super-net with shared weights; (3) The interference between child models is a main factor that induces high variance; (4) Properly reducing the degree of weight sharing could effectively reduce variance and improve performance.

Under review as a conference paper at ICLR 2020
DEEPER INSIGHTS INTO WEIGHT SHARING IN NEU- RAL ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
With the success of deep neural networks, Neural Architecture Search (NAS) way of automatic model design has attracted wide attention. As training every child model from scratch very time-consuming, recent works leverage weight-sharing to speed up the model evaluation procedure. These approaches greatly reduce computation by maintaining single copy of weights on the super-net and share the weights among every child model. However, weight-sharing has no theoretical guarantee and its impact has not been well studied before. In this paper, we conduct comprehensive experiments to reveal the impact of weight-sharing (1) The best- performing models from different runs or even from consecutive epochs within the same run have significant variance; (2) Even with high variance, can extract valuable information from training the super-net with shared weights; (3) The interference between child models is main factor that induces high variance; (4) Properly reducing the degree of weight sharing could effectively reduce variance and improve performance.
INTRODUCTION
Learning to design neural architectures automatically has aroused wide interests recently due to its success in many different machine learning tasks. One stream of neural architectures search (NAS) methods is based on reinforcement learning (RL) (Zoph & Le 2016 Zoph et al. 2018; Tan et al. 2019), where a neural architecture is built from actions and its performance is used as reward. This approach usually demands considerable computation power each search process takes days with hundreds of GPUs. Population based algorithms (Gaier & Ha 2019 Liang al., 2018 Jaderberg et al.] (017) are another popular approach for NAS, new trials could inherit neural architecture from better performing ones as well as their weights, and mutate the architecture to explore better ones. also has high computation cost.
To speed up the search process, family methods attracts increasing attention with greatly reduced computation (Pham al., 2018; Liu et al. [2018c] [Bender et al., (018). Instead of training every child model, they build single model, called super-net, from neural architecture search space, and maintain single copy of weights proposed on this model, e.g., training with RL controller (Pham et al.) 2018). training by applying dropout (Bender et al. 2018) or architecture weights candidate choices (Liu al. 2018c). In these approaches, weight-sharing is the key for the speedup. However, weight sharing has no theoretical guarantee and its impact has not been well studied before. The directions of improving such methods would be more clear if some key questions had been answered: 1) How far is the accuracy of found architecture from the best one within search space? 2) Could the best architecture be stably found in multiple runs of search process? 3) How does weight sharing affect the accuracy and stability of the found architecture?
the super-net. Several training approaches have been
In this paper, we answer the mentioee questions using comprehensive experiments and analysis. To understand the behavior of weight sharing approaches, we use small search space, which makes it possible to have ground truth for comparison. It is a simplified NAS problem, therefore, making it easy to show the ability of the NAS algorithms with weight sharing. As result, we find that the rank of child models is very unstable in different runs of the search process, and also very different from ground truth. fact, the instability commonly exists not only different runs,
In this paper, we instability and variance interchangeably.

---

Under review as a conference paper at ICLR 2020
but also in consecutive training epochs within the same Also worthy of note, in spite of high variance, we can extract statistical information from the variance, the statistics can be innovatively leveraged to prune search space and improve the search result.
To further understand where the variance comes from, we record and analyze more metric data from the experiments. It is witnessed that some child models have interference with each other, and the degree of this interference varies depending on different child models. At the very end of the super-net training, training each child model in one mini-batch can make this model be the best performing one on the validation data. Based on the insights, we further explore partial weight sharing, that is, each child model could selectively share weights with others, rather than all of them sharing the same copy of weights. It can be seen reduced degree of weight sharing. One method we have explored sharing weights of common prefix layers among child models. Another method to cluster child models into groups, each of which shares copy of weights. Experiment results show that partial weight sharing makes the rank of child models more stable and becomes closer to ground truth. implies that with proper degree or control of weight sharing, better child models can be more stably found.
To summarize, our main contributions are follows:
We define new metrics for evaluating the performance of the NAS methods based on weight sharing, and propose down-scaled search space which makes it possible have a deeper analysis by comparing it with ground truth.
We design various experiments, and deliver some interesting observations and insights. More importantly, we reveal that valuable statistics be extracted from training the super-net, which can be leveraged to improve performance.
We take a step further to explain the reasons of high variance. Then we use decreased degree of weight sharing, which shows lower variance and better performance, to support the reasoning.
2 RELATED WORKS
Neural Architecture Search (NAS) invented to relieve human experts from laborious job of engineering neural network components and architectures by automatically searching optimal neural architecture from a human-defined search space. Arguably, the recent growing interesting in NAS research begins from the work by Zoph and Le (Zoph & Le. 2016) where they train a controller using policy gradients (Williams) 1992) to discover and generate network models that achieve state-of- the-art performance. Following these works, there is a growing interest using RL NAS (Pham) et al. 2018: Baker et al.) 2017 Tan et al. 2019 Zoph et al. 2018). There have also been studies in evolutionary approaches (Real et al. 2019 2017 Mikkulainen et al., 2017 Xie & Yuille 2017 Liu
ct al. 2018b). Most of these works still demand high computational cost that is not affordable for large networks or datasets.
Weight sharing approaches Weight sharing means sharing architecture weights among different components or models. Pham et al. (0188) combined this approach with previous work of NAS (Zoph) & Lee 2016) and proposed Efficient Neural Architecture Search (ENAS). where a super-net is constructed which contains every possible architecture in the search space as its child model, and thus all the architectures sampled from this super-net share the weights of their common graph nodes. It significantly reduces the computational complexity of NAS by directly training and evaluating sampled child models directly on the shared weight. After the training is done, a subset of child models is chosen and they are either finetuned or trained from scratch to get the final model.
Many follow-up works leverage weight sharing as a useful technique that can be decoupled from RL controllers, including applying dropout on candidate choices (Bender et al. 2018). converting the discrete search space into differentiable one (Liu et al. 2018c] Wu al.]2018) Xie al. [209)), searching via sparse optimization (Zhang al. 2018), and directly scarching for child models for large-scale target tasks and hardwares (Cai et al. 2019).
Previous studies on stability of weight sharing All the weight-sharing approaches are based on the assumption that the rank of child models obtained by evaluating a child model of the trained super-net is valid, at least, capable of finding one of the best child models in the search space.
2

---

Following DARTS (Liu et al. 2018c), we design search space for a cell, as shown in Figure and stack four cells each of which has the same chosen structure, forming a convolutional neural network. A cell is defined as directed acyclic graph (DAG) of nodes (tensors) Zi,... Zn. A cell starts with To, which is the output tensor of its previous cell fed through a Ixl conv layer to match the targeted Under review as a conference paper at ICLR 2020
However, this assumption does not generally hold. For example, Guo et al. (2019) believed that child models are deeply coupled during optimization, causing high interference among each other. [Sciuto et al.] (2019) discovered that there is little correlation between the rank found by weight sharing and rank of actual performance. Anonymous 2020) also conducted a benchmark experiment that shows similar instability when the search space contains thousands of child models, by leveraging ground truth results measured by (Ying et al. (2019). However on the other hand, research on transfer learning (Razavian et al. 2014), where particular model trained on a particular task can work well on another task, and multitask learning [Luong et al. 2015), where multiple models trained for multiple tasks share the same weights during training, suggest otherwise and encourage the weights to be shared among child models, to reduce the long training time from scratch to convergence (Pham et al. 2018). Therefore, in this paper we show whether weight sharing helps and why, using comprehensive experiments.
3 WEIGHT-SHARING. VARIANCE AND INVARIANCE
3.1 METHODOLOGGY
The space of a typical neural architecture search task usually has more than 10t0 different child models (Tan et al. 2019 Liu al. 2018a 2019), thus, it is impossible train them all, which leads to the problem that without ground truth is hard to assess how good the found child model is in the search space. To solve this problem, we down-scale search space under the assumption that small search space is easier than large search space, if the search methods works in large search space they are also supposed to work in small search space.
Softmax
Output
Cell
O (K.) can be one of the following:
Cell
X2
1. 3 x max pooling
Cell
2. 3 separable convolutions
Oco,2)
3. 5 separable convolutions
Cell
0 (o.1)
4. 3 x dilated separable convolutions
3x3
x2
Input
Images
Figure 1: Down-scaled search space.
number of channels in the current layer. The output of the cell is simply the sum of 21...., In. The DAG is designed to be densely connected, i.e..
Σ (z.)) OSi<s
Jy
where O(i.)) is the selected operation at edge (i,j). In the down-scaled search space, each cell contains only two nodes (i.e., 2) and O(..)) one of the four primitive operations in Figure Thus, child model only has 43 64 possible choices, which makes it easy to have ground truth. For convenience, also name all the child models with three digits (each digit is in [1, 4)), denoting the choice of ((,,)). ( (112)) respectively.
As introduced in the previous section, sharing single copy of weights can be seen as training an expanded super-net. To better understand the effect of weight sharing, we simplify the training process. Specifically,
uniformly generate child models. Each bacc- trains one child model and only the weights of this model are updated by back poookpropggttion. After training the shared weights for number of epochs, these shared weights evaluate the performance of all child models
3

---

Under review as a conference paper at ICLR 2020
on the validation set. On the other hand, the ground truth performance of each child model is obtained by training each of them independently from scratch with the same setting as weight sharing, and averaging over 10 runs with different random seeds for initializations. The lookup table can be found in Appendix C
For the rest of the experiments listed in this paper. if not otherwise specified, the models are trained with the dataset of CIFAR-10 on an NVidia K80 GPU. We use SGD with momentum 0.9 and weight decay 10-3 as our optimizer. The initial learning rate is set to 0.025 and annealed down to 0.001 following a cosine schedule without restart (Loshchilov & Hutter 2016). The batch size is set to 256. Number of epochs is 200. Detailed experiment settings are described in Appendix A
3.2 VARIANCE WEIGIIT SHARING
To measure stability and performance of weight sharing methods, we first need to measure rank, as weight sharing methods use the performance ranks of child models on validation set to choose the final output child model. We leverage Kendall's rank correlation coefficient, i.e., Kendall's Tau (Kendall 1938). which provides a measure of correspondence between two ranks R₁ and R2. Intuitively, T( R1. R2) can be high
if R1 and Rg perfeetly matched, or low -1 when R1 and Rg are exactly inverted. We use instance to denote the procedure of training the super-net and generating rank R3 of child models validation set. We then define the following three metries.
S-Tau: S-Tau is to measure the stability of generated ranks from multiple instances. For N instances with ranks R1, R2,.... RN. S-Tau can be calculated as,
2 N(N - 1)
E ISKJSN
T(R, RJ)
(2)
This metric is to compare the rank produced by an instance with ground truth rank. We also use Kendall's Tau to measure the correlation of the two ranks, i.e., T(R, Rgt).
GT-Tau:
Top-n-Rank (TnR): It is to measure how good an instance is at finding the top child model(s). TnR is obtained by choosing the top n child models from the generated rank of an instance and finding the best ground truth rank of these n child models.
Similar to good deep learning model that could constantly converge to point that has similar performance, weight-sharing NAS is also expected to have such stability. we use the same initialization seed and the same sequence of child models for mini-batches in different instances, they will produce the same rank after the same number of epochs. To measure the stability when applying different seeds or sequences, we do several experiments and the results are shown in Table [ For the first three rows, each of them is an experiment that runs 10 instances. The first one makes initialization seed different in different instances while keeping other configurations the same. The second one uses a random child model sampler with different seeds to generate different order of the 64 child models for different instances, each instance repeats the order in mini-bated training, and seeds for weight initializations are the same for those instances. The only difference between the second and the third one is that after every 64 mini-batches new order of the child models is randomly generated for the next 64
we call it different order with shuffle.
From the numbers, we can see that different initialization seeds make the generated ranks very different. Some instances generate high correlation ranks while some others even show negative correlation. To give an intuitive understanding of the S-Tau values, we also show two baselines, i.e., random rank which includes 10 randomly generated ranks and ground truth which trains the 64 child models independently and generate a rank in every instance. The rank generated by training child models independently is much more stable. S-Tau of different orders with or without shuffle is lower than 0.5. But Tau values of the three experiments under the same epoch are not comparable, because S-Tau T-auu varies lot in different epochs. For example, as shown in Figure 2 S-Tau of the 10 instances with different seeds varies in the range of 0.4 even in the last several epochs - it could be high 0.7, which, some extent, explains inconsistent results from previous works (Sciuto et al. 2019). Observation 1: The rank of child models on validation set is very unstable in different instances.
as low as 0.3
We also compared the generated ranks with the ground truth rank with GT-Tau as shown in Table Similar to S-Tau, GT-Tau values of the three experiments are also much lower than that of ground
4

---

Under review as a conference paper at ICLR 2020
Table 1: Instability of multiple runs (Le., instances) measured with S-Tau. Max Tau means the maximum value of the N1N-12 Taus. Similarly, Min Tau the
Difl. seads
0.7
Dit. orders (shuflie)
2
0.6
Ground trulh
minimum value. The numbers are obtained at the 200-th epoch.
Experiments
S-Tau
Max Tau
Min Tau
Different seeds
0.5415
0.7977
0.2171
0.2
Different orders
0.3930
0.7021
-0.0129
120
140
160
180
200
Diff. orders (shuffle)
0.4403
0.7163
0.0764
Epochs
Random rank
0.0382
0.2181
0.1552
Figure 2: S-Tau evaluated after every epochs for "Diff. seeds", "Diff. orders (shuffle)" and "Ground truth".
Ground truth
0.7120
0.8191
0.6650
Different epochs
0.5310
0.8752
0.0918
Table 2: Comparison with ground truth with GT-Tau and TnR. Each number is an average of 10 numbers, either from 10 instances from 10 epochs of one instance. The subscript shows the standard variance of these 10 numbers.
GT-
TIR
T3R
Different seeds
0..56721.1477
18. 18.5000++1.000 2042
17.6000
Different orders
0.1625=+0.0982
16.9000
11.5000
Diff. orders (shuffle)
0.5108
15.6000+s.470s
11.2000#5.3462
Ground truth
0.798510.0257
4.800043.8419
1.6000
Different epochs
0.505310.1399
15.2000
13.2000
truth, and the variance of GT-Tau across different instances is also high, which implies that the generated rank is not qualified to guide the choosing of good-performing child models. This is further proved by TIR and T3R. TIR ranges from 15 to 19, meaning that if choosing top 1 child model is unlikely to obtain a good-performing model. T3R slightly better than TIR, but at the cost of training more child models from scratch, which is usually not affordable for large search space. Observation 2: Though weight sharing shows the trend of following ground truth (has correlation), the generated rank is still far from the ground truth rank, seemingly having hard limit.
Now that multiple instances have shown high variance, how about the stability of one single instance the end of the training? We then look into single instance by measuring variance of the ranks generated in consecutive epochs. Specifically, for each instances from the previous three experiments, we obtain 10 ranks each from one of the last 10 epochs (i.e., 191 -200), measure the stability of the 10 ranks and compare them with ground truth rank. We calculate S- S-Tau to show the mutual correlation among these 10 ranks. This value turns out to vary between 0.39 0.63 for different orders (shuffle), which means there is high variance between epochs even within single instance. We show the median number among instances in Table T GT-Tau also varies a lot along epochs. Taking one instance from "Diff. orders (shuffle)" with final GT-Tau 0.47, we found that, as shown in Table 2. actually its GT-Tau varies between 0.1 to 0.7, with standard variance 0.14, in the last 10 epochs. Observation 3: The generated ranks in the last several epochs of the same instance highly unstable, indicating that picking a different epoch to generate the rank has great impact on the finally obtained performance.
3.3 EXPLOITABLE FROM VARIANCE
Though the generated ranks show high variance, there is some statistical information that can be extracted from the variance. For the "Diff. orders (shuffle)" experiment, we have 10 ranks on the 200th epoch of the 10 instances. For each child model, we retrieve its rank values in the 10 ranks, and show the distributions in Figure 3al The child models are ordered with their ground truth accuracy, the left ones are better than the right ones. We ranked as bad ones (also observed by Bender et al. (218)), while it is almost not possible to tell which one is better from the good-perfonee performing ones. Furthermore, evaluate the ranks generated
that bad- perfforing models more likely
5

---

Under review as a conference paper at ICLR 2020
from the last 10 epochs of the same instance in the same way. The result is shown in Figure 3b which is almost same result as the mulli-instanee experiment, implying that we can simply run one instance and generate multiple ranks from different epochs, these ranks can be used to quickly filter out par-ornfogning models. Insight Though weight sharing is unstable, the generated ranks can be leveraged quickly filter out bad-performing child models, and potentially used do search space pruning, e.g. progressively discarding the the top ones.
child models and only further training
0
10
10
20
20
Rank
30
30
40
40
50
50
60
26
36
41
56
61
16
21
36
51
56
Subgraphs
Subgraphs
Final rank 10 instances
Rank of last 10 epochs
Figure 3: Distribution of rank achieved for each child model, ordered from the ground-truth-ber to worst. Each box extends from the lower to upper quartile values its corresponding data, with a line marking the median. The whiskers show the range of the data. Outliers are marked with circles.
0.7
0.6
0.7
Accumey
0.5
Accuracy
0.6
0.4
0.3
0.4
0.2
0.1
19500
19560
19580
19600
19620
19640
19660
39140
39160
39180
39200
39220
39240
39260
Mini-batches
Mini-batches
(a) Accuracy, finetune epach 100
(b) Accuracy, finetune at epoch 200
0.8
0.6
0.6
GI-Tau
0.5
0.3
0.0
0,2
-0.2
0.0
19540
19560
19580
19600
19620
19640
19660
39140
39160
39180
39200
39220
39240
39260
Mini-batches
Mimi-batche
(c) GT Tau, finetune epoch 100
(d) GT Tau, finetune epoch 200
Figure 4: For the solid line part, all the child models share one copy, while the dotted lines represent the part where each child model training independently. The accuracies are evaluated after each mini-batch, and GT-Tau is calculated correspondingly.
As shown in the previous experiment, single instance can converge to state that the performance of some child models can no longer be distinguished, which can be seen as a kind stable state. In this state, further training of the super-net does not improve the quality of ranks (also proved in 4.1 We propose to finetune each child models independently by inheriting the weights from a snapshot of the seter-net. Specifically, in Figure 4al train the super-net for 100 epochs and then finetune the child models for 64 mini-batches. We can see with only 10 mini-batches. Finetuning from the 200-th epoch shows even better results in Figure 4b
that GT-Tau quickly increases up to 0.6

---

Under review as a conference paper at ICLR 2020
the convergence is faster (using mini-batcher and GT-Tau is more stable (close to 0.8, Figure 4d). Insight 2: Weight sharing super-net could act as good pretrained model. Finetuning child models with limited could greatly improve the quality of the rank.
4 UNDERSTANDING VARIANCE OF WEIGHT SHARING
Understanding the source of variance is the key to better leverage the power of weight sharing, In this section, we measure more numbers and design different experiments to understand where variance comes from and how to reduce the variance.
4.1 SOURCE OF VARIANCE
The first step is to find out the reasons why there is high variance in consecutive epochs of a single instance. We pick an instance from the "Diff. seeds" experiment. In this instance we evaluate the performance of the 64 child models on the validation set after every mini-batch near the end of training. The result is shown in Figure 5a The curves has obvious periodicity with the length of 64 mini-batches, i.e., the number of child models. Curves with light colors mainly located the upper of the figure, but they are not always the better ones. In some mini-batches the curves with darker colors perform better. In Figure 5| the i-th mini-bate trains child model c, we use diamond marker to label c's curve. We can see that in most of mini batches training child model makes this child model performs the best. Some bad-pes per--erforming child models can also become the best one by training them in mini-batches. It implies that training a child model can easily perturb the rank the previous mini-batch.
mini-batches
0.8
0.7
0.7
0.6
Aceursoy
0.5
Accuracy
0.6
0.4
0.5
0.3
0.2
0.3
39080
39100
39120
39140
39160 39180
39200
39080
39100
39120
39140
39160
39180
39200
Mini-butches
Mini-batches
(a) Ordered
(b) Shuffled
Figure 5: The validation performance of all child models, evaluated after each of the last 128 Each curve corresponds to one child model. Markers are marked on the child model trained in the current mini-batch. Lighter colors correspond to higher ground truth ranks. The difference between ordered and shuffled "shuffled" generate new order of child models every 64 mini-batches. Fach figure shows window of 128 mini-batches. It's clear that in Figure 5a a pattern repeated twice and the periodicity is mini-batches.
To further verify this phenomenon, we show the result of an instance with shuffled sequence of child models in Figiur 5b] There no periodicity, but other results are very similar. Though curves with light colors generally perform better than the other curves, it is still hard to tell which one of them is better than others. The instability rank during the last mini-batches in Figure 3) alaso implies the instability of GT-Tau, which means GT-Tau retrieved the end of the training can also be highly unreliable. In fact, for the instance shown in Figure 5a GT-Tau varies between 0.0 and 0.6 in the last 128 mini-batches.
In the rest of this section, we decrease the degree of weight sharing with different approaches. To faithfully reveal their effectiveness, calculate the average GT-Tau for instance in the last mini-batches due to the variance among mini-batee mini-batches. We call it CT-Tau-Mean-k To see the stability of GT-Tau, also obtain the standard variance of these & GT-Taus, which call GT-Tau-Std-k.
7

---

corresponds to Under review as a conference paper at ICLR 2020
Table 3: GT-Tau-Mean-64 and GT-Tau-Std-64, averaged over 10 instances. standard variance corresponding to the average.
The subscript is the
(a) Group Sharing
(b) Prefix Sharing
Grouping By Random
Grouping By Similarity
Mcan-64
Std-64
7 Mcan-64
Std-64
Mcan-64
Std-64
0.6960
0.012920.0060
0,4837
0.0939=0.0546
1.49884o.088 08.0
0.1497 0..020
+0.055
0.4159 0504
0.1925
2
0.4477=20.042
0.1371 20.02220023 0233
0.3425 0490
0.144220 0424
4
0.27364 0216
0.1340=2 0235
0.7075
0.0702
0.4448 0688
0.1881
+.1888=0.000m 000048
0.5174 03040
0.1592= =0.922000022
8
0.2539
0.1162:#0.0172
0.691
159177-t +0.00077
0.0457 0088
16
0 0316
0.1255
0.7200.aa.or
0.041 $..0000....
0.2903
0256
0.1104
0087
0.7360.1o.0m
0164
0,0364.10 0096
64
0.8032.0.00 0255
0.0151 10.0036
0036
0.8032.1.0225 0255
0.0151 1000036
4.2
GROUP SHARING: REDUCE VARIANCE
4.2.1
GROUPING BY RANDOM
Trying to reduce variance, divide the child models into groups, each of which is trained indepen- dently and maintains one copy of weights. We first, naively, randomly divide all the child models in search space into m groups. Therefore, for a search space of n child models, fully weight sharing and m corresponds to no weight sharing.
We conduct experiments on our search space of 64 child models. is chosen from 2, 4, 16, 32 and 64. For each m, we repeat the experiment for 10 instances, with the same group partition, but different seeds for initialization of weights. We run each group for 200 epochs and evaluate the validation accuracy of every child model at each of the last 64 mini-batches to obtain GT-Tau- -e--a--Mean- 64, GT-Tau-Std-4 64, and average them over instances, as shown Table 3a
Actually, breaking down the complexity through random grouping does not increase stability but actually fickfires. From m 16, the worst performing case, we take an instance for case study. We calculate GT--Tau-Mean-66 for each group, i.e, the including child models. The average GT--Tau- Mean-64 of the 16 groups is as low as 0.2570. To compare, We partition the ranks generated by an instance from way, the average GT-Tau-Ma -aaa-------44 is 0.5610 which much higher than 0.2570. Thus, we argue that the downgrading of GT-Tau the full rank mainly comes from internal instability inside groups. By examining the accuracy and rank of child models in each group, we find that interference among child models commonly exists in almost all the groups, even for 32 where there only child models per group. Such interference causes a drastic reordering of the rank of child models.
into those 16 groups, and calculate
-Maa-666 for each group in the same
0.80
0.70
0.65
Accuracy
0.75
0.60
0.55
0.70
143 (Rank 31)
143 (Rank 31)
(Rank 51)
0.45
(Rank 51)
0.65
344 (Rank 15)
0.40
344 (Rank 15)
424 (Rank
424 (Rank
0.35
39140
39150
39160
39170
39180
39190 39200
39080
39100
39120
39140
39160
39180 39200
Mini-batchos
Mini-batehes
(a) Group when m 16
(b) Group when m
Figure 6: Validation accuracy of group with 4 child models. Markers marked child models trained the current mini-batch.
To dive deeper, plot the accuracy of the child models from group in the instance of m 16, as shown in Figure 6al Some child models facilitate each other, while some others degrade each other. Specifically, the child models 143, 144 and 344 go up and down consistently, while the child model
8

---

Under review as a conference paper at ICLR 2020
424 acts exactly the opposite. Note that, 424 is the best-performing one in ground truth but performs the worst in this group, which indicates that properly choosing the child models for po-training (i.e., weight sharing) is the key to obtain good rank. This is further supported by FFigure 6b which shows the accuracy of these four child models when m = 1. With more other child models joining in for co-training, the four child models' curves become very different from that in Figure 6a For example, the curves of 344 and 424 become very similar.
On the other hand, from the first column Table 3a we can see that GT-Tau-Mean-64 first decreases then increases when m changes from 1 to 64. A possible explanation is that when many child models share a single copy of weights, a single child model cannot bias the group lot, while when each group becomes very small, the child models share less weights with each other, thus also not easy to bias each other too much. Observation I: Two child models have (higher or lower) interference with each other when they share weights. A child model's validation accuracy highly depends on the child models is jointly trained with.
4.2.2 GROUPING BY SIMILARITY
According to the observations above, we further explore how works by grouping child models based on similarity. We sort the child models lexicographically from 111 to 444, then slice the sequence into m groups. For example, when m = 8, group has the child models from 111 to 124, group 2 is from 131 to 144, group 3 from 211 to 224, and so on. The results are shown on the right ofbae 3a We can see that there is a global trend of stabilization from m = 1 to 64, i.e., GT-Tau-Mean- goes higher and GT-Tau-Std Std--44 gets lower. Observation 2: smart grouping can generally improve the stability of training.
In our case, "smart" means "similar". However, this might not be the case for more complex search space, where "similar" can be poorly defined, or the range of the space is too large, or even infinite. Admittedly, for larger and more complex search space, such smart grouping has to found by other means, e.g, correlation matrix among child models. We leave it in future work.
4.3 PREFIX SHARING
Inspired by the great success of transfer learning (Caruana 1995| Mesnil et al. 2011 Kornblith et al., 2019), we try to do a similar thing by sharing one copy of "backbone" network while keeping a separated copy of "head" for each child model. In particular, we use k to denote the number of cells shared, When l: = 0, only the first two conv layers are shared. When & = 4, all the layers except the final fully-connected layers are shared. In the experiments, increase the total epochs from 200 to 2000, as the models require more computation to reasonably converge.
The results are shown in Table 3b Obviously, sharing fewer cells improves the GT-Tau and accuracy (more experiment numbers can be seen in Appendix B). The performance becomes better but at the cost of consuming more computation power. For example, though a high and stable GT-Tau is obtained when & = O, it takes over 1000 epochs for it to climb up to above 0.6. But still, this cost is much lower than ground truth, which takes 64 x 200 12800 epochs total.
CONCLUSION
Neural architecture search becoming feasible way to explore new models, but its excessive computation cost impels researchers to resort to the power of weight sharing. In this paper we use comprehensive experiments to have a close look at weight sharing, and illustrate many interesting insights. By designing more sophisticated experiments, we further dig out the reasons of high variance of weight sharing. To further improve NAS, we believe the key is to figure out how to smartly leverage shared weights. This paper suggests controlling the degree of weight-sharing either model-based and rule-based, evaluating them on the small search space and providing deeper insights. We hope to inspire the community find more stable yet efficient approaches.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anonymous. Was-bench-Ishotl Benchmarking and dissecting one-shot neural architecture search. In Submitted to International Conference on Learning Representatios, 2020. URL https
//openreview net. wwwp://eeeeeoteteeeeeeeeeeeeaae www.rrmmmiissxx. d 9ngStPH
under review.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understanding and simplifying one-shot architecture search. International Conference on Machine Learning, pp. 549-558, 2018.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Rich Caruana. Learning many related tasks the same time with backpropagation. Advances in neural information processing systems, pp. 657-664, 1995.
Adam Gaier and David Ha. Weight agnostic neural networks. arXiv preprint 1rXiv:1906.04338. 2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. CoRR, 2019.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dumning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint 171 1..099966... 2017.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2661-2671, 2019.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. arXiv preprint arXiv: 1XX2::192200638 2019.
Jason Liang, Elliot Meyerson, and Risto Miikkulainen. Evolutionary architecture search for deep multitask networks. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 466-473. ACM, 2018.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 19 19-34, 2018a.
Chenxi Liu, Liang-Chicd Chich Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 82-92, 2019.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierar- chical representations for efficient architecture search. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April May 3, 2018, Conference Track Proceedings, 2018b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, arXiv preprint arXiv:1806.0905. 2018c
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. CoRR, bbs//608.03998. 2016.
10

---

Under review as a conference paper at ICLR 2020
Minh-Thang Luong, Quoe V Le. Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence to sequence learning. arXiv preprint arXiv: 2015.
Grégoire Mesnil, Yann Dauphin, Xavier Glorot, Salah Rifai, Yoshua Bengio, Ian Goodfellow, Erick Lavoie, Xavier Muller, Guillaume Desjardins, David Warde-Farley, et al. Unsupervised and transfer learning challenge: deep learning approach. In Proceedings of the 2011 International Conference on Unsupervised and Transfer Learning workshop-Vonl Volume 27, pp. 97-111. JMLR. org, 2011.
Risto Miikkulainen, Jason Zhi Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. Evolving deep neural networks. CoRR, abs/1703.00544 2017.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv: 000..002222222 2018.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. CNN features off-the-shelf: an astounding baseline for recognition. CoRR, 1401.6033, 2014.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017. pp. 2902-2911, 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. In The Thirty-Thrrd AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019., pp. 17780-789, 2019.
Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architccture search. CoRR, abs/ abs//902.088 142, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning. 8(3):229-2555 May 1992. doi: 10.1007/BF00992666
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. CoRR, abs/1811.03344,, 2018.
Lingxi Xie and Alan L. Yuille. Genetic CNN. CoRR,
2017.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. NAS-Bench- 101: Towards Reproducible Neural Architecture Search. arXiv e-prints, art. arXiv.1902.09635, Feb 2019.
Xinbang Zhang, Zehao Huang, and Naiyan Wang. You only search once: Single shot neural architecture search via direct sparse optimization. CoRR, abs/181 81 1.01567, 2018
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning, arXiv preprint 1xXiv:1611.0157.. 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018
11

---

Under review as a conference paper at ICLR 2020
A EXPERIMENT SETTINGS
A. OVERVIEW
Since our network architecture is very similar with that introduced in DARTS Liu (2018c). we basically follow the settings in DARTS, with few modifications.
1. Batch size: We following the settings from a Py Torch implementation of DARTS and set batch size to 256. The current batch size divides one epoch into 196
2. Number of epochs: Previous works Liu et al. (201cc) [i & Talwalkar (2019) all the number of epochs as a constant number. We follow the similar settings and set it to 200. Empirically, we experiment with epochs from 50 2000, though they show different final accuracies, all our Tau's reported in this paper seems insensitive to the number of epochs trained, except when the number of epochs too small for the instance reach plateau. See Appendix A.22 and Appendix B for more information.
3. Learning rate: Following DARTS, we use an initial learning rate 0.025, annealing down to 0.001 following a cosine schedule depending on the total number of epochs.
4. Optimizer: We use SGD as optimizer train graphs. Weight decay is set to be 10-3 as smaller networks often needs larger weight decays. The momentum is set to be 0.9, and the velocities (momentum buffer) for parameters shares if the corresponding parameter shares.
5. Network:
The first 2 conv layers of the entire network expand the 3-channel image into 48 channels. Each of the following cells will first compress the input into channels and feeds into the DAG and concat the output of nodes (32 channels in our settings). The momentum of batch normalization set to be 0.4 (under the semantics of PyTorch
For reproducibility, we set a certain seed before running all the experiments, and we removed the non-deterministic behavior in CuDNN.
A.2 EFFECTS HYPERPAMETERS
In order to justify results of this paper are not just any coincidence of carefully picked set of hyperparameters. We compare some of the results here with those with difference choices.
A.2.1 NUMBER OF EPOCHS
not To show that the phenomenon in Figure is not just a result of training too few epochs, repeat the same experiment with the same experiment settings, with number epochs set to 2000, obtaining Figure 7 which shows similar periodicity and instability, despite the overall accuracy is higher.
A.2.2 MOMENTUMS
Larger momentums help preserve the information from previous mini-batches, during which other child models are training, thus, presumably stabilize the training.
Batch norm Following the definitions in PyTorch, higher BN momentum indicates that the mean and variance in the current mini-batch have a higher weight. We compare accuracy curves for BN momentum is lower (0.1) and higher (0.9), each repeating times with different seeds for initializations. Experiments show that lower BN momentum helps stabilize the training in short
term, but it's still trembling in the long term, see Figure
SGD momentum We compare the results of accuracy curves when SGD momentum is set to 0 with 0.9. Results are shown as in Figure 9) The accuracy seems to vary in a greater range, compared to Figure 5a and GT-Tau varies between -0.1 and 0.5, which is more unstable.
https: /i thub. www htom//hanrc/pp... darts
The momentum here is different from that used in optimizers. Mathematialllllllllllll it's 2new = (1 momentum)
$ + momentum Reference: htsps://pytorch http:///www..oo///c//ss//aaaaaaa http:://gprrorrt...rr//////s///s _al_mmodull I weeeee//toooo tesstorc/////n/
12

---

Under review as a conference paper at ICLR 2020
0.8
0.7
0.6
Accuracy
0.5
0.3
0.2
391880
391900
391920
391940
391960
391980
392000
Mini-batches
Figure 7: Validation accuracy for all child models for each the last 128 epochs.
of 2000
0.7
0.7
0.6
GT-Tau
GT-Ta9
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.0
39080
39100
39120 39140 39160 39180
39200
39080
39100
39120
39140
39160 39180
39200
Mini-batches
Mini-batches
(a) BN momentum 0.1
(b) BN momentum 0.9
Figure 8: GT-Tau curves over last 128 mini-batches. Each color represents one instance.
0.7
0.6
0.5
Accuracy
0.4
0.3
0.2
0.1
39080
39100
39120
39140
39160
39180
39200
Mini-batches
Figure 9: Validation accuracy for all child models for each of the last 128 mini-batches, when SGD momentum is set to 0.
13

---

Under review as a conference paper at ICLR 2020
B PREFIX SHARING: DETAILS
Table Supplementary to Table 3
ET-Tau-Meant 64 and GT-Tau-Std-66 follow the same scheme. Similar these metries, accuracy of each child model in each instance is first averaged 64 mini-batches, and then mean and std of all child model accuracies an instance are calculated. Finally we average mean and std from each instance over 10 instances. The subscript is the standard variance of the corresponding average.
Epochs
GT-Tau-Mcan-64
GT-Tau-Std-66
Accuracy Mcan
Accuracy Std
2000
+o.960
0.0129 +0.0066
0.7793 0.0000 0.0036
0.0245+0.0010
2000
0.4837 4-00822
0.0939 00.004
0.7691 0000889
0.0378
2000
2
0.4159
0.1925
0.6972
0.0640
2000
0.4448
0.1881
0.6636=0.0177
0.0823=0.0118
2000
0.517440.0340
0.159221.01668
0.6639
0.099520.0166
200
40.144
0.0145
0.6083
0.0173
200
10.091
0.0989.10.0390
0, 0.6435_10.014
0.029810.0058
200
10.00010.00490
0.1801 10.0418
0.6035-10.0138
0.0535
200
3
1..554500.0581
0.1570
0.6081
0.0667 10.0066
200
0.4899+0.0501
0.1622:0000344
0.6177=0.0171
0.0800=0.0181
As shown in Table 4, in the case of 200 epochs, accuracy is much lower than that after 2000 epochs. Also, GT- shows an ascending trend, which quite the opposite of the phenomenon observed in larger epochs. We argue that this is due to the insufficient training for smaller k. Prefix sharing causes the networks to having multiple copies and these copies cannot be trained independently (as opposed parallelizing the groups in (4.2). Running 2000 epochs in prefix sharing is fair and necessary, with acceptable time cost (about 2 days on a GPU for one instance).
C
GROUND TRUTH LOOKUP TABLE
Shown in Table iss is a table containing accuracies of all 64 child models, along with their ranks.
14

---

Under review as a conference paper at ICLR 2020
Table 5: Accuracies of child models when they trained independently. The three digits of child model labels correspond to (o.1)s (0,2)) and ( respectively. Evaluations are repeated 10 times with different seeds of weight initialization, and averages and standard variances calculated. Ranks are based on the average values. The higher, the better.
Child Model
Ace Mean
Acc Std
Rank
Child Model
Ace Mean
Acc Std
Rank
111
70.13
0.23
64
311
81.78
0.20
53
112
78.23
0.75
62
312
83.52
0.25
27
113
81.06
0.42
57
313
83.31
0.34
32
114
79.98
0.40
314
83.48
0.26
121
77.77
0.37
63
321
82.91
0.25
41
122
81.77
0.50
54
322
84.24
0.22
123
83.10
0.21
39
323
84.33
0.29
13
124
82.58
0.43
49
324
84.62
0.19
131
81.14
0.21
56
331
82.91
0.22
42
132
83.15
0.36
38
332
83.98
0.13
133
83.21
0.28
34
333
84.70
0.53
10
134
82.77
0.19
48
334
84.47
0.41
12
141
79.62
1.04
60
341
83.39
0.18
29
142
82.86
0.17
46
342
84.32
0.51
14
143
82.87
0.31
43
343
84.23
0.35
19
144
82.07
0.38
51
344
84.32
0.49
15
211
78.67
0.49
411
80.43
0.73
212
81.86
0.37
52
412
82.94
0.44
40
213
83.31
0.28
413
82.87
0.19
214
82.81
0.30
47
414
83.21
0.53
34
221
81.42
0.15
55
421
83.27
0.21
222
83.20
0.30
36
422
84.76
0.56
223
84.22
0.31
20
423
84.73
0.33
224
84.17
0.32
22
424
84.77
0.57
231
82.87
0.44
45
431
83.39
0.25
30
232
83.81
0.43
26
432
84.80
0.42
233
84.28
0.28
16
433
84.84
0.57
234
84.19
0.50
434
85.23
0.17
241
83.20
0.31
37
441
82.19
0.26
50
242
84.28
0.14
17
442
84.87
0.59
243
84.03
0.19
24
443
84.93
0.23
244
84.08
0.36
23
444
84.85
0.35
15


Reviews:
[{"review_id": "ryxxwa_CYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies weight sharing in neural architecture search (NAS). It constructs a mini search space with 64 possible choices, and performs various comparisons and studies in an exhaustive way. Some of the observations are quite interesting, exploring the limitations of weight sharing.\n\nMy biggest concern is the limited search space. Unlike other NAS works that usually have search space size > 10^10, this paper focuses on a very small search space (64 options in total). Because the search space is so small, a small change in any search option might cause a big difference for the sampled model, which possibly lead to some of the instability observed in this paper (such as observation 3 in Section 3.2 and the implication \"training a child model can easily perturb the rank of the previous mini-batch in section 4.1). However, this might not be true if the search space is big, where changing a few search options may not affect the supernet significantly.\n\nIt would be great if the authors can perform similar study on a larger search space. If evaluation for large search space is difficult, you may consider some pre-defined accuracy lookup tables (such as NAS-Bench-101: https://arxiv.org/abs/1902.09635).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlzZfOtKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "First of all, I have to state that this is not my area of expertise. So, my review here is an educated guess. \n\nThe paper is an empirical study that looks into the effect of weight sharing in neural network architecture search. The basic idea is that by sharing weights among multiple candidate architectures, the training process can be significantly improved.  In the literature, there have been mixed results, either in favor of or against weight sharing. The question this paper aims to address is to determine if weight sharing is justifiable and to what extent. \n\nThe primary subject investigated by the authors is to determine if the variance of the ranks generated by different runs of the algorithm are highly correlated with each other (e.g. using Kendall rank correlation score). Then, they compared such results with the ground truth (i.e. every child is trained independently). They found that the ranks generated by weight sharing are indeed highly correlated with each other, but there is much larger variance in the ranks when compared to the ground truth method. To understand why this is non-trivial: (1) on one hand, weight sharing speeds up the training process by providing an initial point close to a local minimum, but (2) the local minimum point may or may not  be good for the new architecture.  Hence, one does not know apriori under what conditions would weight sharing be a good strategy. \n\nThe authors also looked into variance of the rank within the same instance by examining how the rank changes with mini-batch epochs. They found that variance is large even within the same instance. \n\nMy primary concern is that the paper is entirely empirical with little if any justification of the results. In addition, it is based on a single architecture and a single dataset. This would have been fine if the results were supported with explanation or theoretical justification. Second, the ultimate goal is to improve the prediction accuracy, not the ranking accuracy. These are not necessarily equivalent. For instance, it is possible that the ranks have a high variance simply because many of the candidate architectures have nearly equivalent performance so the order within them becomes nearly random (and unimportant). In fact, I think the results support this conclusion (see for example Figure 10). Third, some of the highlighted observations are trivial. For example, Observation 1, which states that \"Two child models have (higher or lower) interference with each other when they share weights. A child model’s validation accuracy highly depends on the child models it is jointly trained with.\" I think this observation is trivial. \n\nSome other comments:\n- I would appreciate it if the authors could explain briefly how \"prefix sharing\" works so that the paper is self-contained. \n- The goal is to help improve the speed of neural architecture search. The authors mention \"hints for designing more efficient weight-sharing.\" Please state those conclusions precisely and clearly. I understand that the authors suggest similarity-based grouping. So, please mention clearly what you recommend in the conclusion section. \n\n\n========================== \n#post rebuttal remarks\n\nThanks for the response. Figure 10 was a typo from my end and I apologize for it. I actually meant figure 3. \n\nAs I said in my review, having an empirical study is acceptable provided that it covers many datasets, not just a single one. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byx_AfDsdr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Many NAS methods rely on weight sharing. Notably in ENAS, a single weight tensor is used for all candidate operations each edge of a cell. In this paper, the authors take a small NAS search space (64 possible networks) and train each network separately to obtain their individual rankings. They then examine how this ranking correlates when  the same network is trained as part of a super-net with weight sharing, as in NAS algorithms.\n\nGiven the prevalence of NAS algorithms, an examination of the potential pitfalls of weight sharing is very important and I commend the authors for that. There are a lack of typos and grammatical errors, which is nice too!\n\nI have two major issues with this paper however. Firstly, the scope is limited; everything is based on looking at 64 convnets trained on CIFAR-10, this makes it tricky to make any broad statements about weight-sharing in NAS. Secondly, the paper reads as a string of observations, and it is not clear what the takeaways are (it is hinted that one could reduce the search space for Figure 3, but this is not expanded on).\n\nA few chronological comments:\n\n- \"Population based algorithm () is another popular approach\" ---> \"Population-based algorithms are another popular approach\"\n\n- \"In this paper we try to answer\" --> A little confidence wouldn't hurt :)\n\n- \"Surprisingly\". I don't like the reader being told what is surprising/interesting etc. but maybe that's just me.\n\n- As mentioned above, more detail on search space pruning would be really nice. For instance, are particular operations e.g. conv5x5 neglected.\n\n- To clarify, when you have shared weights, but one candidate operation uses more parameters than another, do you just read off the weight tensor until you hit length? e.g. if convA uses X params, and convB uses Y params, are the parameters for ConvA weight(1:X) and convB weight(1:Y)?\n\n- Literature review is good. Figure 1 is nice and straightforward.\n\n- The figure and table captions could do with some more detail. For instance, in Figure 2 the caption should contain the take-home point of the figure.\n\n- \"there are some statistic information\" --> \"there is some statistical information\"\n\n- Figure 3 is nice. It looks like you can't tell what's good, but you can tell what's bad. A comparison of what architectures good v bad comprise off would be a nice addition.\n\n- Figure 5 confused me, as there is a lot going on. What do ordered and shuffled mean? Is it just whether you are mixing up your minibatch selections? \"The curve has obvious periodicity with the length of 64 mini-batches i.e. the number of child model\" doesn't make sense to me. Could you elaborate?\n\n- The accuracies in the plots look very low. ~80% for CIFAR-10 is really bad. Am I missing something?\n\nPros\n------\n- Good topic with a few interesting observations\n- Relatively well-written\n\nCons\n-------\n- Very limited scope. Only 1 dataset and only 64 models\n- The narrative is lacking, what are the key points that people using NAS should be aware of?\n\nI recommend a weak rejection for this paper. The topic is interesting, but I haven't been convinced through the limited scope of the experiments, or the arguments made what the real point is. Should I stop weight sharing with NAS? Should I prune my search space? etc. A few neat observations is nice, but there is a lack of cohesion. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a series of empirical evaluations on a small neural architecture search space with 64 architectures. The experiments are interesting, but limited in scope and limited to 64 architectures trained on CIFAR-10. It is unclear whether lessons learned on this search space would transfer to large search spaces. One upside is that code is available, making the work reproducible.

All reviewers read the rebuttal and participated in the private discussion of reviewers and AC, but none of them changed their mind. All gave a weak rejection score.

I agree with this assessment and therefore recommend rejection.
Decision: Reject