Exploring single-path Architecture Search ranking correlations
Authors: Kevin Alexander Laube
Andreas Zell
Venue:  (2021)
Keywords: Neural Architecture Search
AutoML
Neural Networks

Abstract: Recently presented benchmarks for Neural Architecture Search (NAS) provide the results of training thousands of different architectures in a specific search space, thus enabling the fair and rapid comparison of different methods.
Based on these results, we quantify the ranking correlations of single-path architecture search methods
in different search space subsets and under several training variations;
studying their impact on the expected search results.
The experiments support the few-shot approach and Linear Transformers,
provide evidence against disabling cell topology sharing during the training phase or using strong regularization in the NAS-Bench-201 search space,
and show the necessity of further research regarding super-network size and path sampling strategies.

Under review as a conference paper at ICLR 2021
EXPLORING SINGLE-PATH ARCHITECTURE SEARCH RANKING CORRELATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Recently presented benchmarks for Neural Architecture Search (NAS) provide the results of training thousands of different architectures in specific search space. thus enabling the fair and rapid comparison of different methods. Based on these results, we quantify the ranking correlations of single-path architecture search methods in different search space subsets and under several training variations; studying their impact on the expected search results. The experiments support the few-shot approach and Linear Transformers, provide evidence against disabling cell topology sharing during the training phase or using regularization and other common training additions in the find that network size and path sampling strategies require further research to be understood better.
search space. Additionally, we
INTRODUCTION
The development and study of algorithms that automatically design neural networks, Neural Archi- tecture Search (NAS), has become significant influence in recent years; owed to the promise of creating better models with less human effort and in shorter time.
Whereas the first generations of algorithms required training thousands of networks in thousands of GPU hours using reinforcement learning (Zoph & Le| (2016) Zoph et al. (2018)), greedy progressive optimization (Liu et al. (2018a),, regularized evolution (Real et al. (2018)) and more, the invention of weight sharing during search (Pham et al. (2018)) reduced the computation cost to few GPU hours, and thus made NAS accessible to much wider audience.
While this also enables gradient based NAS (Liu et al. (2018))) the necessity to compare operations against each other leads to an increased memory requirement. The issue commonly alleviated by training a small search network consisting of cells with shared topology, later scaling the resulting architecture up by adding more cells and increasing the number channels. Although the stan- dalone network is often trained from scratch, reusing the search network weights can increase both training speed and final accuracy Yan et al.] (2019): Hu et al. (2020). More recent gradient based methods require have only one path in memory (Dong & Yang (2019; Cai al. (2019): Hu al. 2020)) and can even be applied directly to huge data sets.
However, the aforementioned weight sharing methods only yield a single result, require manually fine-tuning the loss function when there are multiple objectives, and can not guarantee results within constraints (e.g. latency, FLOPs). The single-path one-shot approach seeks combine the best of both worlds, requiring only one additional step in the search phase (Guo et al. (2020)); Firstly model (super-network)
a full sized weight-sharin the available operations at each layer in every training step. Then, specific architectures can be evaluated by choosing the model's operations accordingly, a hyper-parameter optimization method can be used to find combinations of operations maximizing the super-network accuracy. If the rankings of the architectures by their respective super-network accuracy and by their stand-alone model retraining results consistent, the quality of the discovered candidates is high.
fully trained by randomly choosing one of
However, since the single-pane paghe--pthhheee method's search spaces are often gigantic and the network training costly (see e.g. Guo et al. 22000; Chu al. (2019b a)), study of the ranking correlation is usually limited to a handful of architectures. In this work we study the single-path, one-shot super-network predictions and ranking correlation throughout entire search space, all stand-alon model

---

Under review as a conference paper at ICLR 2021
sults are known in advance. This enables us quantify the effects of several super-netwond training variations and search space subsets, to gain further insights on the popular single-pate one-shot method itself.
We briefly list the closest related work in Section 2 and introduce the measurement metric, bench- mark dataset, super-network training and experiment design in Section 3 We then systematically evaluate several variations in the single-patis one-shot approach with novel method, computing the ranking correlation of the trained super-networks with the ground-truth top-N best architectures.
Experiments on search space subsets in Section 4.1 once again demonstrate that the ranking is more difficult the search space increases size, and that the operations that make the ranking especially hard are Zero and Pool. Section 4.2 evaluates Linear Transformers (Chu et al. (2019a)), which we find perform very well in specific search space subsets, and otherwise even harmful.
Furthermore, some commonly used training variations such as learning rate warmup, gradient clip- ping, data augmentation and regularization are evaluated Section |.3 where we find that none of these provides measurable improvement. We further test disabling cell topology sharing only dur- ing training time and find that training the network in the same way as evaluating it more effective.
We finally list some grains of salt in Section 5 and conclude the paper with Section
2 RELATED WORK
A high quality architecture ranking prediction is the foundation of any NAS algorithm. In this paper we explore the effects of several super-network training variations on the ranking predic- tion of the aforementioned single-path one-shot approach (Guo et al. (2020)). Recent efforts have shown improvements by strictly fair operation sampling in the super-network training phase (Chu al. (2019b) and adding a linear 1x convolution to skip connections, improving training stability (Chu et al. (22099)). Other works divide the search space, exploring multiple models with different operation-subee -ubsetts (Zhao et al. (2020), ) or one model with several smaller blocks that use a trained teacher as guiding signal (Li et al. 2020b)).
Due to the often gigantic search spaces and the inherent randomness network training and hyper- parameter optimization algorithms, the reproducibility of NAS methods has become major cern. NAS Benchmarks attempt to alleviate this issue by providing statistics (e.g. validation loss, ac- euracy and latency) of several thousand different networks on multiple data sets (Ying et al. (2019);
Dong & Yang 2020 providing the ground-trutt training results that we use for our evaluation.
METHOD
3.1 METRIC
As we correlate the super-network accuracy prediction and the benchmark results, but are only interested in a correct ranking, we need ranking correlation metric. We choose Kendall's Tau (r, KT), commonly used ranking metric (Sciuto et al. (009); Chu et al. (2019h)) that counts how often all pairs of observations (æ,, y/) and (Z))
1. are concordant, agreeing on a sorting order (x% < æy and y/ yj: D; > Iy and y; > yj)
2. are discordant, disagreeing on a sorting order (x) < æi and y/ yj: wi >> Iy and y; yj)
3. are neither
and is then calculated by their difference and normalized by the number of possible different pairs.
(num concordant)- (num discordent)
ranges from -1 in perfect disagreement to +1 in perfect agreement, and is around zero for inde- pendent X and Y.
A small selection of experiments that additional metrics be found in Appendix D
2

---

Under review as a conference paper at ICLR 2021
$1.01
55.95
$9.02
03.93
01.04
0.00
130
3/0
11:01
-BotDaale and
1eul3 vile ast 0..23
PA390
30.83
95015
MOLL
$5.90
Accuracy predictions of a small super-network axis) and Bench201 results (y axis, in percent) for the top-50, top-250 (left and center columns) and 1000 randomly sampled (right column) benchmark networks on the cifar10-valie data set, ordered by accuracy. While all five operations are available in the top row, Zero is masked in the bottom row.
Figure 2:
3.2 NAS-BENCH-201
NAS-Bench-201 Dong & Yang 22200) is a tabular benchmark, which contains training and evalua- tion statistics of 15625 different architectures on the common vision data sets CIFARTÓ, 1O, CIFAR100 100 Krizhevsky et al. (2009) and a reduced variant of ImageNet (Deng et al. (2009). ). The models differ in the design of the cell, a building block that is stacked several times to create a network. Within the cell, visualized in Figure T at six specific positions (orange edges), one of five operations (Zero, Skip, 1 xx1 Convolution, 3x3 Convolution, 3x3 Average Pooling) is chosen (5 = 15625). The inputs of each node, such as the cell output (rightmost node) averaged.
As we are only interested in the final accuracy of each architecture. average the benchmark test results over all seeds and the last three epochs training. As the models' rankings are quite con- sistent across all data sets (Dong & Yang] (2020)), we focus on the CIFAR- (aldd accuracy. Further results are provided in the sup- plementary material, see Appendix
Since discrepancies of model rankings for the top performing ar- chitectures became apparent (Dong & Yang (2020)), we measure the accuracy of the trained super-networks according to the top-N (10, 25, 50, 150, 250, 500) benchmark architectures, as well as up to 1000 randomly sampled ones. If a reduced search space (due to masking operations, 3 = topologies, it is fully evaluated.
Figure I: A NAS-Bench-201 cell with operations the ange edges
729) contains fewer than 1000 different
3.3 TRAINING
In our experiments we train various NAS-Bench-201 networks. Small variants have 2 cells per stage (total of 8 cells, with 3 stages and fixed cells for spatial reduction) and 32 channels in the first cell, which is roughly similar common topology sharing methods. Medium sized networks have cells per stage and start with 64 channels.
All models were subject to the same training schedule. We used CIFARIO as training set Krizhevsky et al. (2009)), of which we withheld 5000 images for validation. The batch size is 256, we used SGD with momentum of 0.9 and learning rate of 0.025, which was cosine annealed to le-5 over 250 epochs. All results are averaged over five independent runs with different seeds. Further details are listed in Appendix A
3

---

Under review as a conference paper at ICLR 2021
3.4 EXPERIMENT DESIGN
All of the following experiments are structured the same way: The top-N network architectures (ordered by topl accuracy, measured in (AS-Bench-201) are selected, and an over-complete super- network predicts their respective accuracy values, as seen in Figure 2 If an operation is not available to the super-network, the top-N networks are also taken from the bench results without that opera- tion.
Variations the search space and the super-network (structure or training process) affect the ranking correlation т between the bench results and the uper-network, predictions. In the case of Figure removing the Zero operation from the search space improves т.
To make the figures more compact, the exact benchmark and prediction values are ignored in the further figures, only average prediction accuracy and т depending on N will be shown (see Appendix [ for further detailed figures), seen in e.g. Figure 4 We also add the additional metric To which describes the ranking correlation of the average prediction accuracy depending on N. More formally,
is computed as described in Section 3.1 on the series of measurements [100, A1o): (25, A2s). (50, Asu)....] where AN is the accuracy of super-network M with topology T, and weights 0g the validation data Dvalid, averaged over the top-N topologies and multiple seeds.
1
Av
M,, Tú, Dvalsd)
seeds
&Eseeds
As we increase N (10, 25, 50, ...) AN should monotonically decrease (e.g. 0.7, 0.65, 0.6, ...). so that Ta 1 is the case where the super-network estimates match the bench results best.
4
EXPERIMENTS
4.1
SEARCH SPACE SUBSETS
The search space itself plays a significant role for NAS algorithms, not only due to its size or the availability of good models. By preventing specific operations from being used (masking) during training. validation and in the benchmark results, we can compare different subsets of the search space.
visual example of the importance can be found in Figure where for an increasing number of top-N benchmark networks (columns). the ranking correlation the super-network predictions is steadily improving. While there is large number of networks wrongly predicted as useless (10% accuracy, right column) the top row, masking the Zero operation (bottom row) significantly reduces this portion and thus improves the ranking correlation T (KT).
To get a deeper understanding of the search space subsets, we take closer look at the NAS-Bench- 201 rankings, sorted by their top 1 accuracy. Specifically, in Figure 3 we count how many of the NAS-Bench top-N networks use each available operation and how often it used, across all bench- mark results and the five largest subsets (each operation is masked once).
The 3x3 Convolution is arguably the most important operation. Unless it is masked, every single top-500 Bench network makes use of it. On average, it even makes up roughly half of the operation choices, in every other subset. This is hardly surprising, since it adds significantly more capacity to the network than the 1x1 Convolution and especially Zero, Skip, Pool.
The order of operation importance, as implied in their usage, continues with Skip. 1x1 Convolution, Zero, and finally Average Pooling. The wide usage of Skip operations was be expected, they known to make deep networks more easily trainable (He et al. (2016)), however they are not present in every network. Perhaps the most surprising is the low importance of Average Pooling, even lower than Zero appears that all the benefits Pool are already covered by the 3x3 Convolution, so that using the unnecessary operation now decreases the network accuracy.
Two subsets behave notably different than the full search space. Firstly, in the absence of skip con- nections (top right), appears that Average Pooling is used substitute. And secondly, in the
4

---

Under review as a conference paper at ICLR 2021
Dovedea
0.00
0100
0.4
0.4
044
000
100
150
T1p X
1op hest
Tred crteels (10p)(
0.35
0.35
0.00
0.00
0.00
1.4
0.4
Opeeden
beach antovkke
beash
operation at any position in the architecture (top plot in each of the two rows and three columns) and their total usage (each bottom plot) over the top-N networks and all possible architectures (x axis). The operations are Zero (green), Skip (blue). 1x1 I Conv (orange), 3x3 Conv (cyan) and 3 xx3 Avg Pool (purple). One operation is masked in each of the six pairs, except for the top left one. As an example, if of the top- networks use at least one Pool operation, together total of 4 Pool operations, and have all 5 operations available, the usage 3/10 0.3 while the share 4/(5 10) 0.08.
Figure 3: Counting how many top-N Bench architectures
absence of the 3 3x3 Convolution (bottom center), Average Poolings and especially 1x1 Convolu- tions have to make up for the missing capacity and spatial operations.
We now train single-path super-networks in several search space subsets and visualize the results in Figure 4 Ideally, the super-network validation accuracy is highest for the top Bench networks, enabling NAS methods to reliably find them, and the ranking correlation within the top-N bench networks is always significantly greater than zero, thus increasing the expected quality of the se- lected architecture. Neither is the common case. The baseline for small networks (top left, red) has the same averaged prediction accuracy for the top 10 as for the top 500 networks, resulting in To 0, where the predicted accuracy and the bench accuracy have statistical correlation.
However, in some search space subsets, the single-path method works significantly better. By mask- ing each operation individually (Figure 4 left column), we find the most harmful operations to be Zero (green, top left, --=-1 1) and Pool (purple, bottom left, T==-1), 1), which are also the least im- portant ones according to Figure 3 Masking the Convolutions, thus increasing the relative amount of unparameterized operations, is harmful.
Masking Skip (blue, left) is the most harmful to Ta (=1). As seen in Figure the top-N networks have a worse average predicted accuracy than the top-M (for N < M) networks, and sometimes even below the random sample, which is terrible. Interestingly 7 may improve within the predictions for the top-N architectures.
We further mask second operation in addition to Zero (center column) and Pool (right column) in the remaining columns of Figure 4 On small networks, the masking combination of Zero+Pool and arguably Zero+Skip perform even better, while masking Pool in almost any combination harmful.
It is quite obvious that medium sized super-networks require additional care. The super-networks in several search space subsets fail to generalize at all, even though they learn the training set. In the other spaces, they still behave differently. This may be beneficial, such as in the baseline (left, red). but is more often harmful. Even worse, the averaged predicted accuracy of top-N networks in several subsets is lower than that of random subset of networks, despite the often improved
5

---

Under review as a conference paper at ICLR 2021
05
-005
-005
100
160
100 maliem
100
100
300 merion
10.
160
953
Tip
Top
110
017
1.0
raolo [Ror, Chel)
(Skp Phel]]
made- 12a,, Choa)
raall (Phn,, Fese]
Go 1.00
Siadka/CDG Poul]
S-
-0.5
100 natoo
201
neaker
109
290
borhh teteoels
Top setverte
made- Bem, Chail
1-m
M.
CDid))
5u0
M. Mamnk-//Po (Pod))
5-2000
roak--/hee Pla))
C- 0.30
Measuring ranking correlations and average accuracy (y axis) of the top N bench networks (x axis) in different search spaces (color), plotting the mean and std of five runs. We used small sized (S) super-networks in the top row and medium (M) sized ones in the bottom row.
Figure 4:
values. Finally, as seen in Figure the standard deviation over the super-networks that do generalize is notably greater than for their small sized counterparts.
4.2 LINEAR TRANSFORMERS
Many common search spaces for single-path methods contain exclusively Convolution operations (or blocks of such). Adding a Skip operation is useful in theory, enabling the discovery of smaller sized networks, but was also found to impact the stability of the super-network training. After all, in a sequential super-network, the operations at any layer may directly receive the output of any previous layer due to the variable size. However, replacing the Skip operation with linear 1xl Convolution (Linear Transformer, LT) during the search phase was found to stabilize the training (Chu et al. (2019a)). All Linear Transformers are removed after the search, resulting a standalone network with the same capacity.
Figure 3 visualizes the results of super-networks that have Linear Transformers added their Skip or Skip+Pool operations. We also mask Zero (center) and Zero+Pool (right) to observe the super- network in search spaces with fewer operations that are neither Convolution nor Skip. It is notewor- thy that in the absence of the Pool operation (right), both variations of the standard super-network are in fact equal. This is also apparent in the plot, although the randomness of non-deterministic training still be seen.
We find it an interesting observation that, unless the search space contains exclusively Skip or Con- volutions (the context in which the Transformers have been proposed), the Transformers are always harmful to the ranking correlation 7.. The neter-networks seem to overestimate the benefit of Skip and Pool operations, which, again, often causes the best networks to be estimated as below average. However, in a fitting search space, even the medium sized super-networks generalize very success- fully, have very low standard deviation and reliably improve Ta-

---

Under review as a conference paper at ICLR 2021
0.2
82 -1. Thy
22
227
-0:0
25
100
100
203
500
30
100
150
203
500
25
100
100
203
500
mecoon
Top boach enturiks
Top. Towch seturals
Toge Tawc enturiks
100
5. S.mndd-:2 Zem Podl
(imnda-/2ed (1-(Skp))
Sttakks Pool) LTugSkp)
wile To
03-
100
1000
739
saturdd
31
---2
3., soaainn-ee (2ko))
223
- 34, muai (Baos Pook)
M. 17-p(aap)
2.0
(Xro)) 1T-(Stp))
2.0
31 (ean Pool) 17-cStp)
M. (7dd) Paa)
Minnse-(Bee) Ui Peal)
M. maak (Jox Poo), 1i- Pool)
Figure 5: Adding Linear Transformers (LT) to Skip and Skip+Pool (color).
Finally, are Linear Transformers for Pool operation useful? Since the search spaces that suit single- path networks best do not contain any Pool operation, they are generally not necessary. When a Pool operation is present, as in Figure left and center-left, we find empirical evidence that they beneficial. Although the additional transformers seem to stabilize training, as seen by the lower standard deviation, they also worsen the Ta problem.
4.3 SHARING, SAMPLING, WARM-UP, AND REGULARIZATION
Finally, we group four further variations to super-network training in Figure with the baseline (red).
6 and compare them
Topology sharing (green): As sharing cell topologies does not impact the resource costs of single- path training, it generally not used. However, they shared in the NAS-Bench-201 case, raising the question whether sharing should already be enforced during the super-net training (our default case), or only for the evaluation.
As seen in Figure 6 disabling the sharing during training for small super-networks (top row) is generally not beneficial over the baseline, as Ta is generally worse and r almost the same. However, it enables the medium sized networks in multiple spaces to make any useful predictions at all.
Uniform sampling (blue): Additionally, our default baseline strategy of randomly selecting the paths during training is strictly fair, so that every |O] steps, every operation E O is sampled exactly once; and compare this with the alternative of uniform random sampling.
Interestingly, the absolute validation accuracy value is increased by uniform sampling. However, this is not relevant, as only the correct ranking matters. We find that, on small super-networks, as measured by Ta. the strictly fair baseline performs equal or better than the uniform random sampling strategy. Additionally, we see a trend of being slightly in favor strictly fair randomness, at almost every data point. However, once again, seemingly inferior method variation enables train- ing the medium sized nuter-networks to make above-chance predictions on the validation set. We hypothesize this be downside of the strictly fair weight update schedule, in which update is

---

Under review as a conference paper at ICLR 2021
RT (Troili Tee
witte
-020
100
230
80
nankez
10
20
100
130
500
10
22
150
120
230
200
nindes
Tap. lands adterds
Tip lead sateao
3. (Pee,, (Ie)).
cemplig
4- (Ree, (lid), n complin
5- 0.5
5. modi. (2an0) woraep
200000
- 8. 3. (Bem. Chal). vantey
-pn 5 ander (2mo)) up clpedia
-8- 5. (Bem Clill. uni diiskg
0.00
8. 3, Clad), neslatad
ontaa Tee
30-
-14
300
-301
725
31
3. remb (Bm))
5- 0.5
31. romd- (8mm Clsi)
5. (.80
-0 net duaed
1- 0.21
-8c 1. mack (loe) rot dusee
So 011
-8 31. 1am.mmmm.......m......... Claz duaed
5 0.80
M. susplem
3. mal (am)) sespdng
monk-1Bmm.. Cladis sesplex
0,25
M. cureey
0.0
mak-
(20m,, Chall,
0.25
St gat chyiing
*.- St meskstied
(Zeo)
waw.-peee CIel)
Figure 6: Further variations to the super-network training (color).
only performed every |O] steps (number of operations) over the accumulated gradients, including for the weights of stem and output layer, which may result destructively large steps.
Learning rate warm-up (yellow) and gradient clipping (cyan): To see whether simple learning rate tweaking already solves the aforementioned issue, we add a warm-up phase, linearly increasing the learning rate over 5 epochs to the default starting value of 0.025. It does not. In fact, the effects are detrimental in some search space subsets. On other hand, clipping the gradients so that their L2 norm is in [-5, 5], common practice in e.g. DARTS (Liu et al. (2018b)), has notable effect.
Further training variations to solve the issue with fairly little effort may be excluding the last layer and stem weights from the |Stptt update schedule (but losing strict fairness) lowering their learning rate. However, a much closer look at it seems preferable, to ascertain the root cause and study its implications in greater detail.
Regularization (purple): Finally, the input shifting, horizontal flipping, and normalizing the data), add the CIFAR-10 AutoAug- ment augmentation policies (Cubuk et al. (2018)) and label smoothing of a = 0.1.
is only minimally regularized by default (only
Interestingly, this is also detrimental. As in the top left and center-right plots, Ta decreases below the baseline values, as top ranking architectures are underestimated. The effect is worse with relatively more unparameterized operations available, indicating that the topology estimation is biased in favor of the regularized Convolutions. The effect on medium sized super-networks can be properly measured, as none of these super-network should be used to rank architectures.
8

---

Under review as a conference paper at ICLR 2021
5 GRAINS OF SALT
As in any empirical study, some grains of salt remain. First and foremost, the limited sample sizes in our experiments and the benchmark are a typical concern.
We find it disappointing that, aside from limited search spaces, no experiment displayed high values of T, even though the top-N network groups may be sorted correctly (Ta & -1). This indicates that number of good networks are always wrongly estimated, quite likely due to some of the available operations, or that the single-path one-shot approach is simply not suitable for the given search space network architecture. While the accuracy difference between the best and 10th-best Bench networks is only roughly 0.013%, masking Skip at least increases that difference to roughly 0.027%, possibly also making correct ranking easier. Considering these marginal differences, is very likely that the Bench baseline is also not perfectly correct.
Next, existing NAS benchmarks may be too small biased. Other research has discovered sur- prisingly simple methods that achieve state of the art results (c.g. Mellor et al. (2020); White et al. (220)), ) but suffer from an often significantly reduced performance larger search spaces. This is hardly surprising, considering that the best architectures consist of mostly 3x3 Convolutions (see Figure
And finally, our experiments use single-path one-shot methods, which are commonly employed in search spaces of only Convolution and Skip operations. They are our approach of choice due to their current popularity in the NAS field and the comparably cheap evaluation of many network topologies, which also enables us to study more variations the baseline method.
CONCLUSIONS
Some search space subsets are easier to rank. In this specific case, the removal of the Zero and Pool operations keeps the majority of the top-N networks while also improves how well single-path network can rank them.
Linear Transformers useful when there enable medium sized super-networks to be used at all. However they introduce systematic ranking problems in other search space subsets, limiting their general use. We find evidence that Pool operations with transformers are beneficial.
other operations besides Convolutions Skip. and
Disabling cell topology sharing during the super-network training decreases the ranking correlation Ta, the network should be trained the same way as it is evaluated.
Strictly fair randomness is generally advantageous, but requires further research be understood better. Especially several medium sized super-networks were unable to generalize, in contrast to those trained with uniform sampling, which believe be due the weight updates of the last layer and the stem. Simply adding learning rate warm-up or gradient clipping is insufficient to fix this issue.
Strong regularization during the super-network training was found to be detrimental. This is most likely an issue of the regularization only benefitting Convolutions, biasing the topology estimation, and may not be problem in entirely different search spaces that use fewer to no unparameterized operations.
Whether an increased super-network size is helpful is tricky to evaluate. In few search spaces, e.g. as seen in Figure 4 the increased network size improved Tas however the generally low validation accuracy (usually worse, the super-networks may fail to generalize at all depending on the search space. In specific cases this can be alleviated with Linear Transformers, and possibly through better understanding of path sampling. We present additional Figures in Appendix[D D
20%, on 10 classes) and its huge variance make them too unreliable. Even
Due to the limited space, have only shown the results for the CIFAR- 10-R--10-aall 10-valid Bench accuracy values. However, we provide all Tensorboard (Abadi et al. (2016)) files and the code to parse and generate plots in the supplementary material.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, lan Goodfellow, An- drew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Mur- ray, Chris Olah, Mike Schuster, Jonathon Shiens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete War- den, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large- Scale Machine Learning on Heterogeneous Systems. In 12th [USENIX] symposimm on op- erating systems design and implementation ((OSDI) 16), pp. 265 285, 2016. URL URL.httss: 7/ 7/www. tewsor www.orrlooo.oo. orgg Software available from tensorflow.org.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. In International Conference on Learning Representations, 2019. URL
https://arxx: /ttp://///aaxiiooo////////////// /181//111122 1812.00332222 pdd
Xiangxiang Chu, Bo Zhang, Jixiang Li, Qingyuan Li, and Ruijun Xu. SCARLET-NAS: Bridg- ing the gap Between Scalability and Fairness in Neural Architecture Search. arXiv preprint arXiv: arXiv:/908. 198800000222 2019a. htl.hhtts://aaae htt.:t/ps.//arvv..orrr//////////
06022. pd
Xiangxiang Chu, Bo Zhang. Ruijun Xu, and Jixiang Li. FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search. arXiv preprint arXiv: 2019b. URL
https://arxi:s wwtp://gg/pppptte 2tt.///997.0114555 01845.pdF
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. AutoAugment: Learning Augmentation Policies from Data, 2018. URL httt:://arxiv..orrabbb//// 09501
J. Deng, W. Dong, R. Socher, L.JJ. Li, K. Li, and I.. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR09, 2009.
Xuanyi Dong and Yi Yang. Searching for A Robust Neural Architecture in Four GPU Hours. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1761-1770, 2019, httpht/p//aaxiii wwwp:iiiirrrrrraao abs / 1910. 04465
Xuanyi Dong and Yi Yang. NAS-Bench-201: Extending the Scope of Reproducible Neural Archi- tecture Search. In International Conference Learning Representations (ICLR), 2020. URL
https:://ppeneeval
net forum? Ldt.oooodddd-ddyaaaaaaa Hid-H_xyyHHKKEE
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Sin- gle Path One-Shot Neural Architecture Search with Uniform Sampling. In European Conference on Computer Vision, pp. 544-560. Springer, 2020. URL http::/ http:///axxi...// org/ abs/1909, 00420
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. DSNAS: Direct Neural Architecture Search without Parameter Retraining. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22084-12092 2020.
URL http::/aaxiv..rrtt./ abs/ /ttp//202...0 09128
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 (Canadian Institute for Advanced Research). 2009. URL http: htttp::/www... http::///www.....rottt.tot.to wtt.t/oooo....ooiiiieeeetttieeee
html
Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning. In European Conference on Computer Vision, pp. 639-654. Springer, 2020a. URL hollhttps://aara httt:::://aaxi.oooo/aab///////// wtt.:o/g///bb///0../2222222
10

---

Under review as a conference paper at ICLR 2021
Changlin Li, Jiefeng Peng, Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xiao- jun Chang. Blockwisely Supervised Neural Architecture Search with Knowledge Distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp-
1989-1998, 2020b. URL http:/pp://aaww.or///abbbb
13053)
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fej-Fei, Alan Yuille, Jonathan Huang. and Kevin Murphy. Progressive Neural Architecture Search. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pp. 19-34, 2018a. URL
htttp://arxis http:/iww.or///b/////222// .00599
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable Architecture Search,
2018b. URL http:////aaxx wwwpg//a.//......../. 09055
Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J. Crowley. Neural Architecture Search with- out Training, 2020. URL http:://aaxxv..orr/aa/////000000 0444
Hicu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Lc, and Jeff Dean. Efficient Neural Architecture Search via Parameter Sharing, 2018. URL/ttter//axxiv..orraaa/////
03268
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized Evolution for Image Classifier Architecture Search, 2018. URL htt.::tp///aaa...rr//a//////////
01548
Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the Search Phase of Neural Architecture Search. CoRR, abs/1902.08142, 2019. URL http:
//arxiv.org/ abs/ 1 902.08144 08142
Colin White, Sam Nolen, and Yash Savani. Local Search is State of the Art for Neural Architecture Search Benchmarks, 2020. URL http:/// //azxiiv///i.....//////i http://wwwvv...../aa/..a abs 2005. 02960
Shen Yan, Biyi Fang, Faen Zhang, Yu Zheng, Xiao Zeng, Mi Zhang, and Hui Xu. HM-NAS: Efficient Neural Architecture Search via Hierarchical Masking. In Proceedings of the IEEE In- ternational Conference on Computer Vision Workshops, pp. 0-0, 2019. TRLL:ttp://artt
orq/abs/1909 00122
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pp. 7105-7114, 2019.
abs wwt. 09635
Yiyang Zhao, Linnan Wang, Yuandong Tian, Rodrigo Fonseca, and Tian Guo. Few-shot Neural Architecture Search. arXiv preprint arXiv:2006.06866; 2020. URL http://arxiv.org
abs/2006. 06863
Barret Zoph and Quoc Le. Neural Architecture Search with Reinforcement Learning. 2016. URL
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018. URL http: /ttp:///axiv.../ org/ abs/ abs/ 070012

---

Under review as a conference paper at ICLR 2021
A TRAINING SETUP
A. ENVIRONMENT
Each super-network was trained and evaluated on single Nvidia GTX 1080 GPU, using driver version 440.64, CUDA version 10.0.130 and CuDNN version 7605 in our Slurm cluster. The code run Singularity container using Ubuntu 18.04 with Python 3.6.9, We used PyTorch in version 1.5.1 and nas-bench in version 1.3, further details can be found in the provided sysinfo.txt.
A.2 NETWORK
Aside from the deliberate variations in super-network training and the seeds, all them were trained and evaluated in the same way, listed in Table []] Unless detail is mentioned there, confident of not using it (e.g. we use no regularization or gradient clipping by default). The full list of arguments of each training job can be found in the respective logJask.Itt, Section B.2
seeds
{0, 2, used, not deterministic PyTorch default
CuDNN
weight init.
Data
CIFAR-10 training set same 5000 random images for all
for validation:
for training:
remaining 45000 images, shuffled
batch size
256
Augmentation
like DARTS:
shifting:
4 pixel
flipping:
borizontally
normalizing:
for validation:
only normalizing
Optimizer
SGD:
initial LR
0.025
momentum
0.9
weight decay
0.0003
eexcluding BN
Scheduler
Cosine decay:
final LR
0.00005
epochs
250
Table default neter-k training setup
B PROVIDED DATA AND CODE
Please the supplementary material for the following data and code. Due the amount of Ten- sorboard files, a 7zip compression is necessary to be below the allowed 100MB limit.
B. BENCH101
Since the original MAS--Bech--201 contains far more information than we need for the evaluation and requires impractically many resources (25+ GB RAM), we have reduced version that averages results over seeds and contains only the required stats.
The data file (nasbench201.1. Jmiiiip pmnni.pt) and the code to use it (code/bench.py) are provided.
B.2
RUN DATA
The relevant logs and Tensorboard files of every run (slurm job) are provided in the run data folder, grouped by experiments. The code/parse runs.py script is used extract desired metrics from the
12

---

Under review as a conference paper at ICLR 2021
these files and average them across the jobs that used different seeds. Running the script generates text output (csv format) that used in plots.py.
B.3 PLOTTING
Running plots.py will use the previously generated esv text, containing the mean, standard deviation, etc. and generate the plots from the paper.
C FURTHER FIGURES
Due to space limitations, we could not add further plots to the paper. The remaining ones for the evaluation on Cifar10-valic found below.
If you are interested in evaluation results on the other data sets, please take a look at Appendix B
>ita Tep
-0.5
5 110
-0
100
5 2.0
-0- 22onk
-F- 5 (154)
No 222
220
-4- (raikee (lom Clxt)
Sraiie/Zee Ske]
E3C
Scaak-1520 5220 Cldd)
(tkak Clull
243
(30)
(tadk//fdd Cld)
Sitaika/Spp C3d)
211
-0.00
201
31
+ X4
5 50-0
3. mamb---0 [aod]
5- 123
030
+ M. (aat--(Spp))
4- 331
-4- M. maak. (336)
5- 222
made- (2m, Ctuil
-4- M. (2ms Skap)
M mmmb-(Cii
zudde (84y, Chell
Minaki clid)
02:
H. naa (C20)
(Che
M.neb-(Skpp
Figure 7: See Figure 4 the left column kept the same, adding masking combinations with the 1x1 Convolution (center column) and Skip (right column)
13

---

Under review as a conference paper at ICLR 2021
0.2
R2 -i.b Thy
227 vi.i
-350
25
100
100
203
500
30
100
150
203
500
25
100
100
203
500
mecoo
Top boach enturiks
Tap Tanch seturals
Tog Tawc esturiks
100
S.mnnd-:2en Zem Podl
Sitakk Pood) LT
whilk Tou
03
100
725
731
Tag
31
2--2
3t mttne (2aas Elae))
142
- 34 maai (Baos Poel)
M. 17-1(aap)
2.07
--- M. (1atk--(BBo Elst), 17999
E31
31 (ean Poel) 1T=:Stp)
220
M. (Sdd Daad
-a- M.ttnd-fees Clrl), E2-Baay Pod)
M. raab BJor Podl, Di Pool)
0.25
EST Pboi Thpi
0.10
-12
500
100
150
500
[caed=((Cad). 18-18
S.mnbb-CC1)) 188pp)
(1aad_(((71a))] (2-1 Phod)
- [cae)L/(Ue)) 18- (B80). Phady
Strak---Pool 17- Poll]
4-
KT siow Medd
RT
-14
100
160
233
500
naton
160
160
253
500
ma220
100
150
253
501
maioos
Tig boach snturide
Trp Tewel seturdls
Tog 37 bwack anturike
Poal)
Figure 8: See Figure 5 further search space subsets with Linear Transformers.
14

---

Under review as a conference paper at ICLR 2021
RT Prtvlll Tre
100
230
80
27
100
120
10
150
120
230
200
podes
Top lead taneao
Tip lead satedo
cemplig
(. staygeg
(Bme Paell, sespling
(. wetegg
8. S. (Ber. Poell, mamaap
(. and coppee
5. ( Paell, godt clyptiz
(.zr)-(Cins) repiartod
8. S.morr (Bem, Paoll, eaplenel
wille Top
30-
300
-301
725
31
3. M.nond-[Clall
0000
romde B8m, Phol]]
5 -
net duard
5- 0.21
-8- 1. maak-(Chel ret dand
0-0
--- St. macke- Phil]] skosl
5- 0.5
S... sueplem
3. mam- M.tna--(Cist [Ctes] amplez
madk [Bm, Peall, seneding
currey
0.96
mak-
Peall,
St. gad shpring
*.- St. regakziard
(mme)) repeadden
peen Phil))
Figure 9:
See Figure 9 the left column is kept the same, adding masking combinations with the 1x1 Convolution (center column) and Zero+Pool (right column)
100
sifaric-vald,
cfarrn, Imagehet16-120 KT: 0.83
ImageNe:16-120 KT: 0.84
80
80
0022
40
40
0-
0-
80
100
20
100
cifa/10
diaa100valic
offr20 valid KT: 0.88
ImageNe:16 120, KT: 0.82
80
80
40
difa10,, KT: 0.84
difr10-valld, KT: 0.83
01
0-
cffr100,,
100
20
30
50
clfec100
Figure 10: The accuracy values (x axis) of all architectures on the NAS-Bench-201 data sets and their respective accuracy the other three data sets (color, y axis).
15

---

Under review as a conference paper at ICLR 2021
D ADDITIONAL METRICS AND WIDE-CHANNEL HEPER-NETWORKS
In addition to Kendall's Tau, we now also provide the Spearman Correlation Coefficient (SCC) and the Pearson Correlation Coefficient (PCC) (Li et al. (2020a)) for selection of the experiments, and additional experiments on small but wide super-networks, starting with 96 (instead of 32) channels.
0.9
validation aDcuRACY
0.7
0.6
0.5
0.2
0.2
KT within Top N
NT within Top
-0.2
-044
-0.1
00.6
-06
0.4
SCC wittác Top N
0.25
SCC wishin Top N
0.00
0.0
0.2
-0.25
0.4
-0.00
0.6
-0.75
0.6
POD within Top
POC within Top
0.2
-0.2
0.5
-0.6
25
50
100
150
250
500
raniomm
10
25
100
150
250
500
random
Top N beuth wetwooks
Top N beuth nelwoeks
y,= 0.07
-- 0.43
S. mack (Zero)
-0.0
S. mark=
=
mork== (anel==ZZeeo Zeeoo Pool)
7,==1.00
mark= (Zero, Pool)
7,m-0.98
Figure 11: Additional metrics for small super-networks that start with (left) or 96 channels (right).
16

---

Under review as a conference paper at ICLR 2021
0.8
volicatioon accuzaey
0.8
0.7
0.7
0.6
0.6
0.4
KT within Top N
KT wishin Top N
0.2
0.0
0.2
-0.1
0.4
0.6
0.50
0.4
500 wättác Top N
SCC within Top N
0.25
0.2
0.00
0.0
-0.25
-0.50
0.6
0.6
PCC widii Top N
0.50
PCC within Top N
0.1
0.25
0.00
0.0
00.5
-02-
25
50
100
150
250
500
random
10
25
100
150
250
500
randomm
Top
bench networks
Top benech networks
5. LT=(Ship)
Ya= 0.50
S. LT=(Skio)
0.57
S, moek= (Zer)), [T=(Skip)
0.50
S. mask= (Zero), LT=(Skip)
maek= Zeroo Pool), (S=ipp
S. (Zero, Pool), LT- (Skip))p)
5.-0.44
Figure 12: As Figure
adding linear transformers.
$5.00
90.06
3.25
2.55
535
0.3
540
0.8
370
24 82n 250
CA
Visualizing some metrics, similar to Figure 2 of the small and wide neter-setworks (Figure 12 left, green)
Figure 13:
17

---

Under review as a conference paper at ICLR 2021
validiaiin accunae
0.6
velidacion
0.1
0.2
RT wishin Top N
KT wishin Top N
0.2
200
-02
-04
0.4
SCC willin Top N
SCC widhn Top N
0.
-02
02
0.4
0.4
0.4
POC withie Top N
PCC within Top N
0.2
0.0
002
-02
-0.1
25
100
150
250
500
random
10
25
50
100
150
250
500
zandom
Top bearch networks
Top
C32
0.36
C32, [Skpp
(Skip)
--
C48
T -1000
C48, LT (Skep)
-- 0.57
O64
T -0.86
C64, LT (Skip)
0.57
C64
C61, [T=(Skpp
0.21
Figure 14: Super-networks with to 5 normal cells per stage. Smaller sized weight-sharing super- networks are generally easier to train and better predictors the search space.
18


Reviews:
[{"review_id": "vJzEWZvDffO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an empirical study on the ranking correlation in the singe-path setup. Following the paradigm of NAS-Bench-201,  the authors test the Kendall Tau correlation of networks from NAS training and networks from standalone training. \n\nIn general, I appreciate the authors' effort in bringing more infrastructure to the community of NAS. As a recently emerged community, we do need works like this one, as well as previous ones such as NAS-Bench-101 and NAS-Bench-201, to make the evaluation protocol more scientific. NAS problems are non-trivial as the search space is notoriously large. Colleagues who would like to invest their time and resources in exploring and manifesting this search space to uncover more phenomena are thus worth respect. \n\nHowever, this respectable responsibility also comes with a higher standard to evaluate works attempting to fulfill it. My major concern with this work is that the manuscript is not organized well. Although authors provide substantial details on their empirical study, they did not form a coherent logic flow to present these empirical findings, which makes this work more like a technical report than an academic paper. Readers may find these phenomena interesting but may not get interesting insights after reading this paper. Hence the technical contribution, especially on novelty, seems quite limited, even if there may be some intriguing points in the authors' discovery. I would recommend the authors to pick some phenomena e.g., masking Zero, masking Skip, etc., as examples to provide more analysis, so as to demonstrate to colleagues in our community that these findings can indeed lead to interesting research topics. \n\nSome minors: There are some works missed in the literature review. For example, the authors did not give adequate credits to colleagues who pioneered in using Kendall Tau to evaluate NAS training. As far as I know, Sciuto et al., 2019 was one of the earliest works. In the third paragraph, when reviewing recent progress, the authors did not distinguish the ranking correlation between NAS searching and retraining from the correlation between NAS searching results and stand-alone training. The former one was discussed and addressed in Hu et al., 2020. As the community has not fully realized the subtle but crucial difference between these two correlations, I believe a better framing of this work can be more helpful to other colleagues, especially those new comers. \n\nSciuto et al. 2019, Evaluating the search phase of neural architecture search.\n\nHu et al. 2020, DSNAS: Direct Neural Architecture Search without Parameter Retraining", "extra_scores": {}, "extra_text": {}}, {"review_id": "npIVnLdSOWr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed several findings on the single-path training strategy. The ranking correlation is the main issue. The experiments are conducted on NASBench201. \n\nIntroduction.\n\n'However, the aforementioned weight sharing xxx', there are a number of efficient multi-objective (Pareto front) NAS methods.\n\n'However, since the single-path', please cite some literature related to the ranking issue.\n\nSummarize the main findings in the introduction section.\n\nMethod.\n\nDefine $\\tau_{\\alpha}$ in math. 'describing the ranking correlation of the average prediction accuracy depending on N', what is the average prediction accuracy for two lists?\n\nExperiments.\n\n4.1 'masking the Zero operation (bottom row) significantly reduces this portion and thus improves the ranking correlation $\\tau$ (KT)'. If removing other operations other than zero, will the $\\tau$ be lower? In my opinion, removing any operation leads to smaller search space, and they all have a higher ranking.\n\n4.2, 4.2 examine some of the training strategies proposed by previous works.\n\nReferences.\n\nPlease make the references clear. Add the venues for all the papers. \n\n\n\n\nThis paper tries to explore the single-path training strategy by studying the search space, the supernet, the linear transformer, the strict uniform sampling, the topology sharing, the LR warmup, the regularization, the clipping. The authors have done lots of experiments to clarify the important reasons for the ranking. However, most of the findings are not new to me. They have been discussed more or less by previous works and discovered by my own experiments. So the contribution is not significant. The paper is mostly clear, some paragraphs and references need to be polished, more related works should be added.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_tiCpFGx1NR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the relationship of correlation of ranking of networks sampled from SuperNet and that of stand-alone networks under various settings. They also study the how masking some operations in the search space and different ways of training effect the ranking correlation.\n\nPros:\nThe paper has a lot of experiments to substantiate the claims.\nFigure 3 where every operation is systematically masked, provides more insights about which operations are effective and how NAS behaves if one of the operation is masked.\n\nCons:\nSeveral other papers have already published similar findings. Overall the paper is very incremental.\nMore specifics in the questions\n\nQuestions\n\n1. How is the SuperNet trained?\n2. Figure2: Yu et al [1] have already explored the correlation of ranks of networks sampled from SuperNet and that of stand-alone networks. How is Figure 2 different from that? \n3. RobustDarts [2] has explored the possibility of how subset of NASBENCH search spaces behave. FAIRDarts [3] also explored the influence of skip connection by running DARTS without skip connection, running random search by limiting skip connection to 2 etc. Figure 4 seems to be inspired by that. While it is interesting, this might be a slight extension to the work done by Yu et al [1]\n4. Bender et al [4] postulate that the operations of a SuperNet are subject to co-adaptation and recommended techniques such as regularization, drop path etc to alleviate the same. RobustDarts also suggest some recommendations such as L2 regularization, drop path etc although in the context of DARTS. So while Figure 6 demonstrates this empirically, it is not a new finding.\n\nOverall, the empirical results in the paper are very useful for the NAS community. But the work is still very incremental. This might be better received as a workshop paper instead.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RW2ClSyiPn0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n+ This paper studies the single-path one-shot super-network predictions and ranking correlation throughout an entire search space, as all stand-alone model results are known in advance. This is a crucial step in NAS. As we know, inaccurate architecture rating is the cause of ineffective NAS in almost all existing NAS methods. It makes nearly all previous NAS methods not better the random architecture selection (suggested by two ICLR 2020 papers and many ICLR 2021 submissions). Therefore, analyzing the architecture rating problem is of most importance in NAS. This paper takes a deep insight into the architecture rating problem, which provides a timely metric for evaluating NAS's effectiveness. (+)\n\n\n- In the following text, another paper entitled \"Block-wisely Supervised Neural Architecture Search with Knowledge Distillation\" should be discussed: \"Recent efforts have shown improvements by strictly fair operation sampling in the super-network training phase (Chu et al. (2019b)), by adding a linear 1×1 convolution to skip connections, improving training stability (Chu et al. (2019a)), or by dividing the search space (Zhao et al. (2020)),\" (-)\n\n\n- Kendall's Tau is a good metric. As shown in EagleEye, Spearman Correlation Coefficient(SCC) and Pearson Correlation Coefficient (PCC) are also good metrics. Could the authors also provide a comparison using these two metrics? (-)\n\nEagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning\n\n\n- I think NAS-Bench-201 is not enough. As we know, CIFAR-10 is sometimes considered a toy benchmark, and the sole result on CIFAR-10 is not convincing. Could the authors provide more results in addition to CIFAR-10? (-)\n\n\n- As we know, there may be a gap between the small-channel supernet and the large-channel finally-adopted architecture. We are quite interested in the ranking correlations between a subnet obtained from the small-channel supernet and a channel-expanded version of the subnet trained from scratch. Could the authors provide such a ranking correlation analysis? (-)\n\n\n- Could the authors provide more details in Figure 3. Figure 3 shows that the lines on the top mean the operation is used more frequently. But I am not sure what the value of the y-axis means. (-)\n\n\n- Could the authors present some comments on \"Perhaps the most surprising is the low importance of Average Pooling, even lower than Zero, an operation that does absolutely nothing\"? (-)\n\n\n+ The following observation is believed to be crucial in NAS: \"The baseline for small networks (top left, red) has the same averaged prediction accuracy for the top 10 as for the top 500 networks\". This validates the inefficiency of SPOS in architecture search. (+)\n\n\n+ The following observation is also important in NAS: \"Masking Skip (blue, left) is the most harmful to τa (=1). As seen in Figure 4, the top-N networks have a worse average predicted accuracy than the top-M (for N < M) networks, and sometimes even below the random sample, which is terrible. Interestingly, \\tau may improve within the predictions for the top-N architectures.\" Especially, the phenomenon that masking skip connection reduces the ranking correlations is interesting. As is shown in SCARLET-NAS, the supernet training with skip connection is not fair. But in this paper, we can see that skip connection benefits the ranking correlation. We are interested in this opposite opinion. Specifically, it is fascinating to see that \"Although the additional transformers seem to stabilize training, as seen by the lower standard deviation, they also worsen the τa problem.\" Besides, the phenomenon of \"\\tao may improve within the predictions for the top-N architectures\" indicates that the metric for ranking correlations maybe not perfect. A more reasonable metric may be desirable. (+-)\n\n\n+ The following observation is important: \"medium-sized super-networks require additional care.\" As shown by Figure 4, the averaged predicted accuracy of top-N networks in several subsets is lower than that of a random subset of networks. This is consistent with previous work like DNA, which shows a large search space may be harmful to the architecture rating. Even if a medium-sized supernet has a bad architecture rating, the ranking correlation should be worse in a large-sized supernet. (+)\n\nDNA: Block-wisely Supervised Neural Architecture Search with Knowledge Distillation\n\n\n- The following description is questionable: \"After the architecture search, all Linear Transformers can safely be removed, as they do not impact the network capacity\". Actually, stacking many fully connected layers without non-linear activations could lead to only one fully connected layer. It is an open question of whether optimizing loss(ABCx, y) is as difficult as optimizing loss(Dx, y) using stochastic gradient descent. (-)\n\n\n+ The results providing evidence against disabling cell topology sharing during the training phase are exciting and new to the public. (+)\n\n\n+ The following observation is fascinating: \"The absolute validation accuracy value is increased by uniform sampling. However, this is not relevant, as only the correct ranking matters\". This is against FairNAS. (+)\n\n\n+ It is interesting and convincing that many tricks such as learning rate warm-up, gradient clipping, and regularization do not work to improve the ranking correlation. We are pleased that the authors provide so many experiments to point out some misleading approaches in NAS. I think this paper is very important in the context of AutoML. (+)\n\n\n- The analysis is based on medium-sized and small-sized search space. It would be good to see some analysis of large-sized search space. (-)\n\n\nOverall, this paper provides a timely analysis of the current NAS's ineffectiveness caused by the inaccurate architecture rating problem. As there are many NAS papers published every year and their ineffectiveness may still be not widely recognized by the reviewers and the public, I recommend a strong acceptance for this paper to promote the analysis of the NAS's architecture rating problem.\n\n\n\n\n------------------------------post rebuttal------------------------------------------\n\n\n-------------------Response to the authors' response----------------------\n\n\nThank you for the hard work in responding.\n\nI have read other reviewers' reviews and the response from the authors. The authors have addressed most of my concerns.\n\nI believe this paper deserves acceptance. As we know, variants of efforts have been made to improve NAS's effectiveness since 2016, and a great process has been reached. Despite the high expectation and solemn devotion, NAS's effectiveness is believed to be still low. This is inconsistent with many pioneer researchers' expectations four years ago, in which NAS is expected to be another revolutionary technique similar to 2012's deep learning. Currently, there are many NAS papers published every year. But their effectivenesses are unclear due to the lack of ranking correlation analysis. Differently, this paper comprehensively analyzes the architecture rating problem, which provides a timely analysis of the current NAS's ineffectiveness caused by inaccurate architecture rating. I think this paper can attract the community's attention, encouraging the community to pay attention to the architecture rating in NAS, especially when reviewing a NAS paper. Therefore, I recommend an acceptance for this paper to promote the analysis of the NAS's architecture rating problem.\n\nI agree with R2 that Yu et al. have proposed a similar idea (I assume R2 refers to \"Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, Mathieu Salzmann, Evaluating the Search Phase of Neural Architecture Search\"). But the analysis in this paper is more comprehensive than Yu et al.'s article. Many findings are new (at least they are not in published papers).\n\nI agree with R4 that the authors did not form a coherent logic flow to present these empirical findings, and the paper was similar to a technique report. However, many important articles, e.g., \"Designing Network Design Spaces,\" \"Exploring Simple Siamese Representation Learning,\" \"Is Faster R-CNN Doing Well for Pedestrian Detection?\" are also technique-report-like.\n\nI appreciate R1 for his devotion to finding similar observations in his experiments. I believe these observations are important and deserve publication. I agree with R1 that removing any operation leads to a smaller search space and a higher ranking.\n\nIn summary, I will keep my rating as an acceptance. \n\nUndoubtedly, I also believe the comments from other reviewers can benefit the improvement of your paper.\n\n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received reviews with a very wide range of scores (initially 3,5,5,9; then 5,5,5,9). In the discussion, all reviewers maintained their general position (although a private message by the reviewer giving a score of 9 said he/she would consider going down to an 8).

Because of the high variance, I read the paper in detail myself. I agree with all reviewers that NAS is a very important field of study, that the experiments are interesting, and that purely empirical papers studying what works and what doesn't work (rather than introducing a new method) are definitely needed in the NAS community. But overall, for this particular paper, I agree with the 3 rejecting reviewers. The paper presents a lot of experiments, but I am missing novel deep insights or lasting overarching take-aways. The papers reads a bit like a log book of all the experiments the authors did, before having gone through the next iteration in the process to consolidate findings and gain lasting insight.

In a bit more detail, half the results in Section 4 use medium-sized super networks, which seem broken to me, yielding much worse performance than small super networks. I did not find any motivation for studying these medium-sized networks, no reason given for them to perform poorly, and none stating why the results are still interesting when the networks perform so poorly (apologies if I overlooked these). The poor performance may be due to using a training pipeline that works poorly for these larger networks, but this is hard to know exactly without further experiments. I would either try to fix these networks' performance or drop them from the paper entirely, as I do not see any insights that can be reliable gained from the current results. As is, I believe these results (accounting for half the plots in the paper) only muddy the water and are preventing a crisp presentation of insightful results.

Another factor that I find unfortunate about the paper is that it only uses NAS-Bench-201 for its empirical study, and even for that dataset, mostly only the CIFAR-10 part. After getting rid of isomorphic graphs from the original 15625 architectures, NAS-Bench-201 only has 6466 unique architectures (see Appendix A of NAS-Bench-201), while, e.g., NAS-Bench-101 has 423k unique architectures. As the authors indicate themselves in their section "Grains of Salt", it is unclear whether insights gained on the very small NAS-Bench-201 space generalize to larger spaces. I therefore believe that there should also be some experiments on another, larger space, to study how well some of the findings generalize. An additional benchmark that the authors could have directly used without performing additional experiments themselves is the NAS benchmark NAS-Bench-1shot1 (ICLR 2020: https://openreview.net/forum?id=SJx9ngStPH), which studies 3 different subsets of NAS-Bench-101, and which was created to allow one-shot methods to use the larger space of evaluated architectures in NAS-Bench-101. 

Minor comments:
- It reads as if the authors performed 5 runs, computed averages of the outcomes, and then computed correlation coefficients. That would be a suboptimal experimental setup, though; in practical applications, only one run of the super network would be run, and therefore, in order to assess performance reliably, one should compute correlation coefficients for one run at a time, and then obtain a measurement of reliability of these correlation coefficients across the 5 runs.
- The y axis in Figure 2 appears to be broken: for example, in the left column it goes from 99.978 to 99.994, and the caption says these should be accuracy predictions of NAS-Bench201. However, even the best architectures in NAS-Bench201 only achieve around 95% accuracy.


Overall, I recommend rejection for the current version of the paper.
Going forward, I encourage the authors to continue this line of work and recommend that they iterate over their experiments and extract crisp insights from their experiments. I also recommend performing experiments with a much larger search space than that of NAS-Bench-201 to assess whether the findings generalize.
Decision: Reject