Picking up the pieces: separately evaluating supernet training and architecture selection
Authors: Gabriel Meyer-Lee
Nick Cheney
Venue: ICLR 2022 Submitted (2022)
Keywords: neural architecture search
automl
deep learning theory

Abstract: Differentiable Neural Architecture Search (NAS) has emerged as a simple and efficient method for the automated design of neural networks. Recent research has demonstrated improvements on various aspects on the original algorithm (DARTS), but comparative evaluation of these advances remains costly and difficult. We frame supernet NAS as a two-stage search, decoupling the training of the supernet from the extraction of a final design from the supernet. We propose a set of metrics which utilize benchmark data sets to evaluate each stage of the search process independently. We demonstrate two metrics measuring separately the quality of the supernet's shared weights and the quality of the learned sampling distribution, as well as corresponding statistics approximating the reliance of the second stage search on these components of the supernet. These metrics facilitate both more robust evaluation of NAS algorithms and provide practical method for designing complete NAS algorithms from separate supernet training and architecture selection techniques.

Under review as a conference paper at ICLR 2022
PICKING UP THE PIECES: SEPARATELY EVALUATING SUPERNET TRAINING AND ARCHITECTURE SELECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Differentiable Neural Architecture Search (NAS) has emerged simple and efficient method for the automated design of neural networks. Recent research has demonstrated improvements on various aspects on the original algorithm (DARTS), but comparative evaluation of these advances remains costly and dif- ficult. We frame supernet NAS as a two-stage search, decoupling the training of the supernet from the extraction of a final design from the supernet. We propose set of metries which utilize benchmark data sets evaluate each stage of the search process independently. We demonstrate two metrics measuring separately the quality of the supernet's shared weights and the quality of the learned sam- pling distribution, the second stage search on these components of the supemee.. These metrics facilitate both more robust evaluation of NAS algorithms and provide practical method for designing complete NAS algorithms from separate supernet training and architecture selection techniques.
well as corresponding statistics approximating the reliance
INTRODUCTION
Neural architecture search (NAS) describes the problem of automatically selecting an effective neu- ral network architecture for given data. In their survey of work on NAS, Elsken et. al. establish three necessary components to achieve this goal: search space to define the set possible architectures, search strategy traverse over potential solutions, and a performance estimation strategy to rate the quality of potential solutions considered by the search strategy (Elsken et al. 2019).
In recent years a variety of search spaces and benchmark problems have been proposed NAS (Elsken et al. 2019). Exploring the space of search strategies has similarly generated vast interest, with approaches spanning evolutionary algorithms (Real et al. 2018). Bayesian optimization (Zela ct al. 2018), reinforcement learning (Pham et al. 2018), and, most recently, direct gradient descent (Liu et al. 2019). Various methods for performance estimation have also been proposed, including weight inheritance (Real et al.) (017)), predictive models (Liu et al. 2018)) shortened training (Zoph) ct al. 2018), and weight-sharing (Pham et al. 2018).
However, the search space, search strategy, and performance estimation components of a NAS al- gorithm do not operate in a vacuum, and the interactions and relationships between them may dra- matically affect the behavior and performance of the overall algorithm. For example, performance estimates that extrapolate from previously seen models are biased by the search strategy's 's method for traversing the search space and differentially sampling different models. Here, suggest that the overwhelmingly trend in the current literature is to focus a study on innovations within single one of these aspects (e.g. shared-weight model training for performance estimation) without a broader and systematic consideration, analysis, or even toolset for exploring the effects that this innovation may have on other components of the NAS algorithm (e.g. search strategy for selecting architecture topologies). While we do claim to provide a perfect solution for the analysis algorithmic inno- vation's 's impact on each the different components, here we provide one example of a systematic strategy and open-source toolset for performing such investigation and demonstrate a specific example of how such a perspective modifies the way that we interpret the theoretical impact of an important recent algorithmic and innovation.
This paper focuses specifically on the class of algorithms which combine a weight-sharing perfor- mance estimation strategy and gradient-based search strategy into the training of single model.

---

Under review as a conference paper at ICLR 2022
This approach trains a neural network representing a superposition of every architecture in the search space instead of training individual architectures and obtains performance estimates from this thared-weight model. This technique, which we will refer to the supernet or supergraph method, was originally demonstrated by Pham et ( (2018), however the gradient- training method con- tributed by DARTS (Liu et al.) 2019) eliminated the need train second reinforcement learning model to select architectures, leading to the development of the series of NAS algorithms combining performance estimating and search in a single model considered here. Note that following [ender et al. (2018). the term "One-shot NAS" has been used to refer to supernet methods as they train a single model rather then iteratively training candidate architectures. However, as we will discuss iterative search algorithms implemented on top the supernet, we adopt the supernet terminology used by Wang et al. (202T). Differentiable supernet NAS algorithms have gained popularity due to their speed attained through weight-sharing especially relative to employing the "gold standard" performance measure of training a neural network from multiple different random initializations and measuring the accuracy hours for a single run (Zoph et al., 2018). They also benefit from and their ease of implementation within popular deep learning frameworks a result of their reliance on gradient based optimization.
held-out set of validation data that often requires thousands of GPU
Importantly for this paper's perspective, these algorithms establish a special relationship between the search strategy and the performance estimation strategy collapsing both components to the training of a single neural network. This presents a particular example of the cross-component in- teraction phenomenon described above during the supernet training. where the shared weights are trained to maximize the performance of the previously explored architectures, and are then em- ployed to estimate the performance of new potential architectures. The feedback loop between these two processes thus creates an opportunity to bias search towards familiar solutions, possibly at the expense of undervaluing architectures which have high train-from scrotch performance, but may not perform well with the weights learned on previously explored architectures. This interaction is seen again post-supernet training, when the shared-weight model (itself embodying the chosen perfor- mance estimation strategy) is often critical in the process select the single best architecture from the search space. With both architecture performance estimation and search strategy represented by the training of single suepmeet model, it is perhaps these two are not easily measured or conceptualized. We might further expect the degree to which this may be an issue to vary with the convexity of the search space, and the search strategy used to traverse that landscape.
surprising that the distinet contributions of
In order facilitate a greater understanding of, and more robust comparisons between, differentiable supernet NAS algorithms, here we present two statistics designed to separately evaluate (1) the ar- chitecture search and (2) the performance estimation components of the supernet training process. We also demonstrate the value of separately evaluating performance estimation and search during supernet training by presenting a two-stage framing of differentiable stochastic NAS, delinking the process of training the supernet from the process of selecting
architecture using the supernet. Using our statistics of the Stage-l supernet training, we present a method for approximating the reliance of the Stage-2 architecture selection methods on the performance estimation capabilities of the supernet versus the discovery of quality architectures through supernet training. We offer empirical results demonstrating the capabilities of using the presented statistics to design NAS al- gorithms through the composition of separate supernet training and architecture selection methods, as well highlighting a case-study demonstrating the importance of considering the interactions of Stage-1 supernet training and Stage-2 architecture selection on the analysis of a recent algorithmic innovation.
2
RELATED WORK
2.1
DIFFERENTIABL SUPERNET NAS
DARTS was the first published differentiable NAS algorithm, utilizing a "continuous relaxation" method to parametrize the set of possible architectures, by placing trainable weights on each possible operation (Liu et al., 2019). The final model is selected from these trained architecture weights by simply keeping only the operations with the largest weight. This method is general enough to support the discovery of a broad range of neural network topologies through the use of a "zero" operation, representing the lack of edge between nodes. Numerous works since the publication
2

---

Under review as a conference paper at ICLR 2022
of DARTS have attempted to demonstrate simple improvements on DARTS to stabilize training, avoid the hard prune at the end of training, or improve scalability. For example, PC-DARTS (Xu et al. 2020) has demonstrated partial channel training, allowing the use of larger supernet model. This work addresses the final hard prune by removing the magnitude based prune and substituting perturbation- haaood pruning algorithm. While Chu al. (202T) stabilizes the training process using an auxiliary skip connection, in contrast to prior work which relied on the Hessian eigenvalue regularizer.
In motivating SNAS, Xie et al. comment on the performance estimation aspect of DARTS, demon- strating the disparity in the validation accuracy obtained from the shared weight model of DARTS using the whole seared-weight model and the pruned final architecture (Xie et al.] 2019). Stochastic differentiable NAS algorithms, like SNAS, offer one answer this question of performance estima- tion in differentiable NAS through the use of sampling. These methods preserve the gradient-based approach of DARTS, while proposing specific (and increasingly discrete) architecture at each step.
Motivating the performances estimation abilities of shared weight models more generally, Bender eet al. (2018) implement path-based form of dropout, randomly masking portions the shared weight model during training and demonstrated a strong correlation between performance estimates from shortened training and the shared weight model, suggesting that the two methods of perfor- mance estimation are comparable.
2.2 ZERO-SHOT NAS
Recently, methods have been proposed to select an architecture without training single archi- tecture model. Much of these methods are based on proxies, like EcoNAS (Zhou et al,, 2020). while TE-NAS (Chen al. 2021a) recently demonstrated zero-shot search based on measures emerging from deep learning theory. Abdelfattah et al. (202T) demonstrate a range of measures which have been used to compute saliency in other deep learning problems as possible measures to be useful in NAS. In this work we modify the adapted synflow measure demonstrate by Abdelfattah et al. (2021 as well their implementation Jacobian covariance measure developed by Mellor et al. (2021) to function as architecture selection techniques on supernet.
2.3 SUPERNET NAS EVALUATION
Research on the evaluation of NAS methods has developed as a result of difficulties in comparing the performance NAS algorithms demonstrated in different architecture search spaces, as the selection of search space can have a greater effect on the final performance than the selection of NAS algorithm, which can be made clear through performing a random search (Li & Talwalkar 2019).
Yu et al. 2020) initiated the critical inquiry into the use of weight-sharing for performance timation by evaluating popular weight-sharing NAS algorithms in a reduced search space of 32 architectures. They demonstrated lack of correlation in ranking between the performance esti- mates obtained from the thared-weight model and the trained from scratch test accuracy for and an improvement in performance of the final model achieved by not using weight-sharing. Further research evaluating the rankings of small samples (Yang et al.) (2020) or small search spaces (Zhang) al. 2020) showed high variance in rankings across random seeds after training the shared weight model through random sampling. Like these prior works, we also suggest the the accuracy of the top model alone can be misleading evaluation technique for NAS algorithms. However, our basis for this assertion is not the design of the search space, but that the reported accuracy value may be due to the unintended interactions between algorithmic innovation presented and other components of the NAS algorithm (e.g. performance estimation or search strategy). and not necessarily well repre- sent the merits of the proposed methodology itself. This motivates our proposal here for additional metrics which evaluate separately the performance estimation architecture sampling aspects of given algorithm.
This work is most aligned with direction of NAS evaluation research demonstrated by Zela et al. 2020). In this work they propose the dominant eigenvalue of the Hessian of the validation loss w.r.t. the architecture parameters as statistic which helps to explain the failure modes of DARTS. Specifically, they associate a large eigenvalue with a significant drop in performance resulting from pruning. We are also attempt establish statistics which indicate the potential outcome of archi-
3

---

Under review as a conference paper at ICLR 2022
tecture selection for given trained supernet, however not focused on the pruning-basedd architecture selection used by DARTS. Instead of formulating our statistic to account for the failure modes of specific NAS algorithm, we propose statisties based on the broad theory of the NAS process.
3 CASE STUDY: TWO MODIFICATIONS TO DARTS
To motivate the importance of separating the evaluation of different components of NAS algorithm proposed below, here we present a experimental case study. Two recent papers demonstrated non- overlapping modifications to DARTS named DARTS- (Chu al. 2021 and DARTS-PT (Wang et al. 2021). Each paper aimed to tackle the problem of unstable architecture weights in DARTS. DARTS- addressed this issue by adding an auxiliary skip connection to each mixed operation in DARTS, aiming to stabilize the training of architecture weights. DARTS-PT instead dismissed the architecture weights entirely, using perturbation based method of selecting operations after training the supernet. The authors conclude that the perturbation technique may work better if DARTS is trained without architecture weights at all as they were able to demonstrate a smaller using their method trained by DARTS. This result would suggest that the core innovation of DARTS, the continuous relaxation technique for approximating gradient updates to the architecture of the model, may be of significant use. However, the generality of these conclusions are unclear from DARTS-PT alone.
supernet trained with fixed architecture weights than
supernet
Search Space
DARTS
DARTS-
Max
Perturb
Perturb w/ fix
Max
Perturb
Perturb w/ fix
NAS-Bench- 201
45.7
11.89
6.20
6.75
6.56
7.22
SI
3.84
3.5
2.86
2.62
2.87
2.76
S2
4.85
2.79
2.59
2.65
2.47
2.80
S3
3.34
2.49
2.52
2.58
2.53
2.54
S4
7.2
2.64
2.58
3.44
2.67
2.61
Table 1: A case study showing the importance of separately evaluating supernet training and archtiecture selection. The DARTS-PT architecture selection method standard DARTS (left) shows improved peformance when not training architecture weights. But when the same perturbation method is applied to a supernet trained via DARTS- results from in- cluding auxiliary skip comnection (right), we see improvement when including architecture weights. This reversal to the "Stage-2" perturbation method of selecting architectures due to that method's reliance on the ("Stage- 1"") training of the supernet. The top method for each search space is indi- cated in bold while the top architecture selection technique for the version of DARTS which did not attain the overall top result is italicized.
supernet trained
By implementing DARTS- within the provided codebase for DARTS-PT and replicating their tests, observe reversal of this result in T We have run trials on CIFARIO of perturbation search applied to DARTS- in each of the search spaces where DARTS-PT with a fixed a was shown to outperform DART-PT with a trained alpha (though this was only the case for S3 on data other than CIFARIO) 10). Once training of the architecture weights is stabilized via the auxiliary skip con- nection, training with architecture weights actually does provide an improvement to the perturbation based selection method. Perturbation applied to DARTS trained with fixed a still appears to be an effective method, attaining 2 of the top 4 results it held prior to comparison with DARTS- and the perturbation method appears quite effective overall, attaining the top result for every search space except $1. However, these results do suggest that the striking result that the process of training architecture weights actually results in lower final performance after perturbation-base will likely not generalize beyond the original formulation of DARTS. Although the perturbation method does not utilize the architecture weights, the DARTS- results indicate that they influence the training of the shared weights in a way which can assist the perturbation process the architecture weights are being trained effectively.
selection
We do not present this case study in order to undermine the assertions supporting DARTS-PT about the inadequacy of architecture weight pruning architecture selection method. In fact, this
4

---

Under review as a conference paper at ICLR 2022
assertion aligns closely with the work demonstrated in this paper about alternative architecture lection techniques for differentiable NAS. Rather, we present this case study in order to demonstrate the necessity of a common theoretical frame in order to produce general insights to the architecture selection process in supernet NAS. In this case, an insight into the limitations of the architecture weights, turned out This specificity of the insight would have been difficult to anticipate as supernet methods lack com- mon interpretation of the architecture weights or method evaluate them, beyond the performance of their argmax. We propose a general method for evaluating the supernet training and architecture selection methods separately as path towards more general insights and rigorous understanding of supernet-based NAS algorithms.
longer hold across minor modification of the supernet training method.
4 TWO STAGE SEARCH
Much of the criticism of DARTS has focused specifically on the method by which the final archi- tecture is extracted from the supernet. These issues with the "hard- hourd-prune implemented in DARTS distract from evaluation of the process by which DARTS trains the supernet. We view this as an extreme example of broader phenomenon by which the reliance on the accuracy of final selected architectures as the standard performance metric dilutes our ability to evaluate specific innovations to the shared-weee weight model training and/or final architecture selection components of NAS. To rem- edy the lack of feedback on these individual aspects of a NAS algorithm, we employ a two stage search framework for differentiable supernet-based NAS.
We consider the training of the supernet as a "Stage-1 search algorithm" on its own. Although these algorithms do search the architecture space, their output is not single final architecture but some form of information about the relative quality of architectures. We consider two possible forms this information can take: a weighting/sampling distribution over the architecture space (exemplified by a in DARTS) and a trained supernet model which able to provide performance estimates of individual candidate architectures. Note that the Stage-I search algorithm need only provide one of these, as One-Shot NAS does not utilize any learned distribution over the architecture space, and the default Stage-2 search method for DARTS makes no use of the supernet model's shared weights.
We then denote the architecture selection process inherent to all sapernet-based NAS algorithms as "Stage-2 search", which takes in the supernet model and associated weighted distribution over the architecture space and selects the single top architecture for that search space. The purpose of this terminology choice is to draw attention to distinct components of existing NAS algorithms and investigate their relationship to each other. There are numerous ways that final architecture can be obtained from any supernet training algorithm and, though some selection algorithms may be more suitable for particular training algorithms, no supernet training algorithm is necessarily or trivially tied to a particular architecture selection algorithm, or vice versa. We utilize the original names of the algorithms denote Stage-1 search algorithms for shared-weight model training, such as DARTS and SNAS, while we name the Stage-2 search algorithms after the architecture selection criterion used.
Note that these two search stages are simply a decomposition of the supernet-basee search, and do not also consider or propose additional steps in the NAS search pipeline, such as obtaining train- from-scratch accuracy over multiple trials or further finetuning of the best of these models - as noted by Li & Talwalkar (019) as aspects of the original DARTS search method.
Through this framing of two-stage supemeetbaseed NAS, we are able to unify several disparate strands of NAS research. As the default Stage-2 search method of DARTS notoriously unsta- ble, techniques such as that of [Wang et al. (202T) have been developed to bypass it. At the same time, zero-cost NAS methods, such as those of Abdelfattah et al. (202T) have provided advanced statistics for the rapid evaluation of sampled architectures. Positioning both of these techniques as Stage-2 search algorithms allows us to broaden the application of the former beyond DARTS and integrate the latter with supernet-based NAS techniques.
We evaluate both search stages and their combination using relying on the mark. To begin to explore the generality of these results, we provide results on the three provided benchmarks within the dataset (CIFAR- 10, CIFAR- 100, and (mageNet16-120), Cognizant of the rel-
Dong & Yang 2020), test accuracy metrics provided for each architecture in the bench-
5

---

Under review as a conference paper at ICLR 2022
atively small search space of this setup, we also provide results CIFAR-10 on the larger DARTS search space.
4.1 EVALUATING STAGE-1 SEARCH
We consider the process of training a supernet to be Stage-1 of the architecture search, resulting in a relative ranking estimating the quality of architectures (or their subcomponents of individual odges/nodes/operatioons). How much this process actually constitutes a search of the architecture space depends entirely on the training method. If we were to substitute training subset of the su- pernet for individual model training in an evolutionary NAS algorithm, we would obtain a training method where the supernet is trained via an explicit search of the architecture space. Along these lines, the Random Search with Parameter Sharing (RSPS) (Li & Talwalkar 2019) training method can be viewed random search of the architecture space or, conversely a supernet training method which involves no search of the architecture space (as the information about the observed perfor- mance of sampled architectures is not used inform the search process). The algorithms we con- sider here, differentiable supernet training methods, occupy more ambiguous position. DARTS and its variants do not evaluate individual architectures in the search space during training, but do eval- uate supersets of components of the search space. In NAS- PAS-Bench-201, and any other search space encoded as a set of single-choice categorical variables, the sampling- basen differentiable NAS algo- rithms actually do iterate over architectures within the search space, though only those that impose a discrete constraint like GDAS are actually evaluating individual architectures. In general, these dif- ferentiable supernet training methods can be construed as implicitly searching over the architecture space, performing and retaining record of comparative evaluations within the search space, but not necessarily at the level of individual architectures.
We wish to, as separately as possible, evaluate how these algorithms train their architecture and shared weights. As chitecture weights, after a softmax, correspond exactly to a categorical distribution over operations. From the probability of each operation in model, we can take their product to find the sampling probability mass assigned to each architecture in the search space. We utilize the benchmark data set compute the probability of sampling architectures of a given test accuracy, allowing to evaluate the quality of the architecture weights by directly observing the sampling probability mass they have assigned the highest performing models in the search space. Fig. (Right) shows the cumulative distribution of sampling probability by test accuracy. Points above the baseline of uninformed ran- dom sampling (dashed line) demonstrate algorithms with a tendency to oversample high performing architectures, while curves below the baseline undersample high performing architectures.
search space allows single operation at each edge, the ar-
In order to evaluate the quality of the shared weights learned by the supernet training algorithms, we utilize them to record a validation accuracy for every architecture in the search space. We then rank every architecture according to its approximated performance and compare this ranking to the ground truth ranking obtained using the benchmark test accuracy. For a given test accuracy level, the proportion of architectures with ground truth accuracy of at least that level that are also predicted to be in that ranking is recorded (e.g. if 20% of the architectures in the top decile of models accord- ing to their train-from-scratch test accuracy are predicted to be in the top decile via shared-weight validation accuracy for a given algorithm, then at the test error threshold separating the top decile of architectures from the bottom 90%, that algorithm would have a anp-aarciieeturr-dddnnfiaaaaaiii (i.e recall) rate of 20% or 0.2). Fig. [ (Left) shows this curve relative to a baseline of random validation rankings. Points above the baseline better estimate performance performance-estimanl architectures, while points below the baseline curve are worse at estimating for the performance of perforrierfonming architectures. In addition to the CIFAR-10 10 performance es- timation and sampling probability curves Fig. [ (Top), the metries on Imagenet Benchmark from NAS-Bench-201 in Fig. (Bottom) show that SNAS and Dirichlet Neural Architecture Search (Dr- NAS) highly outperform the other algorithms at both estimating performance of top architectures and biasing the sampling distribution towards them. Interestingly, DARTS (which never samples discrete architecture during the training of the shared-weight model) performs very poorly at both performance estimation and oversampling of top architectures. With the DARTS- innovation, the supernet does improve its ability to oversample serformnn architectures, making the algorithm competitive with the worse performing sampling-bar based approach we investigated (GDAS).
6

---

Under review as a conference paper at ICLR 2022
Although only one Stage-2 method evaluate actually directly relies validating individual archi- tectures using the supernet, by obtaining performance estimates for each architecture in the search space and then attempting to rank them, provide proxy measure the amount of information about the relative quality of individual architectures within the supernet, which should also play a role in determining any other statistic computed over individual architectures using the supernet.
Architecture ranking performance, cifar10
Sampling probability distribution, cifar10
1.0
Sampling probability mass
1.0
0.8
Top arch ID
0.6
0.6
DARTS
0.4
GDAS
GDAS
0.2
DIRICHLET
0.2
DIRICHLET
random baseline
random baseline
0.0
0.0
101
2x101
x 10
10a
2
x 10!
Test err thresh
Test err thresh
Architecture ranking performance, imagenet16-120
Sampling probability distribution, imagenet16-12
1.0
Sampling probability mass
1.0
0.8
Top arch ID
0.6
DARTS
0.6
DARTS
DARTS-
DARTS-
0.4
SNAS
0.4
SNAS
GDAS
GDAS
0.2
DIRICHLET
0.2
DIRICHLET
random baseline
random baseline
6 10
7
B x 10
6x
7 10
8 x 10'
Test err thresh
Test err thresh
Figure 1: (Left) Top Arch ID rate of shared-weight based validation accuracy and benchmark test accuracy across test error threshold groups of the architecture space. (Right) Sampling probability mass across test error threshold groups of the probability space.
4.2 EVALUATING STAGE-2 SEARCI
In our Stage-2 metric, we seek to evaluate the ability of an algorithm to select the top discrete architectures from given supernet. In order to evaluate the architecture selection process, define a standardized input for the Stage-2 search algorithms. We wish to provide a sufficiently broad baseline such that it is suitable for evaluating any Stage-2 search algorithm which can be implemented on top of a supernet training algorithm. As we consider the supernet itself as the input the architecture selection algorithms, our baseline must consist of standardized of supernets, trained in a manner such that evaluating architecture selection methods on them facilitates estimation of their efficacy across a range of Stage-1 1 search methods.
Unfortunately, there is no readily apparent method for training architecture weights that could serve as a suitable baseline. As the method by which they train the architecture weights is the defining characteristic for several of the Stage-1 search algorithms train the architecture weights already one the algorithms (DARTS) is not feasible to de- sign baseline training method which involves training the architecture weights without biasing the bascline towards evaluating compatibility with a specific Stage-1 I algorithm.
examine and the simplest method to
The core of our baseline Stage-2 evaluation technique is given by the algorithm "Random Search with Parameter Sharing" (RSPS) proposed by Li & Talwalkar (2019) as a baseline method for eval- uating the contribution of weight-sharing in NAS. Additionally, sampling architectures uniformly random and training the supernet by adjusting the weights for single architecture at a time presents the simplest method for training supernet.
7

---

Under review as a conference paper at ICLR 2022
In order simulate the benefit of architecture weights without actually training architecture weights, we utilize the relationship between the architecture weights and architecture sampling. As we are using a NAS benchmark to support this evaluation method, have access to the ground truth per- formance of every architecture in the search space and can design an arbitrarily good distribution the architecture space. As such, augment RSPS training method by replacing the uni- form sampling with sampling distribution where the probability of sampling a model is inversely proportional its performance (so the sampling probability linearly decreases over the ranking by test accuracy), formulating and idealized version of discrete sampling based supernet NAS. We also utilize this distribution for evaluation of Stage-2 search algorithms which sample architectures from the supernet.
In addition to the RSPS model evaluated with (Ry and without Ry the biased sampling distribu- tion), we run each Stage-2 search algorithms on a randomly initialized model with and without the idealized sampler (N, and N6 respectively). We compute statistics corresponding to the reliance of the Stage-2 algorithm weights by taking the difference in performance on these baseline models. The measure correspond- ing to reliance the supernet having learned effective architecture parameters is computed as the difference between the performance of the biased sampling supernet and the uniform sampling su- pernet. The measure corresponding of individual candidate architectures using the supernet's weights, is computed as the difference between performance on the uniform sampling supernet and the randomly initialized supernet.
the architecture discovery and performance estimation via supernet shared
reliance obtaining information about the effectiveness
N_I- R
10.0
R_U-N_U RJINNI
7.5
Test Accuracy Gain
5.0
2.5
0.0
-2.5
-5.0
valid
synflow
jacob_cov
perturb
Figure 2: Comparison of Stage-2 search algorithm performance with given baseline models. "Ini- tial" refers to untrained model, "Uniform" refers to a model trained by training a uniformly ran- domly sampled architecture each batch, and "Biased" refers model trained by architecture sampled from an arbitrarily good distribution at each batch.
We examine four techniques for Stage-2 search: searching based on the validation accuracy obtained from the shared weight model ("vaalid"); the top two measure reported Abdelfattah et al. (2021) synflow and jacob_cov, and the perturbation based method demonstrated in Wang et al. (2021) ("per- jacbccobcco
turb"). The first 3 methods are sampling based random searches: sample of 100 architectures drawn from the architecture distribution provided by the supernet and then the canditate architec- tures are ranked by either the validation accuracy, synflow, or jacobian covariance, outputting the highest ranked architecture. The perturbation based method instead operates directly on the super- net, iteratively masking out operations and computing validation accuracies to prune operations one by one until final model is produced.
8

---

Under review as a conference paper at ICLR 2022
5 PUTTING IT TOGETHER
The concept of separate Stage I and Stage 2 search algorithms hinges on the assumption that these algoritm are not closely coupled in practice: that a better result may be obtained by swapping the Stage 2 search algorithm that was originally paired with given Stage search algorithm in an appropriate context.
The results of evaluating all of the reviewed Stage and Stage search algorithms in combination seem to support his assumption.
prune
valid
synflow
jac COV
perturb
darts
40.16 = 7.83
9.07 #1.65
9.48 = 2.33
10.03 0.20
14.34 3.66
darts-
9.36 + 2.17
11.90
0.65
6.29 10.46
8.16 = 2.03
10.17 + 1.16
snas
5.92 #0.25
8.31
= 3.36
6.36 0.10
7.71 = 0.27
7.24 # 1.54
gdas
12.55 + 8.54
17.40
4.39
7.03 +0.20
9.30 + 1.26
8.25 + 1.46
dirichlet
5.64 +0.00
6.35
0.08
6.04 0.28
7.35 = 0.75
5.88 0.22
Table 2: Combined results of all S1 and S2 search algorithms on CIFAR-10 in NAS-Bench-201
We also evaluate each Stage-1 and Stage-2 algorithm and their combinations on CIFAR100 and ImageNet16-120,, with the result provided the Appendix.
5.1 DARTS-SPACE RESULTS
prune
valid
synflow
Jac COV
perturb
darts
3.27 + 0.44
3.48
0.47
3.66 + 0.59
3.01 +0.23
3.25 + 0.10
darts-
2.66 +0.15
2.98
0.22
3.22
0.43
3.11 = 0.40
2.72 0.25
snas
3.10 + 0.14
3.19
+ 0.31
3.61 + 0.67
3.13 + 0.35
2.96 +0.32
gdas
3.04 + 0.39
4.01
0.62
4.94 + 0.29
3.17 + 0.23
2.90 +0.11
dirichlet
2.65 10.06
3.36
0.07
3.67 0.66
3.21 + 0.32
2.78 + 0.05
Table Combined results of all S1 and S2 search algorithms on CIFAR-IO in DARTS-space
CONCLUSION
In order to facilitate integration of recent advances in NAS and well as more robust evaluation in NAS research, we have reframed existing differentiable supernet NAS algorithms combinations of two separate search algorithms, the first being an implicit search via the process of training the supernet using architecture weights and the second being the process of selecting an architecture using the supernet. We propose two statistics approximating the quality of the weighting over the architecture space and the capacity of the supernet to provide information about the relative qual- ity of individual candidate architectures, as well as two corresponding statistics approximating the reliance of architecture selection methods on each of these components of the supernet.
We measure the performance of each of 5 Stage-1 I and 5 Stage-2 search methods in combination, noting that we did not find consistent combinations of algorithms optimal across different datasets, search spaces, and training conditions. We did find, however, that our statistics did elucidate prop- erties of the Stage-1 search algorithms which achieve top performance across search spaces, like the capacity of SNAS or DirichletNAS to estimate performance, that distinguished them from less successful Stage-1 search algorithms.
7 REPRODUCIBILITY
We trained each model single Tesla V100 GPU. We utilized the random seeds [1, 2, 3] in our case study and [10, 11, 12] in our main experiment. The source code utilized in the course this experimentation is provided in full at:

---

Under review as a conference paper at ICLR 2022
REFERENCES
Mohamed S. Abdelfattah, Abhinav Mehrotra, łukasz. Dudziak, and Nicholas D. Lane. Zero-Cost Proxies for Lightweight NAS. In International Conference on Learning Representations (ICLR), 2021.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understand- ing and simplifying one-shot architecture search. In Jennifer Dy and Andreas Krause (eds.), Pro- ceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 550-559, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. http httpttp::/ppccceeiiiii ngs ml
8a. html
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference Learning Representations, 2019. URL https:
77 //oppnreview.eee www.eeeeeoomiieeeeooommo h
Hy1VB3AqYm
Francesco Paolo Casale, Jonathan Gordon, and Nicolo Fusi. Probabilistic neural architecture search, 2019.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: theoretically inspired perspective. In International Conference on Learning Representations, 2021a.
Xiangning Chen and Cho-Jui Hsieh. Stabilizing differentiable architecture search via perturbation- based regularization, 2021.
Xiangning Chen, Ruochen Wang. Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. Dr(nas}: Dirichlet neural architecture search. In International Conference on Learning Representations, 2021b. URL
http:/iiwwwweeeeeeeeee.eeeeeeeee wttp://w.neeeooumme/edeooooo ? FFadd==FFaas66bmm
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE International Confer- ence on Computer Vision, pp. 1294-1303, 2019.
Xiangxiang Chu, Xiaoxing Wang, Bo Zhang, Shun Lu, Xiaolin Wei, and Junchi Yan. Darts-; Ro- bustly stepping out of performance collapse without indicators. In International Conference Learning Representations (ICLR), 2021.
Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019a.
Xuanyi Dong and Yi Yang. One-shot neural architecture search via self-evaluated template network. In The IEEE International Conference Computer Vision (ICCV), October 2019b.
Xuanyi Dong and Yi Yang. Nas-hench-201: Extending the scope of reproducible neural architec-
ture search. In International Conference on Learning Representations, 2020. URL https: //openreview wttp://ett//nmime HJxyykkKKK
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: survey, Jour- nal of Machine Learning Research, 20(55):1-21, 2019. URL v2C/18 598. html
Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. Dsnas: Direct neural architecture search without parameter retraining, 2020.
Guohao Li, Guocheng Qian, Itzel C Delgadillo, Matthias Müller, Ali Thabet, and Bernard Ghanem. Sgas: Sequential greedy architecture search. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Conference on Uncertainty in Artificial Intelligence, July 2019.
10

---

Under review as a conference paper at ICLR 2022
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, pp. 19-35, Cham, 2018. Springer International Publishing. ISBN 978----330-0124--55
Hanxiao Liu, Karen Simonyan. and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019. URL httl:ttttts///peeeeiiiii
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architec- ture optimization. In S. Bengio, H. Wallach, H. Larochelle. K. Grauman, N. Cesa- Bianchi, and R. Garnett (cds.), Advances in Neural Information Processing Systems 31, pp. 7816-7827. Curran Associates, Inc., 2018. URL http:://appeeee http://pappprssp..ccc..ccoo ntps.cc/pappr.. opt imi merizzata zation.
Joseph Mellor. Jack Turner, Amos Storkey, and Elliot J. Crowley, Neural architecture search without training. In International Conference on Machine Learning, 2021.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 2902-2911. JMLR.org, 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. AAAI, 2018.
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsich. Rethinking architecture selection in differentiable nas. In International Conference on Learning Representa- tions (ICLR), 2021.
Sirui Xie, Hchui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In International Conference on Learning Representations, 2019. URL https:// openreview htt.eewww.ee//oooeeeeeeeeeee hett:/oroun.ddxyygoooooo.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. (PC)- [darts}: Partial channel connections for memory-efficciet architecture search. In International Conference on Learning Representations, 2020. httphtttsss///ppeeveee//e net wtw.:ewww.ee/foummmmmmm
1d=BJ1$634tPr
Antoine Yang, Pedro M Esperanca, and Fabio Maria Carlucci. NAS evaluation is frustratingly hard. In International Conference Learning Representations, 2020. URL/httss://arxiv.
pdf 11tf/.91.... 12522.2... pdf
Q. Yao, Xu, W.-W. Tu, and Z. Zhu. Efficient neural architecture search via proximal iterations. In AAAI Conference on Artificial Intelligence, 2020.
Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase neural architecture search. In International Conference on Learning Repre- sentations, 2020. URL
Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. Towards automated deep learning: Ef- ficient joint neural architecture and hyperparameter search. In International Conference Ma- chine Learning AutoML Workshop, 2018. http:htts:://aaa www.aiii..../p.....aaoaa wttpp////////// 06906.
Fdf
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hut- ter. Understanding and robustifying differentiable architecture search. In International Confer- ence on Learning Representations, 2020. URL
H HDNDKDDD
11

---

Under review as a conference paper at ICLR 2022
Yuge Zhang, Zejun Lin, Junyang Jiang. Quanlu Zhang, Yujing Wang, Hui Xue, Chen Zhang, and Yaming Yang. Deeper Insights into Weight Sharing in Neural Architecture Search. arXiv e-prints, art. arXiv:2001.01431, January 2020.
Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. Econas: Finding proxies for economical neural architecture search. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. Bayesnas: A bayesian approach for neural architecttrr search. In International Conference on Machine Learning, 2019. URLhttp:
// //axxi..rr/aab///ee....... 04919
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. Learning transferable architectures for scalable image recognition. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8697-8710, June 2018. doi: 10.1 10.1100/C/....01..00
A APPENDIX
A.I REVIEW OF SEARCH METHODS PRIOR WORK
Figure 3: Stage-2 search methods of published NAS algorithms
We review a set of 20 published supernet NAS algorithms to determine the nature of the Stage-2 search method used. We find that existing techniques overwhelmingly utilize some form of archi- tecture weight or distribution pruning to select final architecture.
Algorithm
S2 Search method
Notes
Reference
DARTS
Arch prune
Suggests multiple trials for final arch
2019
DARTS-
Arch prune
2021 1
PC-DARTS
Arch prune
Combined prune on op, edge weights
2020
RSPS
Random search
2019
SNAS
Arch prune
Prune ops, no method for edge pruning
2019
GDAS
Arch prune
2019a
DrNAS
Arch prune
20216
ProxylesSNAS
Arch prune
Sample during training Uses trained sampler
2019
ENAS
Random search
2018
OSNAS
Random search
2018
DARTS-F PT
Perturbation
2021
P-DARTS
Arch prune
Progressive prune, alter search space
2019
SDARTS
Arch prune
Arch selection not directly addresssed
202T
R-DARTS
Arch prune
Final arch selected from multiple trials
2020
SETN
Random search
Uses trained sampler
2019b
BayesNAS
Arch prune
Entropy-based prune
2019
NASP
Arch prune
Proximal algorithm, applied during training
2020
SGAS
Arch prune
Progressive prune on multiple metrics
2020
NAOnet
Population search
Gradient based updates to population
2018
PARSEC
Arch prune
Sample during training
2019
DSNAS
Arch prune
Sample during training
2020
Table 4: Detailed account Stage-2 Search method review.
12

---

Under review as a conference paper at ICLR 2022
In the above table, note that when "sample during training" is included in the notes, this signifies that discrete architectures are sampled during the training process, while the final architecture selection method does not depend sampling. This is technically not true for GDAS, it samples single operation on each edge in same search space utilized by DARTS which does not allow an operation on every edge, resulting in sampling subset of the supernet that is not necessarily equivalent any architecture within the search space. SNAS and DrNAS also utilize sampling during training, but use relaxed samples, not discrete architectures.
As clarified in the notes, there is significant variation among the architecture techniques labeled as "arch prune." Many methods utilize more complex parameterizations of the architecture space than DARTS, but largely still prune based on magnitude. BayesNAS instead utilizes the entropy of the architecture parameterization as a pruning criterion, while SGAS develops two different criteria incorporating magnitude, entropy, as well as the histogram intersection of the architecture parame- terization over several updates. The technique used by NASP a proximal algorithm, which in the general case would be incorrect to describe as pruning, however the proximal algorithm used in this case applies the constraint of the bullhheresultnggroessssss
A.2 CIFAR-100 RESULTS
Architecture ranking performance, cifar100
Sampling probability distribution, cifar100
1.0
Sampling probability mass
1.0
0.8
Top arch ID
0.6
DARTS
0.6
DARTS
DARTS-
DARTS-
0.4
SNAS
0.4
SNAS
GDAS
GDAS
0.2
DIRICHLET
0.2
DIRICHLET
random baseline
random baseline
30
4 x 10
5x10
10
3x
x 10
5 5x10
10
Test err thresh
Test err thresh
Figure 4: Left: Top arch ID rate across test regret threshold groups of the architecture space. Right: Sampling probability mass across test regret threshold groups of the probability space.
prune
valid
synflow
jac cov
perturb
darts
61.03 + 0.00
36.41 + 3.19
33.24 + 33.24
36.44 + 1.15
44.13 - 5.60
darts-
32.46 + 4.93
44.68 + 7.47
28.51 - 28.51
32.18 + 2.46
35.11 = 3.55
snas
28.37 +: 1.33
36.95 + 6.32
29.08
29.08
32.47
3.77
33.65 + 3.35
gdas
83.58 + 8.80
52.97
+ 7.97
30.21 + 30.21
36.20
2.74
38.81 + 4.05
dirichlet
28.62 + 1.57
30.03 + 0.56
27.48 + 27.48
34.44.3.77
29.29 + 2.43
Table 5: Combined results of all SI and S2 search algorithms on CIFAR-100 in NAS-Bench-201
A.3
IMAGENET 16- 120 RESULTS
prune
valid
synflow
jac cov
perturb
darts
81.59 + 0.00
79.21
5.36
57.41 0.90
71.52 +: 4.47
65.04 + 1.22
darts-
77.03 4 4.70
77.49
5.21
54.68
1.44
61.61 + 5.11
64.65 + 3.03
snas
53.66 + 0.00
53.44
0.11
56.93
2.31
54.64 = 0.76
56.20 + 2.97
gdas
58.98 + 0.00
69.26
+ 4.79
58.62
0.82
63.39
+ 4.54
59.01 + 2.13
dirichlet
53.66 + 0.00
54.68
+ 0.17
53.66 = 0.00
56.15
1.46
56.04 = 3.07
Table 6: Combined results of all S1 and S2 search algorithms on ImageNet16-1D NAS-Benee 201
13

---

Under review as a conference paper at ICLR 2022
N. RJ- R
15.
RJI-NJI
Accuracy Gain
5
-10
valid
synflow
jacob cav
perturb
Figure 5: Comparison of Stage-2 search algorithm performance CIFAR-100 with given baseline models. "Initial" refers to untrained model, "Uniform" refers to a model trained by training uni- formly randomly sampled architecture at each batch, and "Biased" refers to model trained by an architecture sampled from an arbitrarily good distribution at each batch.
75
70
500
65
Test Accuracy
60
1500
Score
2000
50
3000
40
Moooin sous
pasiiccco cov
dinchlet jacoo cov
podeeeeepeeeeee secturr
gels sonnnnlo
estasvalle
ggtas paali
darts- synfoww
adlas-jacob. cov
darts varts-valid
gdas-penine perturs
darts-synflo synflow
darts-jacoob jacob. cov darts varts-valid
darts- darts-pertie perturb
darts jacojjccoboeeooe cay darts-perturr
dinchlet
snas
darts-
Figure 6: ordered by score.
Performance of Stage-1 I and Stage-2 search on CIFAR-100 algorithm combinations,
A.4 PREDICTING COMBINED PERFORMANCE
Separate measures for evaluating Stage-1 and Stage-2 search facilitate estimation of the quality of combining Stage-1 and Stage-2 methods. To empirically test the utility of these measures, we com- bine the separate evaluation methods via simple ranking technique generate unified ranking.
14

---

Under review as a conference paper at ICLR 2022
Architecture ranking performance, imagenet16-120
Sampling probability distribution, imagenet16- 1-1
1.0
Sampling probability mass
1.0
0.8
Top arch rate
0.6
DARTS
0.6
DARTS
DARTS-
DARTS-
0.4
SNAS
0.4
SNAS
GDAS
GDAS
0.2
DIRICHLET
0.2
DIRICHLET
random baseline
random baseline
6 10a
10
8 10
6 101
7 7x
8 10
Test err thresh
Test err thresh
Figure 7: Left: Top arch IDD rate across test regret threshold groups of the architecture space. Right: Sampling probability mass across test regret threshold groups of the probability space.
We compared this estimated ranking based on separate evaluation to the observed performance of each combination of search Stage-1 and search Stage-2 implemented NAS-Bench-200.
We denote the ranking of Stage-1 search algorithms by sampling probability as Ta1 and the ranking by correlation of shared-weing weight validation accuracy with benchmark test accuracy rys- For Stage- 2 models we compute the performance on the uniform baseline as r,2 and the performance on the biased baseline ræ2- Then compute score for each combination of Stage-1 and Stage-2 algorithm as follows:
SCOrE - -Tai -agl + 1) x T9i r32l
The scoring formula uses the difference between ranks to capture a mismatch in the capabilities of Stage- 1 algorithms and those relied by the Stage-2 search algorithms, adding I avoid attributing undue weight to the Stage- and Stage-2 algorithms sharing a rank. We then scale this difference by the Stage-1 ranks they approximations of the supernet model's performance estimation and top architecture discovery capabilities, which should correlate with overall performance. As the Stage-2 ranks computed using the difference in performance between pair of models rather than being a measure of model performance, we do not incorporate their magnitude. Given that NAS algorithms have been demonstrated which do not rely one of either learned distribution over architectures or ever using the supernet to evaluate individual architectures, we use multiplication to combine the two separate measures that a combination which succeeds at one of the two measures is scored above one which is mediocre at both.
After computing the scores, we then test the Stage-2 search algorithms on each Stage- algorithm, producing measurements of the benchmark test accuracy of the final architecture we are able to compare against the ranking implied by our scores, as shown in Fig. 7
These final scores, with the accompanying ordering, suggest that these measures are useful, but imperfect message for estimating the combined performance supernet training algorithm and architecture selection algorithm.
The purpose of this provided ranking is not necessarily to propose a specific method for ranking combinations of Stage-1 and Stage-2 algorithms but rather provide empirical demonstration of the usefulness of the proposed evaluation techniques and broader perspective of evaluating sep- arately the performance estimation, architecture sampling, and architecture selection. These evalu- ation techniques can utilized by researchers who develop new Stage-1 or Stage-2 algorithms to provide additional information about how their algorithm functions and how it might interact with other NAS techniques that is not provided when only the final test accuracy is reported.
15

---

Under review as a conference paper at ICLR 2022
95.0
92.5
90.0
1000
Test Accuracy
1500
87.5
Score
85.0
2500
82.5
3000
80.0
3500
77.5
dirichlet-synflow
dirichlet-jacob.. cov dirichlet-valio
pnichlet-perturt
darts-valio
darts-synflow
darts-jacob cov
mas--yynionn
cov
jacoo cou
aarts-valid
deeeeeee cou
parts-perturn
adlas-pertane
snasy becco
darts-pone
darts-
dartas
daree
Figure 8: Performance of Stage-I I and Stage-2 search algorithm combinations, ordered by decreas- ing score, with benchmark test accuracy plotted in blue on the left axis and score plotted in red on the right axis.
16


Reviews:
[{"review_id": "xpMR5ha4quH", "reviewer": "Reviewer_SuQ3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- This paper is written pretty well - all of the decisions are carefully thought through and explained clearly in writing. \n- The case study in Section 3 is interesting, and the ideas in Sections 4 and 5 are interesting and make sense.\n- The topic of this paper, better methods for evaluating differentiable NAS, can have high impact in the NAS community, because there are many different differentiable techniques proposed.\n\n## Points to be addressed\n- One of my biggest concerns is that all experiments are only done on NAS-Bench-201 (and only on CIFAR10 and CIFAR100, not ImageNet16-120). NAS-Bench-201 is by far the smallest of the popular NAS benchmarks, with 15k architectures, and only 6k unique architectures. Therefore, there is no guarantee that the experimental conclusions will generalize. It would make the paper much better if experiments were done on another benchmark, such as NAS-Bench-1shot1 which contains 300k unique architectures, or NAS-Bench-ASR or TransNAS-Bench-101, etc.\n- My other main concern is that the paper is a little bit too incremental for ICLR. While the ideas are nice, and I have not seen other papers talk about differentiable NAS in this way before, I am not sure it is quite enough content for ICLR. One thing that could help is if the ideas in this paper were used to find a conclusion that is even more novel, surprising, or insightful. (Or, use the new evaluations to propose a new algorithm other than Dirichlet-synflow?)\n- Two of the validation methods, synflow and jacob_cov, have not been used as \"Stage-2 Search\" for differentiable NAS algorithms before, as far as I am aware? So are there only two types of stage-2 search from existing literature?\n- The authors do not give a reproducibility statement or ethics statement. This was optional but encoraged by ICLR. I do not see ethical concerns, but having a reproducibility statment (and releasing code) would have made me feel more positive about the paper.\n- Citing relevant work is sparse. The most glaring emission was not citing jacob_cov [1], even though this is used in the experiments ([1] came out before Abdelfattah et al. 2021). Also, not citing PC-DARTS [2] even though it is mentioned by name in Section 2.1. Other papers that seem relevant to talk about include (at least) [3], [4], [5].\n- Finally (and least important), a few small comments about the figures:\n  - It’s much better for viewing to have the figures be vectorized. E.g. save the plots as pdf's.\n  - The x-axes in Figs 2 and 5 have to be cleaned up since there are overlaps.\n  - In Figs 2 and 5, there are two different types of purples used. It is better to use more distinct colors.\n\n\n[1] Mellor et al., Neural Architecture Search without Training, 2020.\n\n[2] Xu et al., PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search, 2019.\n\n[3] Yu et al., How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS, 2020.\n\n[4] Zela et al. Understanding and robustifying differentiable architecture search, 2020.\n\n[5] Pourchot et al., To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing, 2020.", "summary_of_the_paper": "The authors study the question of evaluating differentiable methods for neural architecture search (NAS). Differentiable techniques are popular in the NAS community, and many recent papers have given criticisms or improvements to various parts of the DARTS algorithm and related algorithms. The authors propose a new way of evaluating differentiable techniques, and new metrics, by separately evaluating architecture training and architecture selection, and evaluating the combinations of the two. The authors run experiments on NAS-Bench-201 and use their evaluation technique to identify the best combination of architecture training and selection.", "summary_of_the_review": "I like the topic of this paper: devising better tools for evaluating differentiable NAS. And I have not seen the ideas in this paper presented in quite this way, which could be impactful in the community. However, there are unfortunately the weaknesses of only using NAS-Bench-201 and being a bit incremental. I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "l4fX3vi3gD-", "reviewer": "Reviewer_XbXs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Developing better strategies to evaluate NAS algorithms is an interesting topic. Although it has received an increasing attention in the past couple of years, there remain open questions.\n- The proposed separation of differentiable NAS methods into two stages is reasonable.\n\nWeaknesses:\n\n1) Clarity: Altogether, I found the paper difficult to follow; it would benefit from a thorough revision.\n1.1) I did not quite understand what the paper was about (beyond differentiable NAS evaluation) from the introduction.\n\n1.2) I struggled to follow the description of the evaluation process for Stage 1 and Stage 2 (Sections 4.1 and 4.2).\n\n1.3) The motivation and description of the score in Section 5 remains unclear to me. For example, is $r_{\\alpha_1}$ a vector (as would the term \"ranking\" suggest), or a single value? Note that $\\beta_1$ was not defined in Section 4 (only $\\alpha_1$, $\\alpha_2$ and $\\beta_2$).\n\n2) Discussion of the results:\n2.1) In Section 4.1, the curves in Figure 2 (left) are briefly mentioned, without any discussion of what these curves mean. The plot in Figure 2 right is not even mentioned. I strongly recommend the authors to comment on these results.\n\n2.2) Similarly, in Section 4.2, the plots in Figure 3 are only briefly mentioned without any discussion of the results.\n\n2.3) Once again, in Section 5, the plot in Figure 4 (referred to as Fig. 5 in the text) is not truly discussed.\n\n3) Proposed score:\n3.1) As acknowledged by the authors themselves, Figure 4 suggests that the proposed score is not truly representative of the test accuracy. Additionally considering that, as mentioned before, the motivation behind this score was not very clear, I recommend the authors to remove this from their paper.\n\n4) Related work: The related work section seems incomplete.\n4.1) While Section 2.1 discusses some differentiable NAS methods, it ignores others even if they are used in the experiments, e.g., GDAS and DrNAS.\n\n4.2) The papers listed in the general discussion of NAS methods fails to mention a number of works, such as Guo et al., Single Path One Shot, 2019; Cai et al., Once for All, ICLR 2020; Zela et al., Understanding and Robustifying Differentiable NAS, ICLR 2020; Luo et al., NAO, NeurIPS 2018; Zhang et al., ShuffleNet, CVPR 2018; Chu et al., FairNAS, 2019.\n\n4.3) Other works that have proposed evaluation strategies for supernet-based methods would be worth discussing: Yang et al., NAS evaluation is frustratingly hard, ICLR 2020, Yu et al., An analysis of training heuristics in weight-sharing NAS, 2020.\n\n4.4) Minor comment: In the discussion of some of the paper, as well as in the experiments, the references to the papers are often missing.\n\n5) Case study: I am not entirely convinced by some of the arguments in the case study of Section 3.\n5.1) The authors make the statement: \"... without explanation of how or why these algorithms are able to positively interact.\" In fact, based on the results in Table 1 (which are not truly explained), these two algorithms do not positively interact; the best results are obtained by the method of Wang et al., 2021.\n\n5.2) The statement \"training with architecture weights does provide an improvement to the perturbation based selection method\" is not entirely correct. It does so when the perturbation-based method is used with fixed $\\alpha$, not if the perturbation method is used with the trained $\\alpha$.", "summary_of_the_paper": "This paper introduces an evaluation methodology for differentiable NAS methods. Specifically, it separates the search of these methods into two stages: Stage 1, which consists of training the supernet, and Stage 2, whose goal is to select an architecture. These two stages are evaluated independently, and the results of these evaluations are combined into a score whose goal is to provide information about how well any combination of one Stage 1 strategy with one Stage 2 strategy would perform.", "summary_of_the_review": "Although this paper tackles an interesting problem, it is currently unconvincing. Many parts of the paper need to be clarified, the results need to be discussed, the proposed score does not serve the desired purpose, and the discussion of the related work should be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dNFH-gyubKa", "reviewer": "Reviewer_JdoT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the studies are helpful, but it is unclear what are the key insights and innovations in this paper.\n\n— Strengths —\n\n1. Comprehensive description on the problems of supernet-based NAS;\n2. Interesting perspective to divide NAS into two-stage process, and study the behavior for each of them;\n\n—Weaknesses—\n\nMy main concern is the lack of insights: this paper is mostly focus on analyzing the two NAS stages, but it is unclear why those two metrics are important and what does it mean to future NAS search. Some of the results are obvious, such as Figure 3 (uniform tends to be slightly worse than others). It is unclear how we can better understand or further improve NAS based on the studies in this paper. Because of this, it is not obvious what is the core value of this paper.\n\nI also have a few  other concerns:\n1. Poor writing. Many concepts are explained well: for example, the abstract and introduction emphasize the novelty of eval metrics, but what are the metrics why they are novel?  To my knowledge, spearman correlation (used in Figure 2) has been widely used in NAS community.\n2. Limited experiments:  as an analysis paper, it provides limited data on a simple nas-bench-201 benchmark. I would expect more experiments on more search spaces and algorithms.", "summary_of_the_paper": "This paper studies the statistics of weight-sharing based neural architecture search (NAS) algorithms. In particular, it separates the training process into two stages: one is for supernet weight training and the other for architecture selection, and then it investigates the Spearman \ncorrelation and other statistics. All experiments are based on NAS-Bench201. ", "summary_of_the_review": "1. Somewhat interesting studies for NAS by dividing the search into two-stages\n2. Lack of insights;\n3. Limited experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R6rqny9Sot", "reviewer": "Reviewer_dryA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n-- Strength -- \n+ Clearly defining two stages of DARTS method is not novel in terms of research, however, this paper is written in a clear fashion and is good for new practitioner to NAS DARTS field to quickly understand the limit of DARTS based methods. \n+ Study different combinations of DARTS based methods in terms of their super-net training and evaluation is kind of interesting, and the newly proposed score seems to be efficient in their preliminary experiment on NASBench-201. \n\n\n\n-- Weakness -- \n\n- Overall the content is interesting but quite preliminary. NASBench-201 is a very small search space, (only has ~6k unique architectures). The findings on this small space may not generalize to realistic ones (e.g. DARTS space). In other words, I suggest the authors consider using recent NASBench-301 surrogate benchmarks to perform solid test to see the findings can or cannot generalize. \n\nAll figures except Figure 2 are quite large, and can be shrunk to a smaller size. This also indicates the paper seems to have not sufficient content to meet the ICLR standards. As an empirical evaluation paper, lacking enough juice for the community is a critical weakness. \n\n\n- Questions about ranking correlation computation on DARTS-based supernet\n\nAs we know, the differentiable architecture search approach defines a super-net as weighted summation, e.g. if we have two choice branch, b1, and b2, and architecture parameter \\alpha = [a1, a2], the output is usually equal to \\sum ai * bi. In this case, the super-net training depends on \\alpha and b1, b2. And in my experience, the a_i \\in [0,1] during training, and often is quite close to 1/n (n = number of operations) especially during the warm-up phase. \n\nHowever, the final architecture in NASBench-201 only contains 1 operation, i.e. ai \\in {0, 1} as a discrete value. In this case, if we want to select b1, b2 separately to compute the ranking correlation, do the authors set a1 = 1 and a2=1 accordingly? Is this really meaningful? Since the super-net is never trained with a1=1 or a2=1? This detail greatly impact the credibility of Figure 2 result. And I did not find much detail about this implementation. \n\nSpeaking of which, such ranking correlation analysis of DARTS-based method is done in Yu et al. Landmark regularization: Ranking guided super-net training in NAS, CVPR2021. Please see appendix Fig 1 for more details. (https://openaccess.thecvf.com/content/CVPR2021/supplemental/Yu_Landmark_Regularization_Ranking_CVPR_2021_supplemental.pdf) \n", "summary_of_the_paper": "This paper analyzed previous DARTS-based method and summarize these approaches into two stage, stage-1 that mainly trains the super-net, and stage-2 where the sampler try to select the best architecture out of the super-net. This paper proposes to evaluate the previous methods separately on NASBench-201.\n", "summary_of_the_review": "see above", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L9OGGTbsAdU", "reviewer": "Reviewer_eM7q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong Points**\n- It's a good point to study the integration of multiple DARTS variants, and see their joint influence.\n\n**Weak Points and Questions**\n- Missing reference for several methods like GDAS and DrNAS.\n- For the writing part, the authors just put forward the table or statistics, but lack of proper explanation and justification.\nI suggest that the authors add a conclusion part to discuss the analysis results for Section 4 and 5.\n- Although such analysis is well-motivated, it is unclear to me how this conclusion can be extended to other spaces.\nAfter all, 201 is a small space, and many NAS methods claim that they can find the optimal of the whole space.\nI understand that other space is hard to analyze, but still the potential impact of the analysis might be limited.", "summary_of_the_paper": "This paper doesn't propose a new method or algorithm. \nInstead, they perform a detailed empirical study for the two components of differentiable NAS variants, i.e., supernet training and architecture selection.", "summary_of_the_review": "This paper is a purely empirical evaluation paper, putting several advanced differentiable NAS methods together and study their joint effects.\nHowever, the impact can be restricted as all studies are performed on a toy space. Additionally, I think the writing of this paper has a large room to improve, the authors should add some conclusions and discussions rather than just showing the tables.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommended rejection, and I agree.
I encourage the authors to follow the reviewers' recommendation and resubmit.
Decision: Reject