SUMNAS: Supernet with Unbiased Meta-Features for Neural Architecture Search
Authors: Hyeonmin Ha
Ji-Hoon Kim
Semin Park
Byung-Gon Chun
Venue: ICLR 2022 Poster (2022)
Keywords: Neural architecture search

Abstract: One-shot Neural Architecture Search (NAS) usually constructs an over-parameterized network, which we call a supernet, and typically adopts sharing parameters among the sub-models to improve computational efficiency. One-shot NAS often repeatedly samples sub-models from the supernet and trains them to optimize the shared parameters. However, this training strategy suffers from multi-model forgetting. Training a sampled sub-model overrides the previous knowledge learned by the other sub-models, resulting in an unfair performance evaluation between the sub-models. We propose Supernet with Unbiased Meta-Features for Neural Architecture Search (SUMNAS), a supernet learning strategy based on meta-learning to tackle the knowledge forgetting issue. During the training phase, we explicitly address the multi-model forgetting problem and help the supernet learn unbiased meta-features, independent from the sampled sub-models. Once training is over, sub-models can be instantly compared to get the overall ranking or the best sub-model. Our evaluation on the NAS-Bench-201 and MobileNet-based search space demonstrate that SUMNAS shows improved ranking ability and finds architectures whose performance is on par with existing state-of-the-art NAS algorithms.

SUMNAS: SUPERNET WITH UNBIASED META- FEATURES FOR NEURAL ARCHITECTURE SEARCH
Ji-Hoon Kim
Hyeonmin Ha
NAVER AT Lab, NAVER Corporation; NAVER CLOVA, NAVER Corporation
Seoul National University hyeonmin. @ snu.ac.kr
genesis.king kim navercorp.com
Semin Park
Byung-Gon Chun
Yonsei University
Seoul National University: FriendliAI bgchun @ snu.acc.r
seminpark @ google.com
ABSTRACT
One-shot Neural Architecture Search (NAS) usually constructs an over- parameterized network, which we call supernet, and typically adopts sharing parameters among the sub-models to improve computational efficiency. One- shot NAS often repeatedly samples sub-models from the supernet and trains them to optimize the shared parameters. However, this training strategy suffers from multi-model forgetting. Training sampled sub-model overrides the previous knowledge learned by the other sub-models, resulting in an unfair performance evaluation between the sub-models. We propose Supernet with Unbiased Meta- Features for Neural Architecture Search (SUMNAS), supernet learning strat- egy based on meta-learning to tackle the knowledge forgetting issue. During the training phase, we explicitly address the multi-model forgetting problem and help the supernet learn unbiased meta-features, independent from the sampled sub- models. Once training is over, sub-models can be instantly compared to get the overall ranking or the best sub-model. Our evaluation on the NAS-Bench-200 and MobileNet-based search space demonstrate that SUMNAS shows improved ranking ability and finds architectures whose performance is on par with existing state-of-the-art NAS algorithms.
INTRODUCTION
Recent Neural Architecture Search (NAS) algorithms often train an ever-parameterized network called supernet obtain supreme sub-models by sharing parameters rapidly. In this process, the shared parameters are generally trained to reach state where better architecture discovery is possible through comparing the sub-models. Supernet training is usually conducted by sampling one or more sub-models and updating them with their gradients.
Since parameters shared, learning not be biased towards particular sub-model is crucial in training a supernet. However, a parameter update due to sampling a specific sub-model often causes bias for the sampled sub-model; the supernet forgets the knowledge learned from the previous sampling because of the biased update, and other sub-models share the same parameters can be degraded and underrated. [Benyahia al. (2019) referred this phenomenon as multi-model for- getting, and many current architecture search strategies have been designed without considering this phenomenon.
To mitigate this problem, it is important to make the supernet parameters learn unbiased features that are globally suitable for the sub-models. Therefore, we propose pesta-learning-baseee approach that enables the supernet to learn unbiased meta-features. We adopt model-agnostic meta-learning (MAML) (Finn et al. 2017 Nichol et al. 2018) during supernet training. MAML is designed to leamn the meta-features suitable for multiple tasks, and the trained meta-parameters are then quickly adapted to an unseen task through few-shot learning while reusing the learned meta-features. We
"Corresponding authors

---

apply this concept of MAML to supernet training by assuming the multiple tasks in MAML as learning for multiple sub-models in a supernet.
We call the proposed supernet training algorithm Superet with Unbiased Architecture Search (SUMNAS), given that it introduces the meta-lcarning principle to the multi- model forgetting problem. SUMNAS consists of two stages: supernet training and heuristic search with sub-model evaluation. In the supernet training stage, the parameters learn the meta-features for multiple sub-models, which can be considered as MAML's training meta-features for multiple tasks. The meta-parameters obtained from the supernet training can then be directly used without additional training for comparison between sub-models during the evaluation phase.
for Neural
To the best of our knowledge, this work is the first to utilize the meta meta-learning capability- unbiased meta-features- for accurately ranking sub-models in a supernet. There have been ap- proaches (Shaw et al. 2019 Lian et al. 2020; Wang et al. 2020; Elsken et al. 2020) that adopt meta-learning principles to train a supernet using various tasks and search for the best architecture for an unseen task with few data instances, which is the few-shot learning variant of NAS. They train the model parameters such that the models are robust to unseen datasets. On the other hand, we take a fundamentally different view of a task. A task usually refers to dataset, but we show in Section 3 that each sub-model within a search space can be regarded as a separate "task" that the supernet has to adapt to. We therefore apply meta-learning principles to make the model parameters robust to several different sub-models.
We evaluate SUMNAS with qualitative and quantitative experiments the CIFARIO and ImageNet datasets. We show that the architecture rankings SUMNAS predicts have a stronger correlation with the accurate rankings as compared prior NAS algorithms that use supernet. Besides, we observe better architecture search performance when an existing search methodology is applied to SUMNAS and show that SUMNAS parameters properly learn meta-features by investigating the performances of the sub-models.
2 PRELIMINARIES
One-shot NAS: In the context a supernet that shares parameters with its sub-models, the majority of NAS approaches use the supernet as a proxy to indirectly predict the performance of sub-models With the performance oracle, differentiable techniques (Liu et al., 2018 Cai et al. (2019), reinforce- ment learning (Pham et al. 2018), and heuristic methods (Guo et al. [019]] Chu et al. 2019)) are used to determine the best architecture. Although the weight sharing has sigmificantly improved search speed, it is not easy to provide accurate performance indicators when using supernet as a proxy for sub-model performance.
One of the most difficult challenges obtaining an accurate performance oracle is overriding knowl- edge learned by previously sampled sub-models. Researchers have recently introduced the practice of repeatedly updating parameters through one or more sampled sub-models (Guo et al.) 2019: Chu iet al. 2019 ei al. 2020). However, Benyahia et al. (2019) have shown that catastrophic forgetting occurs in the sequential learning of sampled sub-models. Catastrophic forgetting a phenomenon where a neural network forgets previously learned knowledge when learning new information. In sampling-basse based supernet training, repetitive sub-model sampling and training keep introducing new information into shared parameters. The knowledge learned with the previously sampled sub-models forgotten due to training subsequently sampled sub-models without considering former training- Knowledge overriding causes the predicted performance of the architectures to fluctuate depending on the frequency or the sequence of the sampled architectures, which leads to inaccurately predicted rankings. The forgetting problem in the stochastic training of shared supernet parameters is termed multi-model forgeetttnnno To alleviate this problem, [Benyahia et al.) (2019) and andZhang et al. (2020) add regularizer to prevent weights from deviating too far from the posterior distribution learned from previously trained sub-models.
Model-agnostic meta-learning: Model-agnostic meta-learning (MAML) is process of learning initialization for few-shot learning of an unseen task. Let the ith task be T, and 8 be suitable to model parameters for T.. MAML provides an objective function to obtain optimal meta-parameters by minimizing the loss function for each task T,:
i mmeoo = arg min ETSPET [CT. )
(1)
0
2

---

VI
VC(0
Evaluate
Update meska
Rankings
(a) Supernet training
(b) Sub-model evaluation
supernet training and search with evaluation of sub-models. SUMNAS learns meta-features which are globally suitable for the sub-models during supernet train- ing. In the evaluation stage, sub-models are sampled from the trained supernet and are evaluated to decide the outstanding architectures.
Figure 1:
Two stages of SUMNAS:
where is the network function consisting of 0', and is the loss function for each task T. The L. term in equation T Tallows the 0 calculated by few-shot learning with initialization O to learn a suitable feature for each task Tr and mmeta minimizes the expected loss corresponding the tasks. As consequence, the learned meta feature is generalized across tasks, allowing for fast learning of new tasks.
Finn et al. (017)) solve the objective function by repeatedly sampling tasks and updating parameters with gradient descents. They also suggest a first-order approximation avoid the computation of second-order derivatives. [Nchhll et al. (2018) propose a variation of gradient descent for meta- learning, which is called Reptile The first-order MAMI. carries out task optimization using only the last gradient calculated by the inner loop step, ignoring the second-order derivative. However, Reptile uses the average of the gradients calculated over multiple inner loop steps to update the parameters. These averaged values reflect the generalization of the gradients for the data.
3 SUPERNET WITH UNBIASED META-FEATURES FOR NEURAL ARCHITECTURE SEARCH
To tackle the multi-modes forgetting problem, we suggest a new supernet training strategy based on meta-learning, Supernet with Unbiased Meta-Features for Neural Architecture Search (SUMNAS). As mentioned, learning unbiased features that are suitable for sub-models in the supernet essential to alleviate the knowledge overriding. We take the idea of MAML. which learns meta-features suitable for multiple tasks, and apply it to supernet training so that SUMNAS learns such unbiased meta-features of sub-models. In this section, describe approach and explain how it can mitigate the multi-model forgetting problem. We then describe our supernet training algorithm.
3.1 META-FEATURE LEARNING
The approach introduce improve robustness against the multi-model forgetting problem is learning unbiased meta-features. SUMNAS has two stages: supernet training and heuristic search with sub-model evaluation, which are similar existing sampling-
NAS algorithms (Chu al.) 2019 Guo et al.) 2019 Li et al. (2020). as presented in Figure I First. SUMNAS trains parameters to learn meta- features (Figure I [a)), which are not biased to a specific sub-model and are suitable for sub-models, so training them does not overwrite previously trained knowledge. Afterward, the parameters are used to evaluate a specific sub-model during the search (Fiuure ] [b]).
Vanilla sampling-based supernet training optimizes strictly shared supernet parameters on sampled sub-models, and can be expressed as follows:
0,- arg min E,
where f is sub-mode sampled from distribution of sub-moded p(f), and T is given dataset. The parameter 0 is strictly shared among sub-mu models, and has learn model-specific features of sampled sub-models does not consider other
all of the sub-models. The training for
3

---

ones that have been previously sampled, and simply overrides knowledge acquired from previous training to optimize currently sampled sub-models.
In our approach, similar to previous works, parameters are shared during supernet training. How- ever, the parameters are trained to learn unbiased mecta-features Here, the unbiased meta-features make fair comparison possible by providing each sub-model with appropriate level of optimized parameters that are not overfitted for specific sub-model. Our meta-learning approach has been formulated follows:
02 = arg min E (C
s.t. 0- = Ay(C,T,0)
where Ay is an adaptation function optimizing the parameters for the sub-model f with a few inputs. With this optimization, parameters naturally learn meta-features.
This idea is based on MAML, which targets few-shot learning of an unseen task using adaptable parameters. MAML trains parameters to learn meta-features suitable for multiple tasks and uses the meta-parameterss to initialize few-shot learning. Training meta-parameters for weight-shared supernet is similar to training with MAML. We map a task used in meta-learning for few-shot learning to sub-model in NAS using a supernet, seen by comparing [Equation 1 with Viiuttinnation 3 In other words, the difference is that MAML minimizes the expectation of the loss across tasks, but SUMNAS minimizes the expectation of the loss across sub-models
Interestingly, we notice that learning the meta-features in supernet can be viewed as MAML of each operator in the supernet. We can convert Equation 3 into the following equation:
① meta arg min Eowp(o) (--(r [Cz [C o)]]
s.t. I= 3
7)
(4)
0- = A,
where p4i is a composition of sampled operators from i-th layer to j-th layer. A sampled sub-model f0ai1 feeds an intermediate representation f0:--1(x) to the operator o. There is distribution of the intermediate representation fed to the operator, and the distribution is determined by a sub- model sampled from p(f). The operator also has a target distribution of outputs learn determined by the sampled sub-model. In the MAML, examples fed to meta-learning model also follow distribution, and this distribution is a task T; sampled from pTT). Thus, inputs and targets of both multi-task meta-learning model and an operator in a supernet follow sampled distributions. We also aim to train an operator that works well on a set of intermediate representations provided by sampled sub-model, just MAML learns to work well task. That is, a model and a task in MAML can be mapped to an operator and a distribution of intermediate representations fed to the operator in supernet training. In other words, we can regard the distribution of intermediate representations as to the task for the operator, although the task for the operator is parameterized and optimizable differently from tasks in MAML. This implies that we can use any MAML algorithms without alteration to solve Davuatinua 4 Moreover, it supports the assumption that the success of the meta- meta-learning algorithms for few-shot learning will be transferred to NAS.
set of examples given by a sampled
However, supernet training and MAML have fundamental difference. MAML targets to train parameters for an unseen task, but supernet training aims to obtain parameters for pre-defined sub- models, which are known at train time, and participate in the training process. Therefore, after our supernet training, the supernet has knowledge of the sub-models, so additional training, such as fine-tuning for each sub-model not mandatory. In our empirical results, the additional fitting during evaluation may improve the performance of each sub-model, but the ranking performance stays more or less the same.
3.2 SUPERNET TRAINING ALGORITHM
In this section, we present our supernet training algorithm SUMNAS (Algoiithm 1p. SUMNAS repeatedly samples sub-models and leamss meta-features through the sampled sub-models with a MAML-insppred update function. Parameters are updated with the aggregated gradients of mul- tiple sub-models that share the same operators, unlike previous supernet training algorithms that immediately update parameters with the gradients of single sub-model for an operator.
4

---

Algorithm Meta-feature training
1: Input: training epochs N, search layer depth L, candidate ops per layer, training data D, steps for adaptation k; learning rate Eg for outer loop and E for inner loop
2: for i <- to N do
3: for X E D do
4: // Sample paths
5: for <- 0 to I do
6: CL + permutation of the index for the candidate ops of layer /
7: end for
8:
I/ Compute the Reptile gradients for the sampled paths
9:
for i f 0 to m do 61) Adaptation
AC (C. 0, (1,)))
11:
g co 0) // Reptile
12:
end for
13:
// Update the parameters
14:
0
- 0 g
g 6 0
16: end for
end for
Sampling sub-models As suggested by FairNAS (Chu al. 2019), keep strict fairness for fair training among operators, so all of the operators in the supernet are sampled the same number of times and updated simultaneously. To ensure this, SUMNAS sample m sub-models by sampling the candidate operators without replacement for each layer when we have m candidate operators and iterate the sampled sub-models for gradient computation (line 92 This sampling process is represented as permutation of the operators in each layer (line 7
Meta-learning The update function optimize single sub-model with stochastic gradient descent (SGD) is
0, 0 0; Vo, Lly, J(x; A (C, 0)))
where represents the parameters of the supernet, L is the loss function, A is the adaptation function with & steps and a sub-model f. We use SGD for the adaptation function. Computing the gradients of the loss requires computing the hessian of the loss, which is too intensive. Several methods can be used to avoid it, and we use the Reptile algorithm (Nichol et al. 2018) because of its simplicity. Reptile uses the difference between the parameters before and after adaptation (line IT instead the actual gradient. The parameter updates occur after computing and aggregating the gradients of all sampled sub-models (line 15
3.3 SEARCH ALGORITHM
Since we have done most of the heavy lifting in the supernet training phase, we can plug in almost any search algorithm for the evaluation phase. Possible search algorithms include, but are not lim- ited to, differentiable architecture search, reinforcement learning, or evolutionary search. For our experiments, we used the simple evolutionary algorithm proposed by Guo al. (2019).
4 EVALUATION
We evaluate SUMNAS on two search spaces - NAS--Bech-201 (Dong Yang 2020 on CIFAR- 10 (Krizhevsky et al., 2009) and MobileNet blocks on ImageNet (Russakovsky et al. 2015). For both search spaces, we compare sampled architectures' rankings predicted by SUMNAS with their reference rankings obtained from standalone training to evaluate the ranking ability of SUMNAS. We obtain the reference rankings of the sampled architectures for the from the HAS-Bench-22 201 and those for MobileNet blocks from manual training. We also report the best architectures that SUMNAS found in the search spaces. Furthermore, we analyze the sensitivity of SUMNAS with respect to the number of adaptation steps.
search space
5

---

Table 1: Ranking ability of five NAS algorithms and SUMNAS, and top-1 accuracy of architectures the algorithms found on CIFAR-10 under various FLOPs constraints.
Algorithms
Kendall Tau
Top-1 Acc. under FLOPs contraints
20M
<50M
<100M
Unlimited
DARTS
0.2987 + 0.34
89.36111.03
86.23.1.93
8..23
86.23314.93
GDAS
0.2231 + 0.23
1..9
92.10-0.11
92.3510.33
93.26110.22
SPOS
0.8008 0.02
90.34+0.03
92.33+00115
92.63+00.6
92.76=00.02
+WPL
0.6709 = 0.05
89.77=0.46
92.35=0.45
91.90=0.52
92.94=0.78
+NSAS
0.4769 = 0.11
86.85=0.11
92.13=0.12
92.23=0.10
92.23=0.10
FairNAS
0.7862 = 0.01
90.03:10.96
92.41110.15
92.34:10.13
92.40:10.09
Cream
0.8100 = 0.01
89.91
92.24-10.10
92.56++0.41
92.83-10.67
Ours
0.8451+-0.01
90.30++0.01
92.55++0.22
92.93-+0.48
93.09-+0.12
In the experiments, the methodology we to obtain the predicted performance (i.e., accuracy) of architecture from supernets is the following: we separately update the running statistics of batch normalization layers for the corresponding sub-model and then evaluate the calibrated sub-model. We describe the hyperparameters we used in [Appeddix DJ We also present the analysis on the sensitivity learning rates for inner and outer loops in Appendix
4.1 SEARCH SPACE
NAS-Bench-200 We adopt the NAS-Bench-201 search space evaluate the ranking performance of SUMNAS on CIFAR-10. NAS-Bench-201 is benchmark that contains the performance of all of the architectures in its search space CIFAR-10, CIFAR-100, and ImageNet- 16-- 1-120 (Chrabaszez et al. 2017). Each network in the NAS-Bench-22 search space is composed of 15 cells. Each cell has operators and each of them is one of candidate operators: zeroize, identity. 3x3 average pooling, 1x1 convolution, and 3x3 convolution. Cells in a network share the same architecture; therefore, there are 56 possible network architectures in the search space. Details of the search space are described in Appendix A
MobileNet blocks We conduct the experiments on ImageNet using the same search space as Prox-
ylessNAS (Cai et al.] 2019) and FairNAS (Chu et al. 2019). The operator in the search space is the bottlenecked and inverted residual block (Sandler et al. 2018) with squeeze and excitation (Hu
ctt al. 2018) and SiLU (swish-1) activation (Ramachandran et al, 2017 Elfwing al. (2018). This block has been used in various models (Howard et al.) 2019 Tan & Le 2019a| Cai et al. 2019 Chu
al. 2019 Li et al. 2020: Mei al. 2020). The operators' kernel sizes either 3, 5, or 7 and can have an expansion ratio of either 3 or 6. In total, there are candidate operators. A candidate network is sequential execution of the operators (i.e., an operator receives the output from single previous operator) and has 19 operators in total. Therefore, the search space consists of 639 possible candidate network architectures.
4.2 RANKING PREDICTION
We compare the ranking ability of the supernet trained by SUMNAS and other NAS algorithms on the NAS-Bench-201 search space and the MobileNet search space. To measure the ranking ability, use Kendall tau (-1 S 1) (Kendall 1938) between the reference rankings from standalone training and the rankings predicted by NAS algorithm. The details for Kendall tau are described Appendix
To compute Kendall tau, we use the rankings of sampled architectures instead of the entire architecture set defined by the search space. The search space of VAS-Bench-201 con- tains many architectures with very similar performances, and in many cases, multiple architectures with seemingly different edges could be reduced down to a single architecture due to the topolog- ical equilibrium, space is likely to produce a noisy list. Furthermore, its orders might change due to other factors, such which random seed used during training. Therefore sample architectures whose per- formances are different from each other so that the rank correlation could have higher confidence. Specifically, evenly split the accuracy range of NAS-Bench-20 into 400 intervals and sample
shown in Figure 2 Therefore, ranking every single architecture in the search

---

Topological
equilibriu
skip-connect
conv
3x3 conv
avg pool
Figure 2: An example of topological equilibrium. Multiple sub-models with different edges could be reduced down to a single architecture. Different combination of skip-connect and 3x3 conv encapsulated in the box produce exactly the same output and thus are all equal to the graph on the far right.
the best one from each interval; 320 architectures are sampled consequently because 80 intervals do not contain any architecture.
The reference accuracies and rankings of the sampled architectures are obtained from the NAS- Bench-201 benchmark. The rankings predicted by DARTS (Liu et al. 2018). ENAS (Pham al.] 2018), and GDAS (Dong & Yang (019) come from the checkpoints of supernets trained by the NAS-Bench-220 authors. For SPOS (Guo et al. 2019), FairNAS (Chu er al.] 20T9). and Cream PPng et al., 2020), we implement them and train supernets. We also realize WPL (Benyahia al., 2019) and NSAS (Zhang et al. 2020), which aim to resolve multi-model forgetting problem, on SPOS.
Table shows the ranking ability of the seven NAS algorithms and SUMNAS. We find that SUM- NAS shows more than 4.3% higher Kendall tau value, compared other satte-of-the-arr NAS algo- rithms, such as SPOS, FairNAS, and Cream. Moreover, we note that SUMNAS is superior to other NAS algorithms resolve multi-model forgetting, such WPL or NSAS. The outstanding ranking capability of our supernet resulted from the robustness our meta-feature learning achieves against multi-model forgetting. We also check whether other NAS algorithms can earn gain from more training time since SUMNAS consumes more time to process a single sampled sub-model because of the inner loop (line i0 Algorithm Ip. Specifically, the training time of SUMNAS is longer than FairNAS by multiple of the adaptation steps because SUMNAS performs forward and backward iterations for each sampled sub-model. For example, when the number of adaptation steps is two, the training time of SUMNAS is twice that of FairNAS. We train supernets with FairNAS as rep- resentative given more training time. FairNAS shows a little improvement,
the gain still minor comparing that of SUMNAS. The details of the result and analysis are presented in Appendix G
Table Ranking ability of FairNAS and SUMNAS on MobileNet search space and ImageNet.
Algorithm
FairNAS
SUMNAS
Kendall tau
0.7895
0.8526
We measure the ranking abilities FairNAS and SUMNAS on the MobileNet search space and ImageNet and present them in Table We sample 20 architectures evenly spaced in MACs from the search space and train them for 100 epochs to get the reference rankings. The Kendall tau of SUMNAS is 7.5% higher than that of FairNAS. SUMNAS more precisely predicts their rankings than FairNAS, as shown in the NAS-Bench-200 search space.
MobileNet blocks:
4.3 PERFORMANCE OF THE ARCHITECTURE FOUND
We present the best architecture SUMNAS found on the WAS-Bench-22 search space and the Mo- bileNet search space. For the NAS-Bench-22 search space and CIFAR-10, we evaluate all 56 sub- models in the supernets that SUMNAS and other five baselines NAS algorithms trained discover the best architecture. We then compare the architectures found. For the MobileNet- based search space and ImageNet, where it is computationally infeasible to evaluate all architectures, SUMNAS finds the best architecture through the evolutionary search algorithm described in Section 3.3 and the details for the algorithm and its settings are presented in Appendix C We present the perfor-

---

Table 3: Comparison of top-1 accuracies of the model found by SUMNAS, and existing models on ImageNet. Models marked with are trained with Auto.Augment (Cubuk et al. 2018).
Models
MACs
Params (M)
Top-1 Acc.
(M)
MobileNet v3
219
5.4
75.2
MnasNet-A2
340
4.8
75.6
MixNet-M
360
5.0
77.0
DNA-B
406
4.9
77.5
AtomNAS-C
363
5,9
77.6
SUMNAS-M
392
5.1
77.3
EfficientNet BO
390
5.3
77.1
SE-DARTS+*
594
6.2
77.5
FairNAS-A*
392
5.9
77.5
SUMNAS-S*
349
5.0
77.6
SUMNAS-M
392
77.8
SUMNAS-L*
440
5.3
78.2
mances of architectures various NAS algorithms found on their own search space and also report MobileNet-based architectures manually crafted compare them with ours.
NAS-Bench-201: Table 1 shows the top-1 accuracy of the architectures found by each algorithm on CIFAR-10. Without FLOPPs constraint, GDAS found the best architecture among the architectures discovered by six algorithms, and our algorithm found the one par. Considering that GDAS shows low Kendall tau, it clear that GDAS finds nigh-performance architectures but fails to properly rank architectures. This becomes problem for finding the best architectures that meet specific resource constraints, such as memory consumption, MACs, or latency. The performance of architectures found under various MACs constraints and shows this issue. The performances of architectures found by GDAS become worse than those from SUMNAS when such constraints are given, since SUMNAS more accurately rank all the models in the supernet
mentioned in [Section 4.2
MobileNet blocks: The performance of the architecture we found on ImageNet is presented in Table 3 along with the performances of other comparable models (Howard et al., 2019) Tan et al. 2019 an & Le 2019b; Li et al. 2020 Mei et al., 2020 Tan & Le 2019a; Liang et al., 2019 inverted residual blocks with squeeze and excitation and swish activation as primitive operators. FairNAS and SUMNAS have the same search space, but the main difference between them is our meta-learning principle. Therefore, the performance improvement of SUMNAS should be due to the effect of the meta-learning approach. The model we found also has higher performance than many other state-of- tfte-offthhe-r models, such as DNA and SE-DARTS. We also note that some of the other algorithms, such as AtomNAS and DNA, used finer- grained search spaces, with wider choices of the number of blocks and channel sizes of the bottleneck, or incorporates blocks that mix up various kernel sizes. We note that SUMNAS could adopt such search spaces obtain more superior architectures.
Chu et al. 2019). All of the models except for MixNet
Table The ranking ability of SUMNAS and top-1 accuracies SUMNAS found on
and CIFAR-10 with various adaptation steps. The results are the average of 3 runs.
Adaptation step
1 (=FairNAS)
2
3
4
Kendall tau Top-1 Acc.
0.7862++0.02
0.8361+0000 0.8362+00.00 92.40-10.09
0.8429 93.09-00.12
9..64.0.30
92.92=0.55
4.4 SENSITIVITY ANALYSIS
In this section, we analyze how the performance of our algorithm changes with varying numbers of adaptation steps. We train supernets created with the with combinations of four different adaptation steps {1, 2, 3, 4). Finally, measure the Kendall tau between the actual ranking and the ranking predicted by the supernets as in Section 4.2,, and the top-1 accuracy of the architectures SUMNAS found.
search space on CIFAR-10 10
8

---

the sub-models. Adapcation stap
77.57
79.79
70-
73.00
Validation Loss
3.5
60+
'9.4
Stdev Acc.
50-
7.64
40
30-
0.5
20
10- -
50
100
150
200
Epoch
Adaptation Step
Figure 3: Validation losses of 20 sampled sub-models during SUMNAS supernet training (left), and means and standard deviations of sub-model accuracies with the supernet parameters (right) with various number of adaptation steps on CIFAR- 10 and the NAS-Bench-200 search space.
Table shows the results of each configuration. The ranking correlation dramatically increases where the adaptation step increases from one to two, and meta-feature learning is applied. Then the correlation is further improved given more adaptation steps. Our meta-learning principle consis- tently enhances the ranking prediction regardless of the number of adaptation steps. Top-1 accura- cies of the architectures SUMNAS found also show similar trend to the ranking correlation. We also present the results with the larger number of adaptation steps than four in Appendix H
4.5 SUB-MODEL PERFORMANCE
Here, we evaluate how meta-feature learning affects the performances of sub-models in a supernet. Since meta-features in SUMNAS cover many sub-models (as opposed to MAML meta-features, which span multiple tasks), the overall sub-model performances using parameters of supernet trained with SUMNAS are more outstanding than those from a supernet trained without meta-featur learning. We evaluate the performances of sub-modek, with respect various adaptation steps to assess the effectiveness of SUMNAS.
The left of Figure shows the averaged validation losses of 20 sampled sub-models during super- net training by SUMNAS, with diverse adaptation steps. SUMNAS with adaptation step 1 (blue lines) does not teach meta-features and is the same FairNASS The validation losses dramatically decrease when meta-feature learning is applied, as shown in the results of adaptation step (red). Moreover, the losses further decrease as the number of adaptation steps increases. These observa- tions demonstrate that SUMNAS properly teaches meta-features that sub-models generally utilize.
We also measure the validation accuracies of all sub-models in supernets trained by SUMNAS, with the various number of adaptation steps. The right of Figure 3) presents the mean and standard deviation of the accuracies. The improvement in mean accuracy as the number of adaptation steps increases shows that the overall performance becomes higher with meta-feature learning. This result consistent with the loss results and also suggests that meta-feature learning is effective. Further- more, the increased standard deviations indicate that the performances of sub-models are distributed over a wider range. The wider distribution of the performances means performance differences between sub- mub-models become larger and implies higher resolution of ranking prediction.
CONCLUSION
In this paper, we have proposed new one-shot NAS algorithm which we call SUMNAS, to miti- gate the multi-modes forgetting problem. In order to achieve this, we took the idea of MAML and applied it to supernet training so that SUMNAS learns unbiased meta-features By explicitly helping the shared parameters to learn unbiased meta-features, we were able to ad- dress the multi-model forgetting problem efficiently. As a result, SUMNAS shows better ranking performance than other tfte-of-the-ar art supernet training algorithms, and the performance of the sub models it finds on par with hand-crafted models and those discovered by state-of- the-art NAS algorithms.

---

ACKNOWLEDGEMENTS
This work was supported by SNU-Naver Hyperscale Al Center.
REFERENCES
Yassine Benyahia, Kaicheng Yu, Kamil Bennani-Smires, Martin Jaggi, Anthony Davison, Mathieu Salzmann, and Claudiu Musat. Overcoming multi-model forgetting. In Proceedings of the 36th International Conference Machine Learning, 2019.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task
and hardware. In International Conference on Learning Representations, 2019. URL https:
//axxiv.org/pdf/lll22. 01t.//31121000.....12
Patryk Chrabaszez, Nya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint 17X.v0/707.088) 2017.
Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Fairnas: Rethinking evaluation faimess of weight sharing neural architecture search. arXiv preprint arXiv:1907.01844 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.095500 2018.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Xuanyi Dong and Yi Yang. Nas-bench-200 201:
Extending the scope of reproducible neural architecture search. International Conference on Learning Representations (ICLR), 2020. URL https: 77 //openzeview. wwt
HJ ZkBKDr
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. higmoid-weighter linear units for neural network function approximation in reinforcement learning. Neural Networks, 2018..
Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. Meta-learning of neural architectures for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML' 2017.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint 1944v:1904.004200, 2019.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE International Conference on Computer Vision, pp. 1314-1324, 2019.
Jie Hu, Li Shen, and Gang Sun. snneezze-nddexxcitee conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
networks. In Proceedings of the IEEE
Maurice G Kendall. new measure of rank correlation. Biometrika, 30(1/2):81-95 1938.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Changlin Li, Jiefeng Peng, Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xiao- jun Chang. Blockywisely supervised neural architecture search with knowledge distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
Dongze Lian, Yin Zheng, Yintao Xu, Yanxiong Lu, Leyu Lin, Peilin Zhao Junzhou Huang, and Shenghua Gao. Towards fast adaptation neural architectures with meta learning. Interna- fional Conference on Learning Representations, 2020.
10

---

Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping. arXiv preprint wwX.v::109.060000 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.00000 2018.
Jieru Mei, Yingwei Li, Xiaochen Lian, Xiaojie Jin, Linjie Yang, Alan Yuille, and Jianchao Yang. Atomnas: Fine-gradd graiegr end-to-end neural architecture search. In International Conference on Learning Representations, 2020.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint arXiv: 2018.
Houwen Peng, Hao Du, Hongyuan Yu, Qi Li, Jing Liao, and Jianlong Fu. Cream of the crop: Dis- tilling prioritized paths for one-shot neural architecture search. Advances in Neural Information Processing Systems, 33, 2020.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoe V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference on Machine Learning, 2018.
Prajit Ramachandran, Barret Zoph, and Quoe Le. Searching for activation functions. arXiv preprint arXiv: (rXiv::110.05944. 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision doi: 10. 10.1007/s1 12000/551263--1--811616
(IJCV), 115(3), 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 45200 2018.
Albert Shaw, Wei Wei, Weiyang Liu, Le Song, and Bo Dai. Meta architecture search. Advances in Neural Information Processing Systems, 12.11227-11337,2011
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114 14, 2019a.
Mingxing Tan and Quoe Le. Mixconv: Mixed depthwise convolutional kernels. arXiv preprint
arXiv: 0.955.9 2019b.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Jiaxing Wang, Jiaxiang Wu, Haoli Bai, and Jian Cheng. M-nas: Meta neural architecture search. In AAAI, 2020.
Miao Zhang, Huiqi Li, Shirui Pan, Xiaojun Chang, and Steven Su. Overcoming multi-model forget- ting in one-shot nas with diversity maximization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
11

---

A SEARCH SPACE
NAS-Bench-201 Architecture in the NAS-Bench-201 search space (Dong Yang, 2020) used the proposed method consists of 15 cells and the reduce block is inserted every 5 cells. The architecture has convolutional layer for the stem before the input layer is connected to the cell, and global pooling and fully connected layer the end, described in Figure A cell is densely connected DAG where each node is an intermediate representation and an edge is an operator that transforms the source node and propagates to the destination node, where the output of multiple incoming edges are combined into one. The reduce block doubles the channel size and halves the height and width of the feature map. Each cell has six edges and four nodes including the input and output nodes. The direction of an edge is from the i-th node to j-th node where i < j. Each edge is one of 5 candidate operators: zeroize, identity, 3x3 average pooling, Ixl convolution, and 3x3 convolution. All cells of an arbitrary architecture sampled from the search space have the same operator configuration corresponding to each edge. Therefore, there are (operator set #edge = 56 candidate architectures in the search space.
C=16
16
32
32
64
Pool & FC
Conv
Cell
X
Reduce
Cell
Reduce
Cell
X
Cell
Input
Output
skip connect
1x1 conv
3x8
avg pool
Figure 4: The architecture SUMNAS found on the NAS-Bench-201 search space and CIFAR-10 Krizhevsky al. 2009).
MobileNet Blocks We use the same MobileNet-based search space (Howard et al. 2019) as Prox- ylessNAS (Cai et al. 2019 and FairNAS (Chu al., 2019) in our ImageNet (Russakovsky et al. 2013) experiments for the sake of fair comparison. Each architecture in the search space is a sequen- tial execution two convolutional layers for stem, 19 MobileNet blocks, and a convolutional layer and fully connected layer for the head, depicted in Figure 5
Cxd2
32
102
1250
Comv
322
192
320
0-20
16
2200
Cony
Cony
St
Conv
&
Pool
Figure 5: The architecture SUMNAS found on the MobileNet-basee search space and ImageNet (Russakovsky al. 2015). MBe k l represents MobileNet block whose expansion ratio and kernel size is k. specifies the stride of the block. The grey line indicates where the channel size is changed and the number over the line is the channel size.
The MobileNet blocks are the bottlenecked and inverted residual blocks (Sandler et al. 2018 that squeeze and excitation (Hu et al., 2018) and SiLU (swish-1) 1) activation (Elfwing et al. 2018 Ra- machandran al. 2017) are mounted. Each architecture consists of blocks with different kernel sizes and expansion ratios. kernel size of block is either 3, 5, or 7, and an expansion ratio of block is either or 6; there 6 candidate blocks. Architecture in the search space has 19 blocks, and each block is one the candidate blocks, which means the search space has 619 candidate architectures in total.
12

---

Algorithm 2 Evolutionary Search
1: Input: generations G, population P, number of offsprings C, mutation probability MACs
contraint M
2: Output: the architecture with the highest validation accuracy that the supernet predict under given MACs constraint.
3: pop <- sample(P)
4: for + to G do
5: // Generate offssrings
6:
m - mutatelp, pop, C/2)
7:
c ー crossover(pop, C/2)
8:
off < muc C
9:
// Evaluate the offssrings and keep the best samples.
10: pop + top k(pop U off, P)
end for
12:
return top.k(pop.) 1)
B KENDALL TAU
In Section 4.2, we measure the ranking ability with Kendall (Kendall) 1938) between standalone validation accuracies from the benchmark and predicted validation accuracies from the supernet. Kendall tau a measure of rank correlation:
the rankings of the two variables. Given set of observations {I11, 11), (x/2- 122..... .... (I', Un)) of the two joint random variable X and Y. the measure is the ratio of the discordant pairs of the observations subtracted from the ratio the concordant pairs. In our experiments, 2; represents standalone accuracy of architecture from the benchmark and yi indicates the accuracy of the corresponding architecture that a supernet predicts.
similarity
The Kendall tau is formally defined as following:
nh =1[(( B:), (Ie,, Mj) B;))[i < j.
((x; < (z) A (y: < vj))
V ((a; > x2) A (yi > 0;))}
nd = K(((,, (ا));;))) <
(6)
((r < xz) A (yi > 4n))
V ((x, xz)) (v 1)))H
Th nd
The number of concordant pairs n. and the number of discordant pairs na is (9) at most; therefore, -1 < < r > O, there are more concordant pairs than discordant pairs and this means the two rankings partially (or perfectly when 1) agree. On the other hand, when < 0, the two rankings disagree.
C SEARCH ALGORITHM
To discover the best architecture on the MobileNet-based search space and ImageNet (Russakovsky al. (2055), we adopt a simple evolutionary algorithm same as SPOS (Guo al. 2019). The con- crete algorithm is described in |Algrrithm 2) The search algorithm repeatedly generates offsprings from the population using mutation and crossover, and keep the top / samples among the popu- lation and the offsprings for the population of the next iteration. The mutation randomly changes each operator to one of the others with the probability p, and the crossover randomly mixes two ar- chitectures in the population. The search requires several hyperparameters, and generations 30, population = 20, the number of offsprings = 64, mutation probability = 0.1, and MACs constraint M 400 M) for the experiment.
13

---

Table 5: Search space for the hyperparameters. LRmuut is LRinner X Rouueer-
CIFAR-10
ImageNet
L.muatt
{0.01, 0.05, 0.25) [0.2, 1, 15} {constant, cosine) {constant, cosine)
(0.3)
LoRuter
{0.5. 1}
L Route scheduler
(cosine)
LRinner scheduler
{constant]
Table 6: Hyperparameters we use for the experiments.
CIFAR-10
ImageNet
Outer loop
T. RoRuer
LRoster scheduler
cosine SGD 4e-5
cosine
Optimizer Weight decay
SGD
4e-5
Inner loop
RRnner 0.05
0.3
LR innnner scheduler
constant SGD
constant SGD
Optimizer
Opt. momentum
0.9
0.9
Adaptation step
{1,2,3,4)
2
D EXPERIMENTAL SETTING
NAS-Bench-201 search space and CIFAR-10 For the experiments of the NAS-Bench-201 search space (Dong & Yang 2020) and CIFAR-10 (Krizheskyy et al., (009). we train the supernets on the entire training set of CIFAR-10. On the test set, the hyperparameters are tuned and the reported Kendall tau and accuracies are measured. We also search for the best architecture on the test set. We tune the hyperparameters where the number of adaptation steps is 4, and the hyperparameter search space is presented in Table 5) We choose the best hyperparameter set which shows the best Kendall tau for the sampled architectures. The searched hyperparameters are listed in (Tall 6) The tuned hyperparameters are also used in SUMNAS with other numbers of adaptation steps, but the learning rate for the inner loop LRunner scaled so that actual parameter update sizes keep same by multiplying 4/ adaptation_step. SPOS and FairNAS adopt the hyperparameters for the outer loop of SUMNAS, but use LRmuit for the learning rate, which is LRguter X LRinner with scaling as SUMNAS where adaptation step one.
MobileNet-basse search space and ImageNet For the experiment of the MobileNet- search space and ImageNet (Russakovsky et al., 2015), tune hyperparameters and search the best chitecture using a validation set that includes about 50K examples sampled from the training set. Due to the time-consuming task of obtaining the standalone (from scratch) validation accuracies of the sampled architectures to compute Kendall tau, we use the average validation accuracy (from shared parameter) determine the best performing hyperparameter set. The hyperparameter search space is described in (aale 5) and Table 6)presents the selected hyperparameters. We also adopt task batching of MAMI. and aggregate the gradients of three sub-mot mub-models for each operator to update its parameters.
E PERFORMANCE SENSITIVITY TO THE LEARNING RATES FOR INNER AND OUTER LOOPS
We analyze the sensitivity of SUMNAS performance to the inner and outer learning rates to show that SUMNAS does not demand much extra effort tune the additional learning rate. We measure
14

---

Table Mean and standard deviation of the Kendall tau over the outer learning rates for each LR scale.
LR scale
0.01
0.05
0.25
mean
0.8065
0.8405
0.8372
stdev
0.0329
0.0145
0.0164
the mean and standard deviation of the Kendall tau over the outer learning rates for each LR scale we suggested in Appendiix DJ The results are presented in (Table 7
For LR scales of 0.05 and 0.25, the average of the Kendall taus similar, but at 0.01, the perfor- mance is obviously degraded. However, we conducted an ablation study in a sufficiently wided LR scale range, and it be that the degradation insensitive enough for SUMNAS to the LR scale. The standard deviations show the sensitivity with respect to the relation between the outer and inner learning rates, Standard deviations are usually small, and SUMNAS is robust the relation of LRs when we take sufficient LR scales, such as 0.05 or 0.25.
F DISCOVERED ARCHITECTURES
Figure 4 and Figure 5 visualize the architectures SUMNAS found on the MAN-Bench--21 search space (Dong & Yang 2020) and the MobilNet-basee search space (Howard et al. 2019 respectively.
G RANKING ABILITY OF FAIRNAS WITH MORE TRAINING TIME
Table 8: Ranking ability of FairNAS with various training epochs.
Epochs
200
400
600
800
Kendall tau
0.749:10.01
0.755110.02
0.702+0001
0.691110001
We also check whether FairNAS can earn gain from more training time, since SUMNAS consumes more time to process single sampled sub-model because of the inner loop (line 10jin Algorithm
Table presents the Kendall tau of FairNAS when more training epochs are given. Note that the learning rate we use here is different from one present Apppnddi D The supernet for FairNAS in Table I is trained for 200 epochs, so the Kendall tau for more than 200 epochs shows the loss gain from more training time. In 400 epochs, the performance improves for 0.8%. However, the performances degrades 600 or 800 epochs. We conjecture that the degradation results from cosine learning rate decay, which was used in the FairNAS paper. Increasing the number of epochs might have resulted in longer training time in high learning rates, which might have unstabilized the overall training. Therefore, FairNAS improves slightly with more training time, but the improvement is minor comparing to the gain from SUMNAS.
H SUMNAS WITH MORE ADAPTATION STEPS
Table 9: The ranking ability of SUMNAS and top-1 accuracies of architectures that SUMNAS found, given the larger number of adaptation steps than four. The results are averaged over runs.
Adaptation step
5
6
Kendall tau Top-1 Acc.
0.8339+0001 93.0550..47
0.8379+0.00
93.03300.44
Table 9 presents the ranking ability and the performance of the searched architecture on CIFAR-10 and TAS-Bench-201 with the number of adaptation steps above four. The Kendall tau decreases
15

---

with the number of adaptation steps when it is larger than four, As the number of adaptation steps increases, each sub-model has to incorporate more data points for a single update, which increases the standard deviation of the gradient that needs to be updated according to the sensitivity of the sub-model. Large standard deviation above a certain level is likely to interfere with learning the unbiased meta feature. Consequently, excessive adaptation step will have the effect of lowering the ranking ability.
16


Reviews:
[{"review_id": "UEl7rnQbBfb", "reviewer": "Reviewer_z8FZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper is in general well-written and easy to follow. The idea is conceptually simple and part of the results (especially the ones related to Kentall’s tau) are looking promising.\n\nWeaknesses:\n\nResults in Table 4 are supposed to be critical as they aim to verify the effectiveness of the proposed MAML-style supernet training strategy. However, they're not yet fully convincing because some key ablation studies are missing, such as the results without any adaptation step. It would also be interesting to see whether the performance would saturate as we increase the number of adaptation steps further above 4.\n\nI also believe the search space of NAS-Bench-201 is probably too small to make any robust conclusions. While the authors did conduct ImageNet experiments as additional evidence, results there do not appear strong as compared to existing methods. For example, the new method achieved only 0.3% improvement over FairNAS on ImageNet at the same MAdds (Table 3) despite substantially improved Kendall’s tau (Table 2). This makes me wonder whether Kendall’s tau is the right metric to measure the effectiveness of the proposed method. One hypothesis that might explain those observations is that the method indeed improved rank correlations for the vast majority of the \"average\" architectures but not the calibration for the small number of high-performance ones. An analysis on this direction would be helpful.\n", "summary_of_the_paper": "The paper proposed an improved training strategy for oneshot NAS supernetworks. The key idea is to view the training of each subnetwork as a \"task\", and then to apply an MAML/Reptile-style meta-learning scheme to ensure efficient cross-task adaptivity. Experiments on NAS-Bench-201 and ImageNet show improved calibration between the supernet’s predictions and the architectures' true rankings.", "summary_of_the_review": "While I'm not fully convinced by all the experimental results (see comments above), this is an overall interesting paper because the method is conceptually simple and the results are looking promising (at least in terms of rank correlations). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "93uboNaRXw-", "reviewer": "Reviewer_m3L7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.\tThe paper is well-written and the analysis gives insight.\n2.\tIt addressed an important problem——multi-model forgetting problem by an elegant algorithm based on meta-learning.\n\nWeakness:\n1.\tThe top-1 performance (Table 1 and Table 3) is not impressive although the ranking correlation seems to be improved. \n2.\tThe training time of supernet should also be listed for comparison.\n3.\tThe comparisons are based on less competitive baselines (e.g., I am curious about the performance of KD based method, like BigNAS (https://arxiv.org/abs/2003.11142), Cream (https://arxiv.org/abs/2003.11142), etc.\n\n\n==========================after rebuttal====================\nI appreciate the authors taking the time, attempting to address the comments through more details and new experiments. After reading the authors' response and revisions, I think my concerns are almost addressed. If the authors could present the comparison of training (search) time in the main Table, that will be better. I lean to accept this work. ", "summary_of_the_paper": "This paper addressed the multi-model forgetting problem in supernet training by a supernet learning strategy based on meta-learning. The evaluation on the NAS-Bench-201 and MobileNet-based search space demonstrated an improved ranking correlation and promising performance.", "summary_of_the_review": "This paper is well-written and the analysis is insightful. However, some related papers are not taken in the comparison, such as BigNAS and Cream.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-ntxyDq8Y0C", "reviewer": "Reviewer_LNNQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper utilizes a simple while interesting meta-learning strategy to train the supernet in one-shot NAS, and the experimental results on the NAS-Bench-201 and MobileNet show the effectiveness of the proposed method. However, it seems that the proposed method is a direct application of MAML on one-shot NAS. More importantly, the experimental results can hardly support the claim. For example, although the proposed method achieves a high Kendall Tau value with 0.84 on NAS-Bench-201, we should notice that this search space is very simple, and even without any training, the zero-cost NAS [1] can achieve a similar Kendall Tau value. In addition, paper [2] also consider meta-learning for the neural architecture search.\n\nThe marginal improvements in experimental results could hardly convince me that the proposed paradigm can benefit the one-shot NAS. I am confused on the right of Figure 3, where the proposed method SUMNAS achieves 79.79+-9.43 accuracy while Table 3 only achieves 77.3. Please clearly state the difference between the two settings.\n\n[1] Zero-cost Proxies for Lightweight NAS\n [2] M-NAS: Meta Neural Architecture Search", "summary_of_the_paper": "This paper focuses on improving the predictive ability of supernet in one-shot NAS, which considers a meta-learning strategy to tackle the multi-model forgetting issue. Through utilizing MAML to minimize the expectation of the loss across sub-models, the supernet can learn unbiased meta-features to improve the ranking ability.", "summary_of_the_review": "The idea is not so surprising and the experiments can not convince me. I tend to reject this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-_jZzG3hMCT", "reviewer": "Reviewer_Yfxi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\nI generally agree with the proposed idea. Naively train the supernet will introduce unfair competition among operators. Instead, the supernet's weights should be adaptive to its subnetworks. Correlation and Accuracy on NAS-Bench-201 and MobileNet space are competitive.\n\nQuestions:\n1. How sensitive Algo.1 is to the inner and outer learning rate?\n2. How does the index permutation work is not well explained (Line 4 in Algo.1). Why is it required? Is this for the purpose of randomness during meta-training? Will there be any gap if no such permutation?\n3. Is $g$ aggregating parameter changes ($\\theta - \\tilde{\\theta}$) from all subnetworks (Line 11 in Algo.1)? Will the method work if only part of subnetworks are sampled?\n4. I understand that Reptile is efficient since the calculation of Hessian is avoided. I am curious if the authors have ever tried other meta-learning methods, such as the original MAML? If we do not consider computation cost, will the original MAML also work?", "summary_of_the_paper": "This work targets better ranking performance after supernet training in NAS. The authors leverage meta-learning to make the shared supernet weights adaptive to randomly sampled subnetworks. Experiments are conducted on NAS-Bench-201 and MobileNet space.", "summary_of_the_review": "I generally like this idea, although seems straightforward, applying meta-learning during training the supernet is a reasonable and correct strategy. The analysis and experiments are also very comprehensive. I vote to accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a supernet learning strategy for NAS based on meta-learning to tackle the knowledge forgetting issue. Forgetting happens when training a sampled sub-model to optimize the shared parameters overrides the previous knowledge learned by the other sub-models. The main idea of the paper is to consider training of each subnetwork as a task, and then apply MAML to ensure efficient cross-task adaptivity. While the reviewers found the proposed method mainly an application of the existing meta-learning strategies to one-shot NAS, additional experimental results provided by the authors mostly convinced them about the effectiveness of the proposed method.
Decision: Accept (Poster)