MOTE-NAS: Multi-Objective Training-based Estimate for Efficient Neural Architecture Search
Authors: Yuming Zhang
Jun Wei Hsieh
Xin Li
Ming-Ching Chang
Chun-Chieh Lee
Kuo-Chin Fan
Venue: NeurIPS 2024 poster (2024)
Keywords: neural architecture search
few-cost
training-related estimate.
Primary area: optimization_for_deep_networks
TL;DR: a few cost estimate for NAS

Abstract: Neural Architecture Search (NAS) methods seek effective optimization toward performance metrics regarding model accuracy and generalization while facing challenges regarding search costs and GPU resources. Recent Neural Tangent Kernel (NTK) NAS methods achieve remarkable search efficiency based on a training-free model estimate; however, they overlook the non-convex nature of the DNNs in the search process. In this paper, we develop Multi-Objective Training-based Estimate (MOTE) for efficient NAS, retaining search effectiveness and achieving the new state-of-the-art in the accuracy and cost trade-off. To improve NTK and inspired by the Training Speed Estimation (TSE) method, MOTE is designed to model the actual performance of DNNs from macro to micro perspective by draw loss landscape and convergence speed simultaneously. Using two reduction strategies, the MOTE is generated based on a reduced architecture and a reduced dataset. Inspired by evolutionary search, our iterative ranking-based, coarse-to-fine architecture search is highly effective. Experiments on NASBench-201 show MOTE-NAS achieves 94.32% accuracy on CIFAR-10, 72.81% on CIFAR-100, and 46.38% on ImageNet-16-120, outperforming NTK-based NAS approaches. An evaluation-free (EF) version of MOTE-NAS delivers high efficiency in only 5 minutes, delivering a model more accurate than KNAS.

MOTE-NAS: Multi-Objective Training-based Estimate for Efficient Neural Architecture Search
Yu-Ming Zhang'
Jun-Wei Hsieh² Xin Li
Ming-Ching Chang?
Chun-Chich Lee! Kuo-Chin Fan!
National Central University Uniiersity at Albany
Natiooal Yang Ming Chiao Tung University
1085220360g.o ncu. edu. juhsieh@nycu. edw.:nnccuuuuu.u..e.mmm edu. tw
Abstract
Neural Architecture Search (NAS) methods seek effective optimization toward performance metrics regarding model accuracy and generalization while facing challenges regarding search costs and GPU resources. Recent Neural Tangent Kernel (NTK) NAS methods achieve remarkable search efficiency based on a training-free model estimate. However, they overlook the non-convex nature of the DNNs in the search process. In this paper, we develop Multi-Ohjective Training- based Estimate (MOTE) for efficient NAS, retaining search effectiveness and achieving the new state-of-the-art in the accuracy and cost trade-off. To improve NTK and inspired by the Training Speed Estimation (TSE) method, MOTE is designed to model the actual performance of DNNs from macro to micro per- spective by drawing the loss landscape and convergence speed simultancously. Using two reduction strategies, the MOTE is generated based on reduced ar- chitecture and reduced dataset. Inspired by evolutionary search, our iterative ranking- based, coarse-to-fin fine architecture search is highly effective. Experiments on NASBench-201 show MOTE-NAS achieves 94.32% accuracy on CIFAR-10 10, 72.81 11% CIFAR-100 100, and 46.38% on 1mageNet-16-11 12,, outperforming NTK- based NAS approaches. An evaluation-free (EF) version of MOTE-NAS delivers high efficiency in only minutes, delivering model more accurate than KNAS.
Introduction
Neural Architecture Search (NAS) [52] tackles the challenge of automating the design and search for suitable neural network architectures in many applications. NAS approaches mainly comprise two stages: a model search stage dedicated to identifying promising candidates within the architecture search space, and an evaluation stage where candidate performance is assessed. In the search stage, the search space can be exponentially large. To reduce search complexity, the cell-based tabular search space [49, 10, 38] is widely considered. Reinforcement learning [52, 2, 39] and evolutionary algorithms [27, 31, 33, 46, 9] are also used to accelerate the search process. However, the primary cost of NAS lies in the evaluation stage, where candidate models must undergo intensive training until convergence to obtain a precise performance assessment. This incurs significant time costs (e.g., NASBench-201 requires 3-10K GPU seconds for convergence after 200 epochs). So, various proxy estimates (e.g., zero-cost proxy [1] and training speed estimation (34J) have been developed to rank candidates, mitigating computational demands for model evaluation.
Recently, several estimates based on NTK have been proposed, including TE-NAS [6], KNAS [47], and Eigen-NAS [51]. NTK- estimates serve as condensed representations of gradients and their correlations. The NTK theory aims to macro-model the actual performance of Deep Neural Networks (DNNs). It assumes that the performance of an infinite-win width DNN can be fully described by the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

cunsaida
% 16 %
NTN
Speidd 10te
Speedient
Speediart
0.
Epoch2
Epocht
Epocht
(a) CIFAR-10
(b) CIFAR-100
(c) ImageNet-16
Figure I: Post-training rank correlation for randomly chosen 1000 candidates on NASBench-201.
The predictive performance of the proposed two terms gradually improves as epochs increase.
NTK at initialization and that the NTK's value remains unchanged after training [19]. Consequently, NTK-based estimates can predict the actual performance of candidate architectures without training. However, real DNNs have limited width and exhibit a highly to encounter significant nonlinear changes during training and resulting in limitations in accurately predicting the actual performance of candidates. Fig. shows the NTK suffers unstable performance during training.
nature, leading the NTK
To address this macro-modeling issue, we propose novel landscape term that leverages the idea from the study [12] to linearly combine the differences between the initial weights and the post-training weights of the candidate architectures, which allows us to capture the non-convex nature of these candidates by the landscape slice. If the landscape is flatter, the candidate's performance tends to be better, as more readily converges aspect have been conducted to model this issue, such as Snip [20], Grasp [41], and SynFlow [40, 1]. which use gradient change to predict the performance of candidate architectures. As the gradient integrated into the model training, it may reflect the current effectiveness of changes in the model training. Similarly, TSE [34] directly sums up the training loss to represent the convergence speed to predict candidate performance. In summary, these methods are more intuitive. Although they may not theoretically capture the macroscopic non-convex nature of DNNs, practice, as shown in Fig. 1, the performance TSE even exceeds NTK. This observation inspired us to propose speed term that sums the training loss per unit of time, providing a microscopic description of the convergence speed of candidates.
the global optima. Furthermore, studies from micro-
This paper introduces a Multi-Objectie Training- based Estimate (MOTE) that considers landscape term for macroscopic view and speed term for microscopic view to predict the performance of candidates a joint optimization. This dual perspective offers comprehensive consideration and an accurate estimate for candidates. Furthermore, we introduce two reduction strategies to reduce the time costs by generating MOTE, which consists of landscape and speed terms. We also present MOTE by integrating the evolutionary algorithm [32], named MOTE-NAS. This design is based on coarse-to- iterative procedure for architecture search. In the search stage of MOTE-NAS, we maintain several dozen to several hundred promising candidate architectures in the pool. In the evaluation stage, we first select the top-K K architectures based on the MOTE, then select the best architecture by the early stopping version of the test accuracy. We further develop a stripped-down. evaluation- free version named MOTE-NAS-EF, which achieves high efficiency and can finish a NAS in merely eight minutes.
MOTE-NAS outperforms mainstream NTK-based NAS methods. Fig. 2 compares it with TE- NAS [6], KNAS [47], Eigen-NAS [51], and Label-Gradient Alignment (LGA) [30] on CIFAR-100 of NASBench-200 201. The accuracy of the final architecture discovered by MOTE-NAS is significantly superior to other methods. In our experiments, MOTE-NAS achieved 94.32% accuracy on CIFAR-10. 72.81% % on CIFAR- 100, and 46.38% on ImageNet-166 16-120. The evaluation-fia version, MOTE-NAS- EF, achieves results comparable to KNAS's, s, where the search is completed in only eight minutes. The technical contributions of this work are summarized as follows:
Our proposed NAS approach utilizes an efficient training- based estimate to optimize landscape view and convergence speed objectives jointly. This design comprehensively captures the non- convex nature of DNNs from a maero perspective and monitors the convergence speed from a micro perspective, enabling precise actual performance predictions for desired architectures.

---

accuracy of finnal architacture
0.520
MOTE S.
MOTE NAS EF NASWOT
mmn
(g/G) (1-g/G]e
KNASIK-2O.WN Eigen-NASKew2U 20) RS+IGA REA
0.720
Landscape Term
0.15
en
Speed Term
0.705
Training Loss
0.0
2.5
Total costithousand Seconds)
7.5 16.0 12.5
15.0
17.5
20.0
(a) Two terms of MOTE
(b) MOTE-NAS NASBench-201
Figure 2: (a) The landscape term draws the slice of loss landscape to capture its macroscopic non-convex nature of the candidate architecture. The speed term analyzes the training changes over the training time, providing microscopic insights into the candidate's s convergence speed. (b) Comparison of MOTE-NAS and an Evaluation- version MOTE-NAS-EF against other recent efficient NAS methods on NASBench-201 (CIFAR-100).
To enable lightweight training- bassed estimates, we introduce two reduction strategies for speeding up MOTE generation. Unlike other benchmarks, readuced meta-architecture is used, and the proposed reduced dataset is built by selecting representative labels of CIFAR-100.
Our MOTE-NAS achieves a new state-of-the-art for NAS in the accuracy-cost plot (refer to Fig. 2(b)). stripped-down, evaluation-free version of MOTE-NAS is highly efficient, with performance of the resulting model still outperforming some NTK-based methods, such as the KNAS.
2 Related Work
DARTS, Instead of exploring a discrete set of architectures, the Differentiable ARchiTecture Search (DARTS) [26] transforms the combinatorial challenge of finding optimal operations into a continuous optimization problem within a differentiable search space. A notable challenge with DARTS is the potential dominance of easily optimized operators, such as skip-connections and pooling operations in the early stages. This issue impedes selecting more potent operations like convolutions with large kernels. [7, 22], a robust prior introduced to restrict the number of skip connections within cell to predefined value. The progressive search strategy employed in P-DARTS [7] gradually increases network depth and refines candidate operations based on mixed operation weight. DARTS methods are efficient when running with limited computational resources. However, the architecture found comes with stability and generalizability issues. Furthermore, DARTS algorithms often prefer shallow and wide structures [37].
On the other hand, NAS methods generally comprise two stages: the search stage and the evaluation stage. The former focuses on collecting promising candidate architectures, while the latter involves assessing the performance of these candidate architectures.
Search Stage. Numerous studies have concentrated on the search stage. Some NAS methods leverage reinforcement learning [52, 2, 39], while others are rooted in evolutionary algorithms [27,31, 33, 46, 91. The cell-based tabular search space [49, 10, 38] is effective in reducing exhaustic search into more manageable scale, using meta-architecture with predefined operations, hyperparameters, filters, and strides. The candidate architectures under consideration range from tens to hundreds of thousands of candidate architectures. Recently, predictor- gained popularity. These methods construct predictors trained with architecture-accume forecast the performance
approaches [24, 28, 43, 11, 42, 45, 15] have
pairs to candidate architecture. These predictors encompass a range of models from graph convolutional networks [17] to MLPs and other regression models. However, obtaining high-quality set of schitecture-accurace pairs for NAS is non- trivial.
Evaluation Stage. Compared with the cost of the search stage, the burden of NAS mainly resides in the time-consuming evaluation process. Various studies have proposed proxy estimates reduce the need for real performance evaluation. A prominent approach is zero/few--osst estimate [1, 34,
3

---

23, 29, 6, 47, 51, 30], which substitutes performance indicators such as accuracy with alternative estimates, The zero-cost proxy [1] introduces zero-cost performance estimates [20, 41, 40] and TSE [34] introduces a training speed estimate. More recently, Neural Tangent Kernel (NTK)-based estimates [6, 47, 51, 30] have gained popularity based on the assumption that DNNs can predict their convergence performance at initialization. However, it is experimentally found in [30] that NTKs cannot capture the non-linear characteristics of DNN training dynamic well. Recent NAS methods integrate multiple approaches to achieve remarkable performance. For instance, OMNI [44]
and ProxyBO [36] propose few-cost NAS methods by combining zero-cost estimates with more resource-intensive techniques like Bayesian optimization and performance predictors.
3 MOTE-NAS
3.1
Multi-Objective Training-based Estimate
NTK theory tries to describe gradient change by a macro- peacro-perppectvve, but its fundamental assumption about infinite-width DNN cannot fit the real DNNs that have finite width. For example, KNAS [47], GKH asserts the existence of gradient feature that can serve coarse-grained proxy to support downstream training when evaluating randomly initialized architectures. However, this does not propose concrete solution to identify such non-linear gradient feature during training. LGA[30] finds that the sensitivity for weight initialization that leads to NTK cannot perform stably, demon- strating that the value of NTK does not change [19]. In addition, Fig. 1 shows that NTK do not accurately predict the actual performance of candidates in practice. Despite NTK- estimates, there are estimates that make predictions by training loss as a proxy estimate to represent the convergence speed. The convergence speed as an important factor for model performance has been extensively discussed in the literature [13, 16, 34]. Specifically, these studies inspired the proposal of MOTE. MOTE introduces a new landscape term to capture the non-convex nature of models by macro- pereppetereeecvveee through the linear combination between two weights to observe the loss landscape. Simultaneously. MOTE introduces another new speed lerm to measure the convergence speed of the model by nicro-perspective through calculating the unit time training loss. Incorporating the multi-objectives enables MOTE to comprehensively describe candidates' non-convex nature and convergence speed from macro-to-micro perspective.
Among them, TSE sums up the
Landscape Term. In order to capture non-convexity of loss landscape by macro- persseetee,,,,,, we introduce landscape term that linearly combines the two weights before and after few- few-training to interpolate the weights for intermediate state, and then calculates the loss values (ross--ettoppy) of these weights, which means to cutting section from loss landscape to observe its nature [12], so we sum these loss values to determine whether the loss landscape is smooth, detailed follows.
Let 0 denote the trained weights of the candidate. To macro-model the actual performance of candidate architectures, we linearly combine the initial weights with the trained weights to obtain the combined weights, denoted (gg), for describing the non-convex landscape of the candidates. Then, we obtain the combined weights 0(g) in the form:
((g) =0g 0
9
G )0
9
where fnst denotes the initial weights, and is the number represent how dense the linear combi- nation and set to be 10 based on most of the experiments from [12]. Let y-((0) denote the model prediction labels based on the weights ((g), and Ygt be the ground-truth labels of the training data.
Then, we use cross-entropy to measure the difference between y=f() and YGt- After that, we sum the loss value of these middle weights as landscape term as follows.
G
Σ
Jolo = - CC(Y .Yt).
9=
The poposed landscape term can capture the non-convexity of models where a lower value indicates a flatter loss landscape, implying an efficient convergence of the candidate and avoiding the problem of sharp minimum [50].
Speed Term. On the other hand, the idea to model the actual performance of candidates by micro- perspective, such as TSE, still perform strong and important, shown in Fig. 1. Therefore, we are

---

Meto Meto-Architecture (MS-201)
Cell
abew
Pieniing weig
IRO
IR0
IPQ
IRQ
Xo1g slgg
(PP
1P0
180
Indmo
Datasets
Accuracy
C10, C100, Ingilatio
Reduced Architecture
Cell
lin
110
GAP
R0
Reduced
100g
MOTE
Dataset
Figure 3: The generation pipelines of accuracy (upper part) and MOTE (bottom part). The proposed reduced architecture and dataset, MOTE, are colored red in their respective sections.
inspired by TSE [34] to introduce speed term. first calculates training losses (cross-entropy) over an epoch, and divides by the time cost of an epoch, converting into unit time training loss to measure convergence speed, which means that observes the convergence speed of the candidates under standardized time expenditure, which helps standardize the measurement of architectures of different sizes. It can also be seen from the Fig. that the speed term performs better than the TSE [34]. The detailed speed term is defined:
E
CE(Y be
pa: ytt)
where Ypu denotes the model prediction labels in epoch e, and Yg means the ground truth labels of the training data. Then, we use cross-entropy to calculate the loss between ypd and Ygt. After that, divide the loss value by the time cost within epoch and then sum up all as speed term. Note that a lower value indicates faster convergence, which could imply better performance.
Now, MOTE integrates the landsacpe term and speed term to model actual performance of candidates from macro to micro perspective, and is defined as follows:
E
MOTE = f Jo())
9
where the first term is the proposed landscape term, the latter term is the proposed speed term, where To denotes the function used to determine whether the loss landscape is smooth by linear combining initial weights fast and trained weights 8, E is the number of maximum training epochs to search candidate architectures, le is the training toss(uuually measured by cross- (ross-entropy) for the eth epoch, le denotes the time cost the epoch e, and the function f denotes the non-linear transformation to restrict the range of values.
Due the different ranges of landscape term and speed term, to transform and normalize them; more comparisons of other transformation methods are detailed in Appendix A.1. MOTE can consider both objectives by adding transformed method assess their combined impact. Since lower values for both landscape term and speed term suggest a potentially better performance of the models, smaller MOTE value indicates better performance in practical application. Fig. 2(a) illustrates the concepts of landscape term and speed term.
the box-cox transformation [3]
3.2 Reduction Strategies for MOTE Generation
MOTE requires little training to obtain landscape term and speed term, which makes crucial to balance minimizing training time and ensuring adequate training change. Consequently, we introduce more compact meta-architecture called the reduced architecture. We also propose the reduced dataset method, which involves smaller dataset built by CIFAR-100. MOTE combines these reduction strategies and the training-based objectives introduced earlier to produce promising estimates with few costs. Fig. 3 depicts this process.
Reduced Architecture (RA). MOTE is not the actual performance of DNN after combining the cell with aeta-architectures. Instead, MOTE is generated from the change in weight and loss acquired during training. The generation of MOTE relies on the proposed reduced architecture. The reduced architecture is designed to eliminate redundant layers from most meta- aettaarrcitecturrs[45 10], resulting
5

---

Img Embod Code 1 Img Embd Code? ?
Mon
VGG-16 pretrained on ImgNet
Embdl Code Img Embi Code 4
Aug Img
Embd Codes (Lobel: Manj
CIFAR 100
Jmg Embd Code
Reduced Dataset
Figure 4: After encoding the images of CIFAR-I00 through VGG, the encodings for each label are obtained by averaging image embedding codes. Then we used K-Means and Farthest Point Sampling (FPS) to select a representative set of labels, forming the reduced dataset.
Rooats Bo commaati
eccomnyrl in tederape AA
NJ.
5 S. 9,
corse
20
and costb cat
300
138
swarape codtist per oal
209
(a) NASBench-101
(b) NASBench-201
Figure 5: The proposed terms via aggressive reduction strategies on NASBench- 101 and NASBench- 201. RA means reduced architecture, RD means reduced dataset
in a simplified network structure to save the time cost of each epoch. It only comprises a convolutional layer as the stem layer and two cell layers and employs aggressive downsampling through a pooling layer with a kernel size of 4 x 4. This architectural simplification significantly accelerates the training process and substantially reduces the cost of obtaining MOTE, The structure of the reduced architecture is shown at the bottom of Fig. 3.
Reduced Dataset (RD). To minimize the computational cost of MOTE generation, we propose sub-dataset derived from CIFAR- 100 [18]. referred to as the reduced dataset. As shown in Fig. 4, the process of constructing the reduced dataset involves several steps: 1) We use VGG-16 model pre-trained on ImageNet- K [8] to extract logits from images; 2) Flattened the logits and averaged them according specific label, resulting in label embedding codes; 3) K-Means to cluster the label embedding codes into r groups; 4) Farthest Point Sampling (FPS) to select label embedding codes from group to represent the labels of the reduced dataset; 5) The images associated with the chosen r labels are reserved for building the reduced dataset. The reduced dataset is a proxy dataset and sub-dataset of CIFAR-100, containing a representative reduced dataset is equivalent to CIFAR-100. As r decreases, the reduced dataset becomes smaller and easier to fit for candidate models. However, regardless of the value of r. the reduced dataset maintains the original image distribution for each label. The K K-means and FPS techniques ensure that the reduced dataset represents a significant part of CIFAR- even when r < 100. The most important thing is MOTE generation that rely on reduced dataset can save remarkable time cost.
of labels. When r is set to 100, the
Two Terms of MOTE with Reduction Stratigies. To further observe how reduction strategies work, we randomly selected 1K candidates of NASBench-201 as toy experimental subjects. As shown in Fig. 5, the early stop version of test accuracy (after 12 epochs) has high correlation with test accuracy (after 200 epochs), but training then getting it required about 220 gpu seconds, which is still a remarkable cost, When RA is applied, the time cost decreases 60%, but the correlation also drops to about 0.5 from 0.65. After further RD is applied, as the extraction ratio gradually decreases (the further to the left the smaller is), the time cost is also greatly saved, but the correlation suffers intolerable losses. The correlation of the leftmost (r = 10) is even less than 0.1. In contrast, the proposed landscape term and speed term always maintain a high correlation when applying RA and RD. As the extraction ratio r gradually decreases, the time cost is reduced to about 10 gpu seconds from 220 seconds, and the correlation still about 0.65. This is because the two proposed terms based on weight and loss changes do not rely on the excluded middle state of true or false. More comparisons between various reduction strategies refer to Appendix A.2.
3.3 Integrating MOTE with Evolutionary Search
Although MOTE consistently maintains impressive performance under the influence of the reduction strategy, MOTE remains a proxy estimate. A comprehensive NAS method still requires the participa- tion of accuracy to evaluate the actual performance of the candidates discovered in the evaluation

---

(Pp- by MOTE
Add to Pool P
top-k(P)
top-1(Pk) by Validation
Mutation
by MOTE
Search Stage (total iters m0)(k,10)
Evaluation Stage
Figure 6: The left side depicts MOTE-NAS's search stage, utilizing MOTE for architecture selection + k iterations. On the right side is the evaluation stage, where MOTE selects the top-k architectures for evaluation. MOTE-NAS-ER simplifies this by relying solely on MOTE to choose the top-1 architecture without the evaluation stage.
through an evolutionary loop, terminating
NB201-C100
N5201-C100
N6201-1mg16
NS101
SynFloe/-15
(5E(15.0g)
TE-NASI-18)
KNAS(-15)
Zen-Score(-ts)
LGN(-36)
ZICO
Tm(7-66
Spd Tm//
MOTEIT
Figure 7: The Kendall's Tau Correlation comparison of the proposed speed term, landscape term and MOTE with other estimates on NASBench- 101 and NASBench-201. Note that the "(s)" the GPU seconds per cell cost.
stage, similar to previous NAS methods [6, 47, 51, 30]. However, mentioned the outset, the time cost of accuracy exceedingly high, often demanding thousands of GPU seconds. Therefore, our proposed MOTE-NAS first employs MOTE to assist the evolutionary process in obtaining small subset promising candidates. Then, through the early stopping version of accuracy assessment, the best is identified. The entire procedure is illustrated in Fig. 6.
In the search stage, MOTE-NAS maintains a pool of promising candidates P, with batch size B initially set to 10. With the continuous evolution loop, every ten cycles, B is incremented by 10. In each evolutionary process, MOTE is generated to sort P and take the top 10% the candidates to mutation stage. For the mutation stage, we are inspired by predictor-based methods [43, 11, 45] to encode candidates into adjacency and operation matrices. Subsequently, we calculate the Euclidean distance between each pair of candidates to select up to 10 x k mutated candidates, which are then added to P. This evolutionary loop continues maz (k 10) times, then stops and enters the evaluation stage. For the evaluation stage, MOTE is used to select the top k architectures of P (k = 5, 10, 20), then select their best architecture based on the early topping version of the test accuracy.
4 Experimental Results
Experimental Setup. We used NASBench-100 and NASBench-200 201, both cell-based search spaces. NASBench-101 has 423,621 candidates trained on CIFAR-10 for 108 epochs. NASBench-2011 includes 15,625 candidates trained on CIFAR-10, CIFAR-100) and ImageNet-16-1 1-02 for 200 epochs Tesla V100 GPUs, with MOTE MOTE-NAS costs calculated specifically on V100. Our experiment had three parts: comparing MOTE with other estimates on NASBench- 101 and NASBench-201. evaluating MOTE-NAS against other NAS methods on NASBench-201, and using MOTE-NAS to search for mobilenet-like architecture on ImageNet-IK. IK. Further, we visualize the rankings of MOTE and KNAS to perceive their differences in Fig. MOTE is generated from the proposed reduced architecture and dataset. We used reduced dataset with sampling rate based on the results in Fig. 5. The hyperparameters batch size 256, epochs 50, learning rate 0.001 with Adam optimizer, and cross-entropy loss function. MOTE generation per cell took about seven GPU seconds under these settings.
each. Computation
hyperparameter
4.1 Comparison of MOTE and Other Estimates
To explore the performance gaps between MOTE and other relevant estimates, we compared NASBench-101 and NASBench-201. We ranked candidates using speed term, landscape term and MOTE or other estimates and compared the resulting rankings to the actual ranking, calculating Kendall's s Tau correlation to gauge the performance of these estimates. The experimental results are presented in Fig. 7.
7

---

1000
1033
low-acc highh-es
low-acc high-est
200un002200 eporhsl
800-
802
#03
high-accl low-est
high-accl low-ost
high-acc 200 high-est
KHAA rankingg
600
1035
high-acc 200 high-est
"R
ROTE rankiing
438
600
3000
Figure 8: Comparison of the distribution of MOTE (red) and KNAS (green) on NASBench-201 (CIFAR-100).
NTK-based Estimates. MOTE leverages the landscape term to capture the non-convex nature from the candidate architectures, compensating for the shortcomings in NTK-based estimates. As shown in Fig. 7, MOTE achieves correlations of 0.68, 0.66, 0.62, and 0.47 in NASBench- 101 and NASBench- 201. Compared to TE-NAS and KNAS, MOTE shows performance improvements ranging from 13% to 62% on NASBench-200 2011 In addition, we observed poor performance of TE-NAS and KNAS in NASBench-100 101, with KNAS demonstrating correlation of merely 0.09. This illustrates that NTKs struggle to adapt to larger search spaces and more diverse candidate architectures in environments like NASBench-101 In contrast, MOTE maintains high correlation of 0.47, demonstrating significant increase of 26 % and 422% compared to TE-NAS and KNAS, respectively. Compared to the state-of-the-art ZICO, MOTE still performs better than it does on benchmarks. Remarkably, these MOTE performance gains were achieved with an average cost of only seven seconds per candidate.
Other Estimates. When comparing MOTE with other estimates such as SynFlow [40, 1] and TSE [34), MOTE maintains a significant advantage. Compared to SynFlow and TSE, MOTE shows performance gains of 1 % to 45% on NASBench-201 and 96% and 104% on NASBench- 101, respectively. It should be noted that both TSE and MOTE are training- based estimates, and MOTE accelerates 2x faster than TSE, outperforming it ieniffiantly.Addiiiinn
we can see that the proposed speed term and landscape term also perform well in Fig. 7, both of which are essential components of MOTE.
4.2 Visualization of MOTE and NTK based Estimate
We depicted the distribution of MOTE in comparison KNAS to facilitate analysis regarding MOTE and NTK-based estimates. As shown in Fig. 8, the experimental results involve random selection of 1K candidate architectures from NASBench-201 (CIFAR-100), The x-axis represents the estimate ranking based on MOTE or KNAS, while the y-axis represents the actual ranking based on the test accuracy after 200 epochs. Each node in the figure represents candidate architecture, with its (x, y) coordinates indicating its position in the estimate and the actual rankings, respectively. Both the estimate ranking and the actual ranking are sorted from high to low scores.
In the left subfigure of Fig. 8, we present the distribution of KNAS, while the right subfigure displays the distribution of MOTE. In general, MOTE is more concentrated than KNAS, indicating that MOTE is closer to the actual ranking of the candidates than KNAS. This observation aligns with the superior performance of MOTE Kendall's Tau correlation comparisons, as shown in Fig. Further, focusing on the nigh-performance region (the blue circle in the lower left corner of the two sub-figures). MOTE exhibits more concentrated trend compared to the chaotic distribution of KNAS. This suggests that MOTE outperforms KNAS in predicting promising architectures. Finally, examining the purple boxes in the lower right and upper left corners of the two subfigures reveals candidates for which the estimate indicates good. However, the actual performance is poor, or vice versa. both cases, MOTE's misjudgments are significantly fewer than those KNAS, visually confirming that MOTE is highly competitive compared to the mainstream NTK-based estimate.
8

---

Table 1: Comparison of the proposed MOTE-NAS and others on NASBench-201, Note that 'Cost (s)' indicates the cost in seconds calculated on Tesla V100. Entries in bold with underlines indicate the best performance, and those in bold alone represent the second-best performance.
CIFAR-I0
CIFAR-I00
ImgNet-16
Type
Model
Ac)(S))
Cost(s)
Acc(S))
Cost(s)
Acc(%)
Cost(s)
Neural Predictor [43]
94.07
840K
72.18
840K
46.39
2.4M
Arch-Graph [15]
73.38
840K
Predictor
WeakNAS [45]
94.23
840K
73.42
840K
46.79
2.4M
Proxy-BO [36]
73.48
1.2M
47.18
3.2M
NASWOT [29]
92.96
2.2K
70.03
4.6K
44.43
I0K
TE-NAS [6]
93.90
2.2K
71.24
4.6K
42.38
10K
KNAS (k=20) [47]
93.38
4.4K
70.78
9.2K
44.63
20K
KNAS (k=40) [47]
93.43
8.8K
71.05
18.4K
45.05
40K
Eigen-NAS (k=20) [5]]
93.46
4.4K
71.42
9.2K
45.53
20K
Few-Cost
RS + LGA [30]
94.05
5,4K
71.56
7.0K
46.30
15K
REA + LGA [30]
94.30
3.6K
72.42
5,4K
45.30
3.6K
MOTE-NAS (k=5)
93.97
2.2K
71.89
2.4K
46.10
5.8K
MOTE-NAS (k=10)
94.15
4.2K
72.54
4.3K
46.38
11,3K
MOTE-NAS (k=20)
94.32
8.5K
72.81
8.5K
46.34
22.7K
MOTE-NAS-EF
93.54
0.5K
71.59
0.6K
44.73
0.6K
4.3 Comparisons of MOTE-NAS and Other NAS
To compare the performance differences between MOTE-NAS and other NAS methods, we carried out experiments on NASBench-200 Tab. presents the experimental results, where "Ace(%)" represents the accuracy of the final architecture discovered by the NAS methods on the test set of the respective dataset. At the same time, "Cost(s)" indicates the total seconds used by NAS methods to discover this final architecture.
MOTE-NAS with Top-k Evaluation. MOTE-NAS combines MOTE with an evolutionary algorithm to filter and mutate potential high-scoring candidates by MOTE. Ultimately, the top k: high-scoring candidates selected using the early stopping version of the test accuracy (after 12 epochs), similar to the approaches [47, 51]. The time consumption of MOTE-NAS lies in training candidates to obtain MOTE during the search stage and the early stopping accuracy obtained during the evaluation stage. However, the cost of each MOTE is only about seven gpu seconds, so primary consumption is still evaluate the top-k candidates. We have set the k range from 5, 10, 20. When A: 5, the final architecture found by MOTE-NAS achieves significantly higher accuracy on three datasets of NASBench-201. compared TE-NAS[6]. KNAS[47]] and Eigen-NAS[51] with speedups ranging from 1.9x to 6.9x. It only slightly lags behind REA + LGA[30] in test accuracy. However, when k 10 or & 20, with a time consumption of 8.5K gpu seconds, the final architecture discovered by MOTE-NAS achieves remarkable accuracy of 94.32% on CIFAR- 10 and 72.81 % on CIFAR-100 100. Moreover, the MOTE-NAS discovered candidate architecture achieves 46.38% on ImageNet- 16 with 1 11.3K seconds. Compared to NTK-based NAS (TE-NAS, KNAS, Eigen-NAS, LGA), the proposed MOTE-NAS consistently achieves the best accuracy with the lowest cost.
Assessing the Evaluation-Free Version MOTE-NAS. The proposed MOTE-NAS has shown impressive performance in balancing time consumption and efficiency. The substantial time cost led us consider omitting the evaluation stage to pursue faster MOTE-NAS framework. Especially considering that MOTE, compared to other estimates, achieves higher Kendall's Tau correlation, indicating a significant improvement in MOTE's predictive performance. Hence, omitting additional validation information became viable option. To accomplish this, we removed the entire evaluation stage from MOTE-NAS. At the end of the search stage, we utilized MOTE to select top-1 1 as the final architecture. This variant is referred to as MOTE-NAS-EF in Tab. 1.
Although MOTE-NAS-EP experiences an accuracy loss, the search cost savings are notable. MOTE- NAS-EF achieved 93.54% accuracy on CIFAR- 10, 71 71.59% on CIFAR- 100, and 44.73% on ImageNet- 16-120 with the search cost of only about 0.6K gpu seconds. In contrast, KNAS requires 4.4K, 18.4K, and 20K seconds to achieve accuracies of 93.38%, % 71.05%, and 44.63%, respectively. In particular, MOTE-NAS-EF matches KNAS in accuracy but accelerates the process by 4.8x to 22.2x, underscoring the superiority of MOTE-NAS-ER in speed.

---

Table 2: This table shows the top-1 accuracy of architectures found on ImageNet using MOTE-NAS and other NAS methods.
Model
MFLOPs
fop-1(%)
Cost(d) N/A
MobileNetV2 [35]
300
71.5*
MobileNetV3 [14]
220
74.1*
OFA [4]
406
77.7
50
BN-NAS [5]
470
75.7
0.8
NASNet-B [53]
488
72.8
1800
CARS-D [48]
496
73.3
0.4
ZICO [21]
448
75.8*
0.4
MOTE-NAS
387
76.2
0.1
MOTE-NAS
473
77.1
0.1
4.4 MOTE-NAS on ImageNet-IK
Search Space. We search for promising architecture based the mobilenetV3 search space using MOTE, then train and evaluate it on imagenet- IK. The mobilenetV3 search space is open search space that has five inverted residual blocks with the SE module. Every block has several hyperparameters, such as the expansion ratio for input channel expansion, kernel size, and SE module attached or not. Based on it, we restrict the selection range for each hyperparameter. We restrict the expansion ratio range from 2, 4, 6, kernel size range from 3, 5, 7, and the SE module used or not.
Rescaled Reduced Architecture for Macro-Search. Our study introduces Rescaled Reduced Architecture for Macro-Searde where we modify the reduced architecture to accommodate the simultaneous assessment of five blocks and their collective performance. By expanding the cell layers from two to five and independently sampling the structure of each layer, our rescaled approach enables macro-search capabilities beyond single-cell exploration. Further technical specifics of this rescaled reduced architecture are outlined in Appendix A.5. Subsequently, employing MOTE-NAS with this modified architecture, we conducted search within the mobilenetv3 space under approximately 400M FLOPs. Following 200 epochs of training using 10 GTX 2080Ti GPUs on the imagenet- 1 dataset, the results (see Table 2) demonstrate the efficacy of our approach. While the accuracy of MOTE-NAS (76.2% and 77. 1) trails slightly behind OFA's 77.7%, MOTE-NAS achieves this with a significantly reduced computational cost of GPU days compared to OFA's 50 days, representing 500x speed improvement. Furthermore, retraining of ZICO's best architecture yielded a 75.8% accuracy on imagenet, surpassed by MOTE-NAS with its superior accuracies and a 4x faster search cost than the 0.4 day of ZICO.
Conclusion
In this paper, we design a novel training-based estimate for efficient Neural Architecture Search (NAS) from multi-objective optimization perspective. The key idea is use landscape terms to capture the non-convex nature of candidate architectures from a macro perspective, and use speed terms monitor convergence speed from micro perspective into the estimated design. The proposed MOTE efficiently generates the landscape and speed terms with two reduction strategies, which wisely trade-off the consideration of architecture and dataset. These designs can effectively capture the non- linear characteristics of deep neural network training, address the drawbacks of NTK methods, and achieve new state-of-the-e oote----rttt-ttt---- state. We extend our approach by iterative ranking- basking-based evolutionary search, then deduce an evaluation-free version (MOTE-NAS-EI experimental results demonstrate the superiority of our new NAS methods over other frontier NAS methods, including KNAS, LGA, and ZICO, on NASBench-101 101, NASBench-201. and ImageNet- IK. Future works include expanding MOTE other NAS frameworks, such as predictor-based methods, to pursue precise search results while exploring MOTE's generalization ability. Another line of extension to work with more challenging NASBench-300 benchmark dataset [38] that offers a much larger and more complex architecture space than both NASBench-10) and NASBench-201.
that runs even faster. Extensive
References
[1] Mohamed S Abdelfattah, Abhinav Mehrotra, Fukasz Dudziak, and Nicholas D Lane. Zero-cost proxies for lightweight NAS. arXiv preprint arXiv:2101.08734 2021.
10

---

[2] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using reinforcement learning, arXiv preprint arXiv:161..02167,0 2016.
[3] George EP Box and David R Cox. An analysis of transformations. Journal of the Royal Statistical Society Series B: Statistical Methodology, (262):211-245 1964.
[4] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint 19XV::900.00991. 2019.
[5] Boyu Chen, Peixia Li, Baopu Li, Chen Lin, Chuming Li, Ming Sun, Junjie Yan, and Wanli Ouyang. BN-NAS: Neural architecture search with batch normalization. In Proceedings of the IEEE/CVE international conference on computer vision, pages 307-316, 2021.
[6] Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on ImageNet in four GPU hours: A theoretically inspired perspective. arXiv preprini arXi::2102.1155: 2021.
[7] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In ICCV, pages
2019.
[8] Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of ImageNet as an alternative to the CIFAR datasets. arXiv preprint urXiv:1707.08811 2017.
[9] Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, et al. FBNetV3: Joint architecture-recipe search using predictor pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (6276-16285, 2021.
[10] Xuanyi Dong and Yi Yang. NAS-Bench-201-
Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00322 2020.
[11] Lukasz Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas Lane. BRP-NAS: Prediction-bass nas using GCNs, Advances in Neural Information Processing Systems, 03:10480-10490, 2020.
[12] Ian Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv: arXiv:1412.65444 2014.
[13] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pages 1225-1234. PMLR, 2016
[14] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chict Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for Netile In Proceedings of the IEFE/CVF international conference on computer vision, pages 1314-1324, 2019.
[15] Minbin Huang, Zhijian Huang. Changlin Li, Xin Chen, Hang Xu, Zhenguo Li, and Xiaodan Liang. Arch-Graph: Acyelic architecture relation predictor for transfraneeee neural archi- tecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11881-11899 2022.
[16] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv: 1912:2/112.02788 2019.
[17] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.088
2016.
[18] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
[19] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
11

---

[20] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. SNIP: Single-shot network pruning based on connection sensitivity. arXiv preprint 1rX1v:1810.02340, 2018.
[21] Guihong Li, Yuedong Yang, Kartikeya Bhardwaj. and Radu Marculescu. ZiCo: Zero-shot NAS via inverse coefficient of variation on gradients. arXiv preprint arXiv:2301. 2023.
[22] Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. DARTS+: Improved differentiable architecture search with early stopping. arXiv:1909. 06035, 2019.
[23] Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-NAS: zero-shot NAS for segh-performacc image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 347-356, 2021.
[24] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European conference computer vision (ECCV), pages 19 19-34, 2018.
[25] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. arXiv preprint 1806:0880000005 2018.
[26] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019.
[27] Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf, NSGA-Net: neural architecture search using multi-objective genetic algorithm. In Proceedings of the genetic and evolutionary computation conference, pages 419 419-427,200 2019.
[28] Rengian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimiza- tion. Advances in neural information processing systems, 31, 2018.
[29] Joe Mellor, Jack Turner, Amos Storkey, and Elliot Crowley. Neural architecture search without training. In International Conference on Machine Learning, pages 7588-7598. PMLR, 2021.
[30] Jisoo Mok, Byunggook Na, Ji-Hoon Kim, Dongyoon Han, and Sungroh Yoon. Demystifying the neural tangent kernel from a practical perspective: Can it be trusted for neural architecture search without training? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11861-11870, 2022.
[31] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780-4789, 2019.
[32] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780-4789, 2019.
[33] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoe Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In International Conference Machine Learning, pages 2902-2911, 1. PMLR, 2017.
[34] Robin Ru, Clare Lyle, Lisa Schut, Miroslav Fil, Mark van der Wilk, and Yarin Gal. Speedy per- formance estimation for neural architecture search. Advances in Neural Information Processing Systems, 34:4079-4092, 2021.
[35] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chich Chen. MobileNetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018
[36] Yu Shen, Yang Li, Jian Zheng, Wentao Zhang, Peng Yao, Jixiang Li, Sen Yang. Ji Liu, and Bin Cui. ProxyBO: Accelerating neural architecture search via Bayesian optimization with zero-cost proxies. arXiv preprint 10014:2110.10433, 2021.
12

---

[37] Yao Shu, Wei Wang, and Shaofeng Cai. Understanding architectures learnt by cell-based neural architecture search. In ICLR, 2019.
[38] Julien Niklas Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. NAS-Bench-301 and the case for surrogate benchmarks for neural architecture search, 2020.
[39] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. MnasNet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2820-2828, 2019.
[40] Hidenori Tanaka, Daniel Kunin, Daniel I. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33:6377-6389) 2020.
[41] Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376. 2020.
[42] Chen Wei, Chuang Niu, Yiping Tang, Yue Wang, Haihong Hu, and Jimin Liang. NPENAS: Neural predictor guided evolution for neural architecture search. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[43] Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In Europeun Conference on Computer Vision, pages 660-676. Springer, 2020.
[44] Colin White, Arber Zela, Robin Ru, Yang Liu, and Frank Hutter. How powerful are performance predictors in neural architecture search? Advances in Neural Information Processing Systems, 01224444288466 2021.
[45] Junru Wu, Xiyang Dai, Dongdong Chen, Yinpeng Chen, Mengchen Liu, Ye Yu, Zhangyang Wang, Zicheng Liu, Mei Chen, and Lu Yuan. Stronger NAS with weaker predictors. Advances in Neural Information Processing Systems,
2188904-28918 2021.
[46] Lingxi Xie and Alan Yuille. Genetic CNN. In Proceedings of the IEEE international conference on computer vision, pages 1379-1388, 2017.
[47] Jingjing Xu, Liang Zhao, Junyang Lin, Rundong Gao, Xu Sun, and Hongxia Yang. KNAS: green neural architecture search. In International Conference on Machine Learning, pages 11613-11625. PMLR, 2021.
[48] Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. CARS: Continuous evolution for efficient neural architecture search. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1838, 2020.
[49] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS-Bench- 101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pages 7105-7114. PMLR, 2019.
[50] Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakehi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. In Proceedings of the International Conference on Learning Representations, 2020.
[51] Zhenyu Zhu, Fanghui Liu, Grigorios Chrysos, and Volkan Cevher. Generalization properties of NAS under activation and skip connection search. Advances in Neural Information Processing Systems, 35:23551-23565. 2022.
[52] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint /61 1.01578, 2016.
[53] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 2018.
13

---

A Appendix
Table A1: Kendall's correlation between the MOTE scoring output and the test accuracy(after 200 epochs). Experiments are performed to compare four versions of MOTE on three sub-datasets (CIFAR-10, CIFAR-100, ImgNet-16) 16) of the NASBench-201.
CIFAR-10
CIFAR-100
ImgNet-16
MOTE w/o transformation
0.66
0.64
0.59
MOTE w/ logarithm
0.66
0.64
0.60
MOTE w/ reciprocal
0.67
0.65
0.60
MOTE w/ box-cox
0.68
0.66
0.62
A.1 Non-linear Transformation for MOTE
The proposed Multi-Objective Training-based Estimate (MOTE) achieves efficient Network Archi- tecture Search (NAS) based These terms and their variants play a pivotal role in modeling the training dynamics and thus serve as performance estimate for candidate architectures during the search. To address the considerable variability of these terms, in the main paper, we propose to incorporate the box-cox method in the MOTE design to constrain both terms. We next provide additional experimental results support such a design by evaluating the influence of other designs and variants of these terms. Specifically, we compute kandall's correlation on NASBench- 201 to compare four versions of MOTE that contain logarithm, reciprocal, box-cox, and the original.
essential components, namely the landscape and speed terms.
Table Al shows the comparative results of four versions of MOTE. MOTE w/ box-cox performs the best. The box-cox transformation leads to a notable improvement compared to the original in correlation: 2% improvement for CIFAR-10, 2% improvement for CIFAR- 100, and 3% improvement for ImageNet- 16 120. These improvements are attributed to the ability to contract outliers, bringing them closer the normal data distribution and reducing the impact on MOTE.
A.2 Analysis of Two Reduction Strategies
Although the proposed MOTE-NAS approach can greatly reduce the required NAS cost producing suitable network architecture, the time cost of model training and evaluation is still the bottleneck and trade-off. We further analyze the two proposed reduction strategies and present the results of the ablation study regarding the computation time trade-off. Fig. 3 provides the MOTE generation pipeline that contains the reduced architecture and the reduced dataset.
Experiments setting. We conducted a random sampling 1,000 candidate architectures from NASBench-101 and NASBench-200 respectively. Notably, these candidate architectures consist of their predefined meta-architecture and cells within their search spaces. Each cell shares identical hyperparameterss and operations. Therefore, to obtain complete models, the candidate architectures (equivalent to candidate cells) must be assembled into either the meta-architecture or our reduced architecture. The proposed reduced architecture speeds up model search by retaining only single convolutional layer as the stem layer and two cell layers, eliminating the majority of layers from the original aeta-architecture of NASBench-100 101 and NASBench-201. The reduced dataset is representative proxy subset of CIFAR-100, with the aim of accelerating the training process by reducing the number of training images, thus reducing the time needed for each training epoch.
Reduced Architecture vs. Meta-Architecture. Table A2 compares the model parameters and FLOPs of predefined meta-architecture and the peoposed reduced architecture, when evaluated on the NASBench- 101 and NASBench-200 benchmarks. For the NASBench-101 case, the reduced architecture contains much fewer parameters of 34.4K, in contrast to the 2.3M parameters of the original meta- arttaaarchhteturee This is a significant reduction of 64x. Similarly, the FLOPs for the reduced architecture amount to 18.3M, while the original meta-architecture records 1.0G FLOPs, indicating an acceleration of x35x. In the case of NASBench-201 candidates, the reduced architecture exhibits reductionss/ccceeeaaiii
of 15x and 4x compared to the original meta-architecture. These results indicate a superior advantage the proposed reduction strategy in both memory requirements and computational speed on the two benchmarks.
14

---

Table A2:
Comparisons of memory and computation requirements between our reduced archi- tecture and original predefined meta-architecture of NASBench- 101 and NASbench-201.
Params
FLOPs
Meta-Architecture (NAS101)
2.3M
1.0G
Reduced Architecture (NAS101)
34,4K
18.3M
Meta-Architecture (NAS201)
388.7K
52.7M
Reduced Architecture (NAS201)
25.7K
13.6M
Table A3: Ablation study for three reduced strategies on NASBench-101. 7 represents the number of selected labels the reduced dataset. Cost(s) indicates the average seconds each candidate to obtain the estimate. 7(%) is the Kendall's correlation.
T-FLOPs
Cost(s)
r(%)
Acc w/ MA
N/A
190.7B
163.1
34,5
Acc + RA
N/A
8.7B
127.1
8.1
MOTE RA
N/A
8.7B
127.1
50.1
MOTE + RA+ RD
80
7.0B
103.0
49.5
MOTE RA+ RD
60
5.2B
78.2
48.8
MOTE RA+ RD
3.5B
56.8
47.8
MOTE RD
1.7B
26.3
46.8
MOTE + RA+ RD
10
872.6G
13.2
46.6
Ablation Study of Reduction Strategies. We outline the experimental setup for two reduction strategies: (1) reduced architecture inspired by predefined meta-architecture of NASBench- and NASBench-201, (2) reduced dataset for efficiently training. Evaluations performed on the NASBench- 101 benchmark. We rank 1,000 randomly selected candidate architectures under different reduction conditions. The correlation measures MOTE and test accuracy obtained from model training after 108 epochs. The early stopping test accuracy after 4 epochs serves as the baseline. We incrementally introduce the reduced architecture and reduced dataset assess the performance variations induced by the two strategies. Let T denote the number of training images, denote the total epochs in search, and denote the FL.OPs of the candidate architecture. The total FL.OPs (T-FLOPs) required throughout the training process calculated as:
T-FLOPs T xe
(A1)
Table A3 presents the results of the ablation study. In the table, "Acc w/ MA" denotes the early stopping test accuracy (after 4 epochs) from training on the original veta-architecture. When utilized it for candidate ranking, it achieves a correlation of 34.5%. However, owing to the complexity nature of the aetta-ccchtecture, it incurs substantial time costs, with T-FLOPs reaching 190.7B and time cost of 88.2 seconds per candidate. In contrast, "Acc w/ RA" denotes the replacement of the meta-architecture with reduced architecture, leading to a considerable reduction in time cost to 8.7B T-FLOPs and 29.2 seconds. However, this time reduction comes with significant performance loss, as the correlation drops to 8.1%, primarily attributed to the sensitivity nature of the test accuracy.
Finally, "MOTE + RA" represents our proposed solution that addresses trade-off concerns. Using MOTE's dual objectives, it maintains the same time cost while elevating the correlation 50.1% 1%, even surpassing the baseline. Subsequently, "MOTE + RA RD" indicates the further introduction of the reduced dataset, which reduces the number of training images used in the search. The time cost rapidly decreases with the smaller values of the hyperparameter r. At r = 10, it requires only 872.6G T-FLOPs and 13.2 seconds, resulting acceleration rates of 21.7x and 27.5x compared to the baseline, with slight decrease in correlation to 46.6%.
A.3 Implementation Details of MOTE-NAS
Despite MOTE consistently delivering impressive performance aided by reduction strategies, remains only proxy estimate. A comprehensive NAS method still requires the participation of
15

---

Search space
B=10x(1+iter/100)
Rand Sempling(108), when iter>0
Rand Sampling(8), when
Mutation
A A) x)
top-10%(P) by MOTE
E
Add to Pool P
A,X
2 Filtering by Euclidean Distance
Search Stage (total iters max(k,
Figure Al: The mutation step of the evolutionary process in the search. When the iteration count as opposed being greater than O, the number of samples taken from the search space varies. The sampling size B, increases with the number of iterations.
Conv
Pool
Conv
Out
0
0
0
0
0
Candidate OPs:
In, Out, Cony, Pool
0
0
1
1
0
0
1
0
0
1
0
0
1
0
0
0
Adjacency Matrix
Operation Matrix
Figure A2: Encoding cell into the adjacency and operation matrix. This entire procedure closely resembles the predictor-base based NAS.
accuracy to evaluate the actual performance of the candidates discovered, similar to NTK-hased NAS methods [6, 47, 51]. However, the time cost to obtain test accuracy is exceedingly high, often requiring thousands of GPU seconds. Hence, employ MOTE to assist the evolutionary process in the search stage to obtain a small subset of potentially promising candidates, and then evaluate them by test accuracy. As illustrated in Fig. A1, throughout the iterative evolutionary process, MOTE-NAS relies on mutations to quickly transform known promising architectures (in pool P) into candidates for the next iteration. The detailed steps of the mutation process outlined as follows:
1. Select the top 10% candidate architectures from the pool P based on MOTE.
2. Encode candidate architectures S into corresponding adjacency matrices A and operation matrices X.
3. Mutate S to obtain the mutated architectures S'. The encoded matrices A' and X' derived from 5' satisfy Σ(A'-A) -)* +XX -x)" 11
4. If the size of S' exceeds 10 k, randomly select and add them to the pool P. Otherwise, add 5" to P.
Fig. A2 illustrates the methodology employed in Step 2 to encode candidate architectures. In the cell- based search space [25, 49, 10, 38], candidate architectures are decomposed into smaller cells with predefined hyperparameters, including the total number of nodes within a cell, the maximum number of edges, and the allowable operations for each node. The fixed nature of these hyperparameters ensures the consistent shape of each cell. This consistency allows to represent edge connections within the cell using an adjacency matrix, as well as using one-hot vectors represent the selected operations of each node using operation matrix. Generating these fixed-shape shape matrices for all
16

---

Table A4: Comparison of the proposed MOTE-NAS and others on NASBench-201. Cost (s)
indicates the cost in seconds calculated on Tesla V100. Acc (%) represents the accuracy produced by the resulting network on respective datasets. Entries in bold with underlines indicate the best performance, and those bold alone represent the second-best performance.
CIFAR-10
CIFAR-100
ImgNet-16
Model
Ac)(%)
Cost(s)
Acc(%)
Cost(s)
Acc(%)
Cost(s)
NASWOT [29]
92.96
2.2K
70.03
4.6K
44.43
10K
TE-NAS [6]
93.90
2.2K
71.24
4.6K
42.38
10K
KNAS (k=20) [47]
93.38
4.4K
70.78
9.2K
44.63
20K
KNAS (k=40) [47]
93.43
8.8K
71.05
18.4K
45.05
40K
Eigen-NAS (k=20) [51]
93.46
4.4K
71.42
9.2K
45.53
20K
RS + LGA [30]
94.05
5,4K
71.56
7.0K
46.30
15K
REA LGA [30]
94.30
3.6K
72.42
5.4K
45.30
3.6K
MOTE-NAS-RS (k=5)
93.71
1.5K
71.59
1.7K
44.95
5.1K
MOTE-NAS-RS (k=10)
93.93
3.0K
72.11
3.1K
45.84
10.1K
MOTE-NAS-RS (k=20)
94.07
6.1K
72.60
6.8K
46.13
20.7K
Block x2
Block 2 x3
Block 3 x4
Block 4 x2
Block 5 X3
112 16
56 56 24
14x 14x14xx0 80
1 14x
Conv-1x1
7 7 960
Labels Num
Stem
GAP
960
FC
1280
FC
28
MobileNetV3
Rescoled Reduced Architecture
Stem
32 32
Cell
16x16x16
Cell
16x 16x16
Cell 3
8x8x32 32
Cell
8x8x32
Cell
8 x 32
GAP
FC
Labels Num
Tx
T>
DX
Dx
Figure A3: The bottom one the proposed rescaled reduced architecture, it consists of five indepen- dent cells that have different structure than other four.
candidates enables the representation of structural differences by calculating element-wise distances between corresponding matrices of two candidates and summing the results.
A.4 MOTE-NAS with Random Sampling
We remove the evolutionary search stage in MOTE-NAS and apply Random Sampling (RS) to test the performance of MOTE under various sampling strategies. This hybrid approach, termed MOTE-NAS-RS, initially draws 100 x & candidates from the search space by random sampling. The top-k candidates then selected based MOTE, further refined top-1 based on early stopping test accuracy (after 12 epochs). Our experiments on NASBench-201 involved 10 runs for each experiment, with the results averaged and presented in Table A4.
Although MOTE-NAS RS naturally lags behind MOTE-NAS due to the inherent simplicity of random sampling compared the evolutionary algorithm, MOTE-NAS S-RS notably outperforms mainstream NTK-based NAS. Achieving a final architecture accuracy of 94.07%, MOTE-NAS-PP RS surpasses KNAS [47], EigSn-NAS [51], and RS+LGA [30] CIFAR-10. On CIFAR-100, it reaches accuracy of 72.60%, even outpacing REA+LGA [30] at 72.42%. For ImgNet- MiNNeet----120- 120, closely trails RS+LGA. This highlights the effectiveness of MOTE, demonstrating superior performance when paired with a basic sampling strategy. Future studies may explore the potential combination of MOTE with NTK-based NAS without training, suggested [30].
A.5 Rescaled Reduced Architecture for MobileNetV3 Search Space
To analyze the performance of MOTE-NAS in the open search space, we used it to search for promising architectures the mobilenetv3 search space. Mobilenetv3 search space contains five inverted residual blocks with the SE module, on the other word, the search targets are five different
17

---

cells, but the proposed reduced architecture is only for single cell search. Therefore, we rescale the reduced architecture to achieve this new purpose, as shown Fig. A3, the rescaled version consists of five different cells to fit five blocks while still maintaining lightweight scale. It allows us to search an approximate open search space, and the architecture MOTE-NAS found has shown remarkable performance on imageNet- KK the results refer to Tab. 2.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our proposed methods and it's concept have been introduced in abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have shown in experiments that our method does not comprehensively outperform other advanced methods, and we have also illustrated the limitations of our method in experiments imagenet.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
Justification: This paper does not contain any mathematical proofs, only mathematical demonstrations of relevant concepts.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We conduct experiments using public benchmarks, such as NASBench- 101, NASBench-201. and ImageNet-IK, KK.. And we fully describe all hyperparameters and details that reproduce MOTE.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have described the experimental details in the paper and supplementary materials as much as possible, and we will make the code public on github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nipp.ccooocoo wwtppppllcgggccieec guwdes/ /Coee/ubmdeeaiicceeee wwreiisssonoooolo oniiiniyl) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: As mentioned before, we have fully exposed the hyperparameters of all experiments, including those required for training models.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We only use common statistical indicators, such as accuracy, kandall's correla- tion, and fully explain more special experiments. For example, the most experiments are the average of 10 runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We disclosed the type and number of GPUs used, and recorded the time consumed by the entire algorithm related experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https : http:///eurpp..eee http:pp....c.....i/ttttsttiiiiaa
Answer: [Yes]
Justification:
We have read and follow nips rules.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have fully cited the public benchmarks used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "THdm0cH86M", "reviewer": "Reviewer_NBd3", "summary": "This paper proposes a novel limited training NAS method that is able to rank the candidate architectures after training them for a limited number of epochs. The MOTE metric consists of two terms, the landscape term and the speed term. The landscape term is indicative of the loss landscape and it is the cross-entropy loss of the model $\\theta(g)$ obtained by the weighted sum of the initial parameter weights and the current parameter weights. This loss values are summed for a certain number of epochs. As the number of epochs increase, the weight assigned to the initial model weights decreases and that of the current model increases to result in $\\theta(g)$. The speed term is used to estimate the convergence speed of an architecture. It is computed by summing the cross-entropy loss at every epoch divided by the time taken to train for that epoch over all the epochs.\n\nRather than training the original architectures for N epochs, they derive a reduced variant for each architecture that is close enough to the original architecture. This reduced architecture is then trained on a reduced dataset to obtain the training dynamics necessary for MOTE. The reduced dataset is created by first obtaining the logits of a pretrained VGG-16 model and then for each label, k-means and farthest point sampling are applied to yield the reduced dataset.", "questions": "1.  Also, how many architectures did you sample for each search space? Did you run all the other baselines on the same architectures, same data augmentations that MOTE was evaluated on or did you use the results reported in their corresponding papers? Also, if you are evaluating the top-k architectures for some baselines, please change the table to be : (i) evaluation free, top-k=5, top-k=10) for all the baselines and report the accuracies accordingly. Right now the table is inconsistent. \n2. How does a linear combination represent the loss landscape? Given a point $\\theta_{B}$ in the loss landscape  and 2 other intermediate points $\\theta_{C}$ and $\\theta_{D}$, if we draw a line from init to B, it might not actually pass through intermediate points in the loss landscape. So how is it capturing the landscape? The main reason this might be working better than TSE is that it is assigning higher weight to the current weights as the epochs increase and lesser weight to the initial weight. How would it compare against TSE-E and TSE-EMA where the loss values of the initial few epochs are discounted. \n3. This paper used architecture reduction and dataset reduction to accelerate the training so that MOTE can be computed faster. Can you clarify if the k-means and the farthest point sampling is applied to datapoints corresponding to each label seperately to obtain a subset of them for that label? If not, how exactly is the clustering and the sampling done?\n Can you show how well the architectures the reduced dataset performs when compared to the original dataset by computing their correlations?  Why did you use k-means followed by FPS instead of other subset selection methods such as set-cover, facility location or coresets [5]? \n4. Can you please elaborate further what the meta architecture and the reduced architectures are comprised of? It is not clear to me.\n5. MOTE uses the landscape term and the speed term. In Table1, the estimation free version of MOTE seems to discover architectures that train much faster than the others. However, once top-k are considered,  the compute cost of training the discovered architecture is not any lesser than those discovered by some of the other baselines. Figure 1 shows that the landscape term performs better than the speed term. Can you also do an ablation and show how the MOTE-landscape and MOTE-speedterm perform in figure 7 and table 1? \n\n[5] https://cords.readthedocs.io/en/latest/strategies/cords.selection_strategies.SL.html#module-cords.selectionstrategies.SL.submodularselectionstrategy", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a novel limited training NAS method that also takes into account the convergence time while training a neural architecture. They were able to consistently achieve better correlation when compared to all the other training-free NAS baselines on 2 search spaces.", "weaknesses": "1. What is the correlation between the accuracies of the (i) reduced architectures and the original architectures (ii) training original architectures on reduced dataset and training the original architecture on the entire dataset (iii) the reduced architectures trained on the reduced dataset and the original architectures. Table A3 shows the correlation between the reduced architecture trained for 4 epochs but that is not the actual setting. The correlation already drops from 34.5 to 8.1%. MOTE when applied on the original architectures and the original datasets already would introduce some errors in ranking the architectures. The reduced architecture and the reduced dataset setting would further cascade the errors as the reduced setting is not 100% correlated to the original setting. So how can this reduced setting be used to begin with? Please include the correlation results for all the search spaces and all the datasets.\n\n2.  The authors should not be comparing their approach against zero-cost methods while MOTE considers training information.  Please evaluate the rest of the baselines also on the trained architectures for fair comparison. However, as mentioned above, it is not clear how the other methods would perform in the reduced setting owing to the concerns stated in 1.  Also, 7 seconds per architecture while the rest of the baselines take only 1 second on an average would become very expensive when the number of architectures in the search space is large. \n\n3. Please evaluate the efficacy of MOTE on other search spaces as well such as DARTS, ENAS [1],[2]. [3] demonstrated that training-free NAS methods don't generalize to tasks beyond image classification. Please evaluate on TransNASBench-101 [4] too similar to ZiCO.\n\n\n[1] NAS-Bench-301and the case for surrogate benchmarks for neural architecture search, Siems et al.\n\n[2] On Network Design Spaces for Visual Recognition, Radosavovic et al.\n\n[3] https://iclr-blog-track.github.io/2022/03/25/zero-cost-proxies/\n\n[4] TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search, Duan et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable feedback. However, we noticed that you might have misunderstood the core idea of MOTE. Specifically, the landscape term does not reduce the proportion of $\\theta_{init}$ (initial model weights) or increase the proportion of $\\theta$ (trained model weights) as epochs increase. MOTE is composed of a macroscopic landscape term and a microscopic speed term. The formula for the landscape term is:\n\n$\\theta(g)=(\\frac{g}{G})\\theta_{init}+(1-\\frac{g}{G})\\theta,$\n\nwhere $g$ ranges from 0 to $G$ and is independent of the number of epochs. Here, $G$ is a hyperparameter (default $G=10$), which indicates the granularity of the linear combination between $\\theta_{init}$ and $\\theta$. Therefore, even if the epochs for training candidate architectures increase, the proportion between $\\theta_{init}$ and $\\theta$ remains unchanged; only the position of $\\theta$ in the loss landscape changes. The landscape term is designed to observe the loss landscape on a macroscopic scale rather than focusing on the minute variations in training loss (as TSE and the speed term do).\n\n**Answers to Questions:**\n\n**A1.** The number of samples ranges from dozens to hundreds. Since we used an evolutionary algorithm for dynamic sampling and conducted each experiment independently ten times to take the average, the specific numbers vary slightly each time. For instance, in NASBench-201, the number of samples for MOTE-NAS-EF ranges from 60 to 100. Some results of our paper in Fig. 1, Fig. 5, Fig. 7 (SynFlow, TSE), and the predictors in Tab. 1 and MobileNetV2/V3 in Tab. 2 are our reproduced experimental results. In contrast, other results directly refer to their corresponding papers. During the MOTE generation process, we did not use any data augmentation techniques (refer to the \"GetTrainData\" function in the \"mote/gen_mote.py\" code in the supplementary materials). Not all NAS algorithms follow a \"search then evaluate\" process; for example, WeakNAS does not follow this process. Therefore, the top-k hyperparameter does not apply to all NAS methods, making it impossible to categorize every NAS method as \"evaluation-free\" or \"top-k\".\n\n**A2.** Plotting 2D or 3D loss landscapes with more information is computationally intensive [1], which is not conducive to efficient NAS. Indeed, the linear combination between two points does not fully capture the loss landscape, but it provides a convenient and quick assessment method. This method slices a 1D section through the loss landscape using two sets of model weights, and we can glean some of the landscape features by observing this slice [2]. The increase in epochs does not affect the proportion of $\\theta$; please refer to the begin sentence. Our implementation of TSE already includes removing the initial 20% loss values (similar to what TSE-E and TSE-EMA do). In short:\n1. The landscape term does not increase the proportion of $\\theta$ when increasing epochs.\n2. Since early loss values (20%) are eliminated in TSE, the proportion of $\\theta$ increases with the epoch. Therefore, the increase in the proportion of $\\theta$ is not why our proposed landscape term performs better than TSE. MOTE aims to describe the loss landscape from a microscopic perspective (speed term) and a macroscopic perspective (landscape term). The speed term can be considered a version of TSE that is aware of time variations.\n\n**A3.** Detailed explanations of RD are addressed in the global rebuttal. The ablation experiments between the original dataset and the reduced one are shown in Fig. 5 of our paper. As the sampling hyperparameter $r$ decreases (RD equals the original dataset if $r=100$), the performance of both the landscape and speed term will degrade, but less than the \"accuracy\". The core idea of RD is simple and intuitive (see the global rebuttal). Other methods can build RD, the attached file shows the ablation study in TABLE III. Based on the study, MOTE still works well if RD is built by other methods. The best reduction method will be explored in the future.\n\n**A4.** Cell-based search spaces require the combination of the predefined meta-architecture and the candidate cell to form a candidate architecture. The meta-architecture predefines the number of layers, downsampling method, other hyperparameters, etc. Different search spaces have distinct meta-architectures (e.g., NASBench-101 and NASBench-201) with huge time complexities. The goal of RA is to reduce redundant layers as much as possible, except for the cell layers. Although RA results in a simplified structure different from the actual candidate architecture (original meta-architecture + cell), it can construct an extremely lightweight and compact architecture to serve as a new  meta-architecture for NASBench-101 and NASBench-201.\n\n**A5.** MOTE-NAS-EF completely discards the evaluation stage to minimize total time consumption (search time + evaluation time). Since many training-free estimates are unreliable during the search stage, the evaluation stage needs to verify more architectures, leading to higher time consumption (e.g., KNAS). Currently, known training-free estimates are not truly \"free\" NAS methods.\n\nOther ablation experiments to evaluate the performance of the landscape term and speed term are addressed in the attached file (see Fig. 1 and TABLE IV).\n\n**Answers to Weaknesses:**\nDue to space limitations, these answers will be moved to global rebuttal.\n\n[1]. Li, Hao, et al. \"Visualizing the loss landscape of neural nets.\" Advances in neural information processing systems, 31 (2018).\n\n[2]. Goodfellow, Ian J., Oriol Vinyals, and Andrew M. Saxe. \"Qualitatively characterizing neural network optimization problems.\" International Conference on Learning Representations (2015).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vLQov8sSFb", "reviewer": "Reviewer_XpJH", "summary": "The paper introduces MOTE-NAS, a novel approach for efficient Neural Architecture Search (NAS). MOTE-NAS suggests a novel proxy utilizing both macro-level loss landscape smoothness and micro-level convergence speed to predict the performance. By utilizing reduced architectures (RA) and datasets (RD), MOTE-NAS achieves state-of-the-art accuracy on benchmarks such as CIFAR-10, CIFAR-100, and ImageNet-16-120 while significantly reducing computational costs.", "questions": "Typos\nTable 2, UTRE-NAS", "limitations": "The authors do mention some of the limitations in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The proxy proposed by the authors has a significant correlation with the final accuracy, showing particularly high correlation in the reduced search space.\nThe RA and RD methods can drastically reduce the dataset and architecture, providing substantial advantages in terms of speed.", "weaknesses": "The design of the skeleton for the reduced architecture seems to require human expert’s guide. It would be better if it were demonstrated that MOTE-NAS is robust across various RA configurations or if there were a general methodology for constructing RA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your high appreciation of our work. We greatly value your feedback.\n\n**Answers to Questions:**\n\n**A1.** If our paper is accepted, all the typos you mentioned will be corrected and eliminated in the camera-ready version.\n\n**Answers to Weaknesses:**\n\n**AW1.** The RA design might be achievable by other algorithms, which remains an open question. However, the proposed RA is simple yet achieves promising results in NABench-101 and NASBench-201, demonstrating its effectiveness across different datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FDT5XtSvB6", "reviewer": "Reviewer_Ndgx", "summary": "The paper presents MOTE, a training-based estimate for Neural Network accuracy, as a proxy method to accelerate Neural Architecture Search. The intuition behind MOTE is the non-convexity and non-linearity in the training loss landscape. Consequently, the authors provide a model that characterizes the training process's non-convexity and convergence speed and consider both in a few-shot joint optimization fashion. The proposed model is applied on a reduced architecture and dataset and requires a few training iterations. The evaluation results are promising and show the superiority of MOTE compared to predictor-based and few-shot NAS methods.", "questions": "- To what extent is MOTE sensitive to weight initialization?\n- What impact does the non-linearity in DNN architectures have on MOTE’s performances? \n- What is the exact process of the DR transformation? Can the same process be adapted to any type of dataset? And what’s the rationale behind choosing a VGG model?\n- What if the search space is a supernet (e.g., P-DARTS) or Transformer-based (e.g., NASViT)? How generalizable is MOTE to these search spaces?", "limitations": "- Highly non-linear DNNs are more challenging for MOTE---Thus the need for an ablation study on non-linear operations.\n- The limited scope of MOTE's evaluation on NAS search spaces—only on NAS-Bench-related search spaces.\n- The RD transformation method is highly specific, hindering its applicability to other tasks/datasets.\n- No detailed study on the impact of weights initlization.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles a critical problem—accelerating the NAS framework with few-shot performance predictors.\n- The problem is nicely formulated, and the motivations are clearly stated. The paper is also well-written and has a nice flow.\n- The evaluation section seems technically sound, with various experiments showing MOTE's superiority compared to state-of-the-art methods.", "weaknesses": "- Zero-shot estimates like NTK struggle with high sensitivity to the weights initialization method. While the authors have raised this point, there's no detailed study on MOTE's sensitivity to weight initialization . Furthermore, the type of initialization adopted by MOTE is not mentioned in the paper.\n- The non-convexity and non-linearity in DNNs is a core reason for the inefficiency of NTK-based methods. The authors should put more emphasis on this point, as it is a significant factor influencing the performance of MOTE. An ablation study on the impact of skip-connection and non-linear operations on the performance of MOTE can help draw more concrete conclusions about the robustness of the proposed approach when the DNN model is highly non-linear. \n- The RD transformation needs to be fully detailed. For CIFAR100, the authors employed a VGG model pretrained on ImageNet1K. However, it's not mentioned whether they perform a fine-tuning for CIFAR100 and the rationale behind the choice of the VGG model specifically or whether VGG should also be employed to perform an RD on any dataset.\n- While the comparison in the evaluation section is more focused on predictor-based and few-shot NAS frameworks, it does not specifically compare against zero-shot approaches [1, 2], especially the latest work that considers the feature map correlation [3].\n- The search spaces considered are mainly from the NAS-Bench line of work. MOTE should also be evaluated on supernets like P-DARTS and Transformer-based models. \n- The MOTE-NAS and MOTE-NAS-EF are built upon evolutionary algorithms. It's therefore unclear whether the superior performance comes from the evolutionary search or MOTE evaluation. The evolutionary search should be compared w/ and w/o MOTE and also against simple baselines (e.g., random search).\n- The conclusion section should discuss MOTE's limitations in more details, especially for multi-objective NAS methods, where efficiency and accuracy need to be optimized in a joint fashion.\n\n**References:**\n- [1]: Lin, Ming, et al. \"Zen-nas: A zero-shot nas for high-performance image recognition.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n- [2]: Bhardwaj, Kartikeya, et al. \"ZiCo-BC: A Bias Corrected Zero-Shot NAS for Vision Tasks.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n- [3]: Jiang, Tangyu, Haodi Wang, and Rongfang Bie. \"MeCo: zero-shot NAS with one data and single forward pass via minimum eigenvalue of correlation.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable feedback.\n\n**Answers to Questions:**\n\n**A1.** To determine whether MOTE is affected by random weight initialization, an additional experiment was conducted, as shown in TABLE II of the attached file. The results indicate that the effect of random weight initialization on MOTE's architecture search is minimal. This is mainly because the landscape and speed terms used in MOTE are derived from a training procedure, making MOTE insensitive to weight initialization.\n\n**A2.** To assess the sensitivity of MOTE to the use of skip connections, we selected four architectures from NASBench-201 to observe changes in accuracy and MOTE values with and without skip connections. These four architectures include Res-Conv(3x3) and Res-Conv(1x1), which use conv-3x3 and conv-1x1 with skip connections, respectively, and NoRes-Conv(3x3) and NoRes-Conv(1x1), which do not include skip connections. As shown in TABLE V and Fig. 2 of the attached file, the speed term, landscape term, and MOTE accurately reflected the changes in accuracy across the four architectures. When the accuracy increases, the values of the two terms and MOTE also increase, and vice versa. These experiments demonstrate that MOTE effectively captures the nonlinear architectures of DNNs.\n\n**A3.** The specific process of RD is discussed in the global rebuttal. We encode each image in the original dataset using a VGG network trained on ImageNet (not fine-tuned on CIFAR-100), inspired by the Inception Score method. Although VGG can be replaced, its inherent knowledge as a simple and high-capacity CNN makes it suitable for image encoding.\n\n**A4.** The MOTE also apply on the open search space(as shown Tab. 2 of our paper), where MOTE-NAS shows significant performance. To evaluate whether MOTE is suitable for other search spaces, an ablation study was performed on NASBench-301 (based on DARTS), as shown in TABLE I of the attached file. In this study, MOTE continues to outperform TSE and Synflow.\n\n**Answers to Weaknesses:**\n\n**AW1.** Please refer to A1.\n\n**AW2.** Please refer to A2.\n\n**AW3.** RD is a method used to accelerate MOTE. The selection method used in RD can be replaced by other methods. The ablation studies, shown in TABLE III of the attached file, indicate that MOTE performs well even if RD is replaced by other methods. Determining the best reduction method will be explored in future research.\n\n**AW4.** In Fig. 7 and Tab. 1 of our paper, most comparison methods such as NASWOT, TE-NAS, KNAS, Zen-Score, and ZiCO are training-free (or zero-shot) NAS methods. However, the term \"free\" only applies to the search stage; these methods still require GPU time in the evaluation stage. The core idea of MOTE focuses on the time spent in the evaluation stage, which is more time-consuming than the search stage.\n\n**AW5.** Please refer to A4.\n\n**AW6.** MOTE-NAS-RS is a version that replaces the evolutionary algorithm with random sampling (please see Table A4 in the supplementary materials).\n\n**AW7.** The limitations and contributions of this paper have been addressed in the global rebuttal and will be added to the camera-ready version if this paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y4E71uYzi8", "reviewer": "Reviewer_33pQ", "summary": "This paper proposes Multi-Objective Training-based Estimate (MOTE) for efficient NAS, leveraging landscape view and convergence speed to estimate the performance of neural architectures. It also introduces two reduction strategies for speeding up MOTE generation. Compared to other training-free NAS methods, MOTE achieves better search performance on NasBench201.", "questions": "I would like to ask about the results on ImageNet in Table 2. OFA costs 50 days, including the training and search process. Do you consider the training cost of the searched architectures? I think 0.1 days is far from enough to well-train a model on ImageNet.", "limitations": "No additional significant limitations. Please refer to the weaknesses. It would be beneficial if the paper could add future research directions for this work and extend this method beyond image classification tasks.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The paper is well-written.\n+ The search cost of NAS is very low, enabling NAS with very limited computing resources.\n+ The theoretical analysis is sound.\n+ The experimental results are promising.", "weaknesses": "- Code is not available.\n- The experimental results on ImageNet are not very good.\n- Regarding the title \"Multi-Objective Training-based Estimate for Efficient NAS,\" on NasBench201, only accuracy is considered as a metric, missing other important metrics like the number of parameters, FLOPs, latency, etc.\n- The reduction strategies lack novelty, as many low-fidelity estimation methods have been proposed, including early stopping[1], training with down-scaled models[1], and training on a subset of the data[2].\n\n\n\n[1]: B. Zoph, V. Vasudevan, J. Shlens, and Q. Le. Learning transferable architectures for scalable image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.\n\n[2]: A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter. Fast bayesian optimization of machine learning hyperparameters on large datasets. In Artificial intelligence and statistics, pages 528–536. PMLR, 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable feedback and positive evaluation of our work.\n\n**Answers to Questions:**\n\n**A1.** Yes, by adjusting the K value in MOTE-NAS to extend the search time, we can achieve better architectures. While the current architecture obtained through our method does not have the highest accuracy on ImageNet, the significantly lower time cost is a considerable advantage.\n\n**Answers to Weaknesses:**\n\n**AW1.** We have included the core code of MOTE in the supplementary materials, and the complete code will soon be available on GitHub (the link is not shown due to the policy of NIPS 2024).\n\n**AW2.** Please refer to A1.\n\n**AW3.** Generally, current NAS research focuses on finding the architecture with the highest accuracy within a given search space (NASBench-101 and NASBench-201) as quickly as possible. Other factors such as parameters, FLOPs, latency, and other metrics can be considered in our cost function in a hybrid form.\n\n**AW4.** The reduction strategies serve as acceleration methods for MOTE. When MOTE applies RD+RA, it does not experience significant performance loss in terms of \"accuracy.\" MOTE can achieve substantial acceleration with minimal performance loss and still outperform other NAS methods, as demonstrated in Fig. 7 and Tab. 1 of our paper. Therefore, we emphasize the robustness of MOTE rather than the novelty of the reduction strategies. In TABLE III of the attached file, we replaced the random selection-based RD-RS, and MOTE still performed well. This further demonstrates that the novelty of the reduction strategies is not a primary factor affecting MOTE's performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all four reviewers for your diligent efforts and valuable suggestions. We appreciate your feedback and comments, which will help us improve the quality of our paper. In this response, we will (1) summarize our paper's contributions and main limitations, (2) address the RD issue raised by the reviewers, (3) respond to the remaining concerns from Reviewer NBd3, and (4) include key experiments to clarify the raised questions.

**Contributions:**

1. Our proposed MOTE-NAS efficiently estimates training outcomes by jointly optimizing landscape view and convergence speed objectives. It captures the non-convex nature of DNNs and monitors convergence speed.
2. We introduce two reduction strategies to accelerate MOTE generation, making the process more lightweight.
3. Our MOTE-NAS establishes a new state-of-the-art accuracy-cost plot for NAS, with an evaluation-free version outperforming some NTK-based methods, such as KNAS.

**Limitations:**

1. Our work currently focuses on image classification; the applicability of MOTE-NAS to other tasks has yet to be explored.
2. While MOTE-NAS has been successfully applied in the closed search spaces (NASBench-101, NASBench-201) and the open search space (MobileNetV3), its effectiveness in broader search spaces, particularly with Transformers, requires further investigation.
---
**To address the reviewers' questions about RD:**

**Q1. Why is RD necessary?**

**A1.** MOTE generation relies on training, which is computationally expensive. For example, 12 epochs on NASBench-201 take about 200 GPU seconds. Reducing the number of training samples can significantly cut this time. RD is designed to accelerate the process by decreasing the number of training samples.

**Q2. Why is it based on CIFAR-100?**

**A2.** Randomly sampling images can lead to underfitting if some labels have too few samples. Sampling by labels instead ensures sufficient sample diversity. CIFAR-100, with many labels and fewer images, is ideal for this method. MNIST or CIFAR-10 would lose diversity at lower sampling rates (e.g., 10%).

**Q3. What is the specific process?**

**A3.** The process involves five steps:

Step 1: Encode each CIFAR-100 image using VGG-16 (trained on ImageNet, without fine-tuning on any datasets), taking the softmax logits.

Step 2: Sum and average the encoding results of images with the same label, resulting in 100 encoding categories for 100 labels.

Step 3: Cluster these 100 encoding categories into $r$ groups using the K-Means algorithm.

Step 4: Extract each center $c_i, 1 \leq i \leq r$ from the $r$ groups.

Step 5: Run FPS (Farthest Point Selection) within each group to find a representative point $l_i, 1 \leq i \leq r$ that is farthest from all $c_i$. Then, {$l_i$} is the set of representative labels required by RD.

Steps 3 to 5 aim to avoid selecting similar labels (e.g., bus and streetcar). We first group labels with K-Means and then use FPS to select labels that are farthest from all group centers.

**Q4. Is RD effective? Are there other methods?**

**A4.** RD has shown high compatibility with MOTE, significantly speeding up calculations despite some loss in performance (refer to Fig. 5). An intuitive method (RD-RS) that randomly samples images was also tested and is shown in the attached PDF file. As shown in TABLE III, although RD outperformed RD-RS, the latter still performed well, indicating that alternative methods for generating RD are possible and inspire further research.

---
**Weaknesses mentioned by Reviewer NBd3:**

**AW1.** Both reduction strategies (RA and RD) accelerate computation but harm performance, affecting both metrics 'accuracy' and 'MOTE'. However, MOTE's performance drops less than 'accuracy' (see Tab. A3). Despite the errors caused by the reduction strategies using RA and RD, MOTE can maintain a high correlation and significant acceleration. The insight of this paper is to prove MOTE's ability to capture the loss landscape from macro and micro perspectives, using the reduction strategy for computation speed gains. Experiments (Fig. 5 and Tab. A3) show that 'Accuracy' adapts poorly to this strategy, while MOTE's performance and speed impact are good trade-offs. Further experiments (Fig. 7, Tab. 1, Tab. 2) illustrate MOTE results with RA+RD applied, showing its effectiveness across different benchmarks, including NASBench-101, NASBench-201, and MobileNetV3 search spaces.

**AW2.** Not all comparison targets are training-free; methods such as TSE, LGA require training. The primary comparison, NTK-based estimates, though training-free, are unstable, as shown in Fig. 1, and a similar conclusion is supported by the LGA paper. The reduction strategies will degrade performance, so current comparison targets without reduction strategies represent their upper-performance bounds. Under this adverse situation, MOTE achieves a higher correlation (Fig. 7), making further reduction strategies and applications on other estimates unnecessary. On the other hand, evaluating the final performance of a searched architecture is very time-consuming (e.g., full training on ImageNet requires days to tens of days of GPU time). While MOTE takes 7 seconds, it is more effective than other estimates in the search stage for finding promising architectures, reducing the burden of the evaluation stage. Consequently, the overall time consumption is faster than other training-free methods (as shown in Tab. 1 and Tab. 2).

**AW3.** Our experiments in Tab. 2 were performed based on an open MobileNetV3 search space, employing a variation of RA—Rescale Reduced Architecture—for searching promising architectures (see section A.6 and Fig. A4). Additional experiments on NASBench-301 (based on DARTS) are included in the attached  file (please refer to TABLE I). Most NAS tasks focus on searching the backbone for image classification. Expanding NAS tasks is crucial in NAS research and not easily finished during the rebuttal time. We plan to explore this further.

Meta-review: This paper proposes to use both macro-level loss landscape smoothness and micro-level convergence speed as proxy for efficient neural architecture search, improving the accuracy and cost trade-off. The proposed method is effective in building a good correlation between the proxy and the final accuracy. Overall, the paper is well-written. Most reviewers are satisfied with the authors’ response to their concerns. Therefore, the AC recommends Accept. The authors should incorporate the comments and suggestions from reviewers when preparing the final version of the paper.
Decision: Accept (poster)