MOTE-NAS: Multi-Objective Training-based Estimate for Efficient Neural Architecture Search

Authors: yu-ming  zhang
jun-wei  hsieh
xin  li
ming-ching  chang
chun-chieh  lee
kuo-chin  fan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:57:17

Abstract: Neural Architecture Search (NAS) methods seek effective optimization toward performance metrics regarding model accuracy and generalization while facing challenges regarding search costs and GPU resources. Recent Neural Tangent Kernel (NTK) NAS methods achieve remarkable search efficiency based on a training-free model estimate; however, they overlook the non-convex nature of the DNNs in the search process. In this paper, we develop Multi-Objective Training-based Estimate (MOTE) for efficient NAS, retaining search effectiveness and achieving the new state-of-the-art in the accuracy and cost trade-off. To improve NTK and inspired by the Training Speed Estimation (TSE) method, MOTE is designed to model the actual performance of DNNs from macro to micro perspective by draw loss landscape and convergence speed simultaneously. Using two reduction strategies, the MOTE is generated based on a reduced architecture and a reduced dataset. Inspired by evolutionary search, our iterative ranking-based, coarse-to-fine architecture search is highly effective. Experiments on NASBench-201 show MOTE-NAS achieves 94.32% accuracy on CIFAR-10, 72.81% on CIFAR-100, and 46.38% on ImageNet-16-120, outperforming NTK-based NAS approaches. An evaluation-free (EF) version of MOTE-NAS delivers high efficiency in only 5 minutes, delivering a model more accurate than KNAS.

Main paper: 
 introduction: 
 Neural Architecture Search (NAS) [52] tackles the challenge of automating the design and search for suitable neural network architectures in many applications. NAS approaches mainly comprise two stages: a model search stage dedicated to identifying promising candidates within the architecture search space, and an evaluation stage where candidate performance is assessed. In the search stage, the search space can be exponentially large. To reduce search complexity, the cell-based tabular search space [49, 10, 38] is widely considered. Reinforcement learning [52, 2, 39] and evolutionary algorithms [27, 31, 33, 46, 9] are also used to accelerate the search process. However, the primary cost of NAS lies in the evaluation stage, where candidate models must undergo intensive training until convergence to obtain a precise performance assessment. This incurs significant time costs (e.g., NASBench-201 requires 3-10K GPU seconds for convergence after 200 epochs). So, various proxy estimates (e.g., zero-cost proxy [1] and training speed estimation [34] ) have been developed to rank candidates, mitigating computational demands for model evaluation.Recently, several estimates based on NTK have been proposed, including TE-NAS [6] , KNAS [47] , and Eigen-NAS [51] . NTK-based estimates serve as condensed representations of gradients and their correlations. The NTK theory aims to macro-model the actual performance of Deep Neural Networks (DNNs). It assumes that the performance of an infinite-width DNN can be fully described by the The predictive performance of the proposed two terms gradually improves as epochs increase.NTK at initialization and that the NTK's value remains unchanged after training [19] . Consequently, NTK-based estimates can predict the actual performance of candidate architectures without training. However, real DNNs have limited width and exhibit a highly non-convex nature, leading the NTK to encounter significant nonlinear changes during training and resulting in limitations in accurately predicting the actual performance of candidates. Fig. 1 shows the NTK suffers unstable performance during training.To address this macro-modeling issue, we propose a novel landscape term that leverages the idea from the study [12] to linearly combine the differences between the initial weights and the post-training weights of the candidate architectures, which allows us to capture the non-convex nature of these candidates by the landscape slice. If the landscape is flatter, the candidate's performance tends to be better, as it more readily converges to the global optima. Furthermore, studies from a microaspect have been conducted to model this issue, such as Snip [20] , Grasp [41] , and SynFlow [40, 1] , which use gradient change to predict the performance of candidate architectures. As the gradient is integrated into the model training, it may reflect the current effectiveness of changes in the model training. Similarly, TSE [34] directly sums up the training loss to represent the convergence speed to predict candidate performance. In summary, these methods are more intuitive. Although they may not theoretically capture the macroscopic non-convex nature of DNNs, in practice, as shown in Fig. 1 , the performance TSE even exceeds NTK. This observation inspired us to propose a speed term that sums the training loss per unit of time, providing a microscopic description of the convergence speed of candidates.This paper introduces a Multi-Objective Training-based Estimate (MOTE) that considers landscape term for macroscopic view and speed term for microscopic view to predict the performance of candidates in a joint optimization. This dual perspective offers a comprehensive consideration and an accurate estimate for candidates. Furthermore, we introduce two reduction strategies to reduce the time costs by generating MOTE, which consists of landscape and speed terms. We also present MOTE by integrating the evolutionary algorithm [32] , named MOTE-NAS. This design is based on a coarse-to-fine iterative procedure for architecture search. In the search stage of MOTE-NAS, we maintain several dozen to several hundred promising candidate architectures in the pool. In the evaluation stage, we first select the top-K architectures based on the MOTE, then select the best architecture by the early stopping version of the test accuracy. We further develop a stripped-down, evaluation-free version named MOTE-NAS-EF, which achieves high efficiency and can finish a NAS run in merely eight minutes.MOTE-NAS outperforms mainstream NTK-based NAS methods. Fig. 2 compares it with TE-NAS [6] , KNAS [47] , Eigen-NAS [51] , and Label-Gradient Alignment (LGA) [30] on CIFAR-100 of NASBench-201. The accuracy of the final architecture discovered by MOTE-NAS is significantly superior to other methods. In our experiments, MOTE-NAS achieved 94.32% accuracy on CIFAR-10, 72.81% on CIFAR-100, and 46.38% on ImageNet-16-120. The evaluation-free version, MOTE-NAS-EF, achieves results comparable to KNAS's, where the search is completed in only eight minutes.The technical contributions of this work are summarized as follows:• Our proposed NAS approach utilizes an efficient training-based estimate to optimize landscape view and convergence speed objectives jointly. This design comprehensively captures the nonconvex nature of DNNs from a macro perspective and monitors the convergence speed from a micro perspective, enabling precise actual performance predictions for desired architectures. 2 Related Work DARTS. Instead of exploring a discrete set of architectures, the Differentiable ARchiTecture Search (DARTS) [26] transforms the combinatorial challenge of finding optimal operations into a continuous optimization problem within a differentiable search space. A notable challenge with DARTS is the potential dominance of easily optimized operators, such as skip-connections and pooling operations in the early stages. This issue impedes selecting more potent operations like convolutions with large kernels. In [7, 22] , a robust prior is introduced to restrict the number of skip connections within a cell to a predefined value. The progressive search strategy employed in P-DARTS [7] gradually increases network depth and refines candidate operations based on mixed operation weight. DARTS methods are efficient when running with limited computational resources. However, the architecture found comes with stability and generalizability issues. Furthermore, DARTS algorithms often prefer shallow and wide structures [37] .On the other hand, NAS methods generally comprise two stages: the search stage and the evaluation stage. The former focuses on collecting promising candidate architectures, while the latter involves assessing the performance of these candidate architectures.Search Stage. Numerous studies have concentrated on the search stage. Some NAS methods leverage reinforcement learning [52, 2, 39] , while others are rooted in evolutionary algorithms [27, 31, 33, 46, 9] . The cell-based tabular search space [49, 10, 38] is effective in reducing exhaustic search into a more manageable scale, using a meta-architecture with predefined operations, hyperparameters, filters, and strides. The candidate architectures under consideration range from tens to hundreds of thousands of candidate architectures. Recently, predictor-based approaches [24, 28, 43, 11, 42, 45, 15] have gained popularity. These methods construct predictors trained with architecture-accuracy pairs to forecast the performance of a candidate architecture. These predictors encompass a range of models from graph convolutional networks [17] to MLPs and other regression models. However, obtaining a high-quality set of architecture-accuracy pairs for NAS is non-trivial.Evaluation Stage. Compared with the cost of the search stage, the burden of NAS mainly resides in the time-consuming evaluation process. Various studies have proposed proxy estimates to reduce the need for a real performance evaluation. A prominent approach is zero/few-cost estimate [1, 34, 23, 29, 6, 47, 51, 30] , which substitutes performance indicators such as accuracy with alternative estimates. The zero-cost proxy [1] introduces zero-cost performance estimates [20, 41, 40] and TSE [34] introduces a training speed estimate. More recently, Neural Tangent Kernel (NTK)-based estimates [6, 47, 51, 30] have gained popularity based on the assumption that DNNs can predict their convergence performance at initialization. However, it is experimentally found in [30] that NTKs cannot capture the non-linear characteristics of DNN training dynamic well. Recent NAS methods integrate multiple approaches to achieve remarkable performance. For instance, OMNI [44] and ProxyBO [36] propose few-cost NAS methods by combining zero-cost estimates with more resource-intensive techniques like Bayesian optimization and performance predictors. 
multi-objective training-based estimate: 
 NTK theory tries to describe gradient change by a macro-perspective, but its fundamental assumption about infinite-width DNN cannot fit the real DNNs that have finite width. For example, in KNAS [47] , GKH asserts the existence of a gradient feature that can serve as a coarse-grained proxy to support downstream training when evaluating randomly initialized architectures. However, this does not propose a concrete solution to identify such a non-linear gradient feature during training.LGA [30] finds that the sensitivity for weight initialization that leads to NTK cannot perform stably, demonstrating that the value of NTK does not change [19] . In addition, Fig. 1 shows that NTK do not accurately predict the actual performance of candidates in practice. Despite NTK-based estimates, there are estimates that make predictions by micro-perspective. Among them, TSE [34] sums up the training loss as a proxy estimate to represent the convergence speed. The convergence speed as an important factor for model performance has been extensively discussed in the literature [13, 16, 34] . Specifically, these studies inspired the proposal of MOTE. MOTE introduces a new landscape term to capture the non-convex nature of models by a macro-perspective through the linear combination between two weights to observe the loss landscape. Simultaneously, MOTE introduces another new speed term to measure the convergence speed of the model by a micro-perspective through calculating the unit time training loss. Incorporating the multi-objectives enables MOTE to comprehensively describe candidates' non-convex nature and convergence speed from a macro-to-micro perspective.Landscape Term. In order to capture non-convexity of loss landscape by macro-perspective, we introduce landscape term that linearly combines the two weights before and after few-training to interpolate the weights for intermediate state, and then calculates the loss values (cross-entropy) of these weights, which means to cutting a section from loss landscape to observe its nature [12] , so we sum these loss values to determine whether the loss landscape is smooth, detailed follows.Let θ denote the trained weights of the candidate. To macro-model the actual performance of candidate architectures, we linearly combine the initial weights with the trained weights θ to obtain the combined weights, denoted θ(g), for describing the non-convex landscape of the candidates. Then,we obtain the combined weights θ(g) in the form:EQUATIONwhere θ init denotes the initial weights, and G is the number represent how dense the linear combination and set to be 10 based on most of the experiments from [12] . Let Y θ(g) pd denote the model prediction labels based on the weights θ(g), and Y gt be the ground-truth labels of the training data. Then, we use cross-entropy to measure the difference between Y θ(g) pd and Y gt . After that, we sum the loss value of these middle weights as landscape term as follows.EQUATIONThe poposed landscape term can capture the non-convexity of models where a lower value indicates a flatter loss landscape, implying an efficient convergence of the candidate and avoiding the problem of sharp minimum [50] .Speed Term. On the other hand, the idea to model the actual performance of candidates by microperspective, such as TSE, still perform strong and important, as shown in Fig. 1 . Therefore, we are inspired by TSE [34] to introduce speed term. It first calculates training losses (cross-entropy) over an epoch, and divides it by the time cost of an epoch, converting it into a unit time training loss to measure convergence speed, which means that it observes the convergence speed of the candidates under standardized time expenditure, which helps to standardize the measurement of architectures of different sizes. It can also be seen from the Fig. 1 that the speed term performs better than the TSE [34] . The detailed speed term is defined:EQUATIONwhere Y e pd denotes the model prediction labels in epoch e, and Y gt means the ground truth labels of the training data. Then, we use cross-entropy to calculate the loss between Y e pd and Y gt . After that, we divide the loss value by the time cost t e within epoch e and then sum up all as speed term. Note that a lower value indicates faster convergence, which could imply better performance. Now, MOTE integrates the landsacpe term and speed term to model actual performance of candidates from macro to micro perspective, and is defined as follows:EQUATIONwhere the first term is the proposed landscape term, the latter term is the proposed speed term, where J θ denotes the function used to determine whether the loss landscape is smooth by linear combining initial weights θ init and trained weights θ. E is the number of maximum training epochs to search candidate architectures, l e is the training loss(usually measured by cross-entropy) for the eth epoch, t e denotes the time cost for the epoch e, and the function f denotes the non-linear transformation to restrict the range of values.Due to the different ranges of landscape term and speed term, we use the box-cox transformation [3] to transform and normalize them; more comparisons of other transformation methods are detailed in Appendix A.1. MOTE can consider both objectives by adding transformed method to assess their combined impact. Since lower values for both landscape term and speed term suggest a potentially better performance of the models, a smaller MOTE value indicates a better performance in practical application. Fig. 2 (a) illustrates the concepts of landscape term and speed term. Reduced Architecture (RA). MOTE is not the actual performance of DNN after combining the cell with meta-architectures. Instead, MOTE is generated from the change in weight and loss acquired during training. The generation of MOTE relies on the proposed reduced architecture. The reduced architecture is designed to eliminate redundant layers from most meta-architectures [49, 10] , resulting To minimize the computational cost of MOTE generation, we propose a sub-dataset derived from CIFAR-100 [18] , referred to as the reduced dataset. As shown in Fig. 4 , the process of constructing the reduced dataset involves several steps: 1) We use a VGG-16 model pre-trained on ImageNet-1K [8] to extract logits from images; 2) Flattened the logits and averaged them according to specific label, resulting in label embedding codes; 3) K-Means to cluster the label embedding codes into r groups; 4) Farthest Point Sampling (FPS) to select r label embedding codes from r group to represent the r labels of the reduced dataset; 5) The images associated with the chosen r labels are reserved for building the reduced dataset. The reduced dataset is a proxy dataset and a sub-dataset of CIFAR-100, containing a representative set of r labels. When r is set to 100, the reduced dataset is equivalent to CIFAR-100. As r decreases, the reduced dataset becomes smaller and easier to fit for candidate models. However, regardless of the value of r, the reduced dataset maintains the original image distribution for each label. The K-means and FPS techniques ensure that the reduced dataset represents a significant part of CIFAR-100 even when r < 100. The most important thing is MOTE generation that rely on a reduced dataset can save remarkable time cost.Two Terms of MOTE with Reduction Stratigies. To further observe how reduction strategies work, we randomly selected 1K candidates of NASBench-201 as toy experimental subjects. As shown in Fig. 5 , the early stop version of test accuracy (after 12 epochs) has a high correlation with test accuracy (after 200 epochs), but training then getting it required about 220 gpu seconds, which is still a remarkable cost. When RA is applied, the time cost decreases 60%, but the correlation also drops to about 0.5 from 0.65. After further RD is applied, as the extraction ratio r gradually decreases (the further to the left the smaller r is), the time cost is also greatly saved, but the correlation suffers intolerable losses. The correlation of the leftmost (r = 10) is even less than 0.1. In contrast, the proposed landscape term and speed term always maintain a high correlation when applying RA and RD. As the extraction ratio r gradually decreases, the time cost is reduced to about 10 gpu seconds from 220 seconds, and the correlation is still about 0.65. This is because the two proposed terms based on weight and loss changes do not rely on the excluded middle state of true or false. More comparisons between various reduction strategies refer to Appendix A.2.Although MOTE consistently maintains impressive performance under the influence of the reduction strategy, MOTE remains a proxy estimate. A comprehensive NAS method still requires the participation of accuracy to evaluate the actual performance of the candidates discovered in the evaluation stage, similar to previous NAS methods [6, 47, 51, 30] . However, as mentioned at the outset, the time cost of accuracy is exceedingly high, often demanding thousands of GPU seconds. Therefore, our proposed MOTE-NAS first employs MOTE to assist the evolutionary process in obtaining a small subset of promising candidates. Then, through the early stopping version of accuracy assessment, the best is identified. The entire procedure is illustrated in Fig. 6 .In the search stage, MOTE-NAS maintains a pool of promising candidates P , with batch size B initially set to 10. With the continuous evolution loop, every ten cycles, B is incremented by 10. In each evolutionary process, MOTE is generated to sort P and take the top 10% of the candidates to the mutation stage. For the mutation stage, we are inspired by predictor-based methods [43, 11, 45] to encode candidates into adjacency and operation matrices. Subsequently, we calculate the Euclidean distance between each pair of candidates to select up to 10 × k mutated candidates, which are then added to P . This evolutionary loop continues max(k, 10) times, then stops and enters the evaluation stage. For the evaluation stage, MOTE is used to select the top k architectures of P (k = 5, 10, 20), then select their best architecture based on the early topping version of the test accuracy.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We have read and follow nips rules.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experimental results: 
 Experimental Setup. We used NASBench-101 and NASBench-201, both cell-based search spaces. NASBench-101 has 423,621 candidates trained on CIFAR-10 for 108 epochs. NASBench-201 includes 15,625 candidates trained on CIFAR-10, CIFAR-100, and ImageNet-16-120 for 200 epochs each. Computation was on Tesla V100 GPUs, with MOTE or MOTE-NAS costs calculated specifically on V100. Our experiment had three parts: comparing MOTE with other estimates on NASBench-101 and NASBench-201, evaluating MOTE-NAS against other NAS methods on NASBench-201, and using MOTE-NAS to search for a mobilenet-like architecture on ImageNet-1K. Further, we visualize the rankings of MOTE and KNAS to perceive their differences in Fig. 8 . MOTE is generated from the proposed reduced architecture and dataset. We used reduced dataset with a sampling rate hyperparameter r = 10 based on the results in Fig. 5 . The hyperparameters are batch size 256, epochs 50, learning rate 0.001 with Adam optimizer, and cross-entropy loss function. MOTE generation per cell took about seven GPU seconds under these settings.To explore the performance gaps between MOTE and other relevant estimates, we compared NASBench-101 and NASBench-201. We ranked candidates using speed term, landscape term and MOTE or other estimates and compared the resulting rankings to the actual ranking, calculating Kendall's Tau correlation to gauge the performance of these estimates. The experimental results are presented in Fig. 7 . NTK-based Estimates. MOTE leverages the landscape term to capture the non-convex nature from the candidate architectures, compensating for the shortcomings in NTK-based estimates. As shown in Fig. 7 , MOTE achieves correlations of 0.68, 0.66, 0.62, and 0.47 in NASBench-101 and NASBench-201. Compared to TE-NAS and KNAS, MOTE shows performance improvements ranging from 13% to 62% on NASBench-201. In addition, we observed poor performance of TE-NAS and KNAS in NASBench-101, with KNAS demonstrating a correlation of merely 0.09. This illustrates that NTKs struggle to adapt to larger search spaces and more diverse candidate architectures in environments like NASBench-101. In contrast, MOTE maintains a high correlation of 0.47, demonstrating a significant increase of 261% and 422% compared to TE-NAS and KNAS, respectively. Compared to the state-of-the-art ZICO, MOTE still performs better than it does on benchmarks. Remarkably, these MOTE performance gains were achieved with an average cost of only seven seconds per candidate.Other Estimates. When comparing MOTE with other estimates such as SynFlow [40, 1] and TSE [34] , MOTE maintains a significant advantage. Compared to SynFlow and TSE, MOTE shows performance gains of 11% to 45% on NASBench-201 and 96% and 104% on NASBench-101, respectively. It should be noted that both TSE and MOTE are training-based estimates, and MOTE accelerates 2× faster than TSE, outperforming it significantly.Additionally, we can see that the proposed speed term and landscape term also perform well in Fig. 7 , both of which are essential components of MOTE.We depicted the distribution of MOTE in comparison to KNAS to facilitate analysis regarding MOTE and NTK-based estimates. As shown in Fig. 8 , the experimental results involve random selection of 1K candidate architectures from NASBench-201 (CIFAR-100). The x-axis represents the estimate ranking based on MOTE or KNAS, while the y-axis represents the actual ranking based on the test accuracy after 200 epochs. Each node in the figure represents a candidate architecture, with its (x, y) coordinates indicating its position in the estimate and the actual rankings, respectively. Both the estimate ranking and the actual ranking are sorted from high to low scores.In the left subfigure of Fig. 8 , we present the distribution of KNAS, while the right subfigure displays the distribution of MOTE. In general, MOTE is more concentrated than KNAS, indicating that MOTE is closer to the actual ranking of the candidates than KNAS. This observation aligns with the superior performance of MOTE in Kendall's Tau correlation comparisons, as shown in Fig. 7 . Further, focusing on the high-performance region (the blue circle in the lower left corner of the two sub-figures), MOTE exhibits a more concentrated trend compared to the chaotic distribution of KNAS. This suggests that MOTE outperforms KNAS in predicting promising architectures. Finally, examining the purple boxes in the lower right and upper left corners of the two subfigures reveals candidates for which the estimate indicates good. However, the actual performance is poor, or vice versa. In both cases, MOTE's misjudgments are significantly fewer than those of KNAS, visually confirming that MOTE is highly competitive compared to the mainstream NTK-based estimate. [47, 51] . The time consumption of MOTE-NAS lies in training candidates to obtain MOTE during the search stage and the early stopping accuracy obtained during the evaluation stage. However, the cost of each MOTE is only about seven gpu seconds, so primary consumption is still to evaluate the top-k candidates. We have set the k range from 5, 10, 20. When k = 5, the final architecture found by MOTE-NAS achieves significantly higher accuracy on three datasets of NASBench-201, compared to TE-NAS [6] , KNAS [47] , and Eigen-NAS [51] , with speedups ranging from 1.9× to 6.9×. It only slightly lags behind REA + LGA [30] Rescaled Reduced Architecture for Macro-Search. Our study introduces a Rescaled Reduced Architecture for Macro-Search, where we modify the reduced architecture to accommodate the simultaneous assessment of five blocks and their collective performance. By expanding the cell layers from two to five and independently sampling the structure of each layer, our rescaled approach enables macro-search capabilities beyond single-cell exploration. Further technical specifics of this rescaled reduced architecture are outlined in Appendix A.5. Subsequently, employing MOTE-NAS with this modified architecture, we conducted a search within the mobilenetv3 space under approximately 400M FLOPs. Following 200 epochs of training using 10 GTX 2080Ti GPUs on the imagenet-1K dataset, the results (see Table 2 ) demonstrate the efficacy of our approach. While the accuracy of MOTE-NAS (76.2% and 77.1%) trails slightly behind OFA's 77.7%, MOTE-NAS achieves this with a significantly reduced computational cost of 0.1 GPU days compared to OFA's 50 days, representing a 500x speed improvement. Furthermore, our retraining of ZICO's best architecture yielded a 75.8% accuracy on imagenet, surpassed by MOTE-NAS with its superior accuracies and a 4x faster search cost than the 0.4 day of ZICO. 
conclusion: 
 In Table A1 shows the comparative results of four versions of MOTE. MOTE w/ box-cox performs the best. The box-cox transformation leads to a notable improvement compared to the original in correlation: 2% improvement for CIFAR-10, 2% improvement for CIFAR-100, and 3% improvement for ImageNet-16-120. These improvements are attributed to the ability to contract outliers, bringing them closer to the normal data distribution and reducing the impact on MOTE. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have fully cited the public benchmarks used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Although the proposed MOTE-NAS approach can greatly reduce the required NAS cost in producing a suitable network architecture, the time cost of model training and evaluation is still the bottleneck and trade-off. We further analyze the two proposed reduction strategies and present the results of the ablation study regarding the computation time trade-off. Fig. 3 provides the MOTE generation pipeline that contains the reduced architecture and the reduced dataset.Experiments setting. We conducted a random sampling of 1,000 candidate architectures from NASBench-101 and NASBench-201, respectively. Notably, these candidate architectures consist of their predefined meta-architecture and cells within their search spaces. Each cell shares identical hyperparameters and operations. Therefore, to obtain complete models, the candidate architectures (equivalent to candidate cells) must be assembled into either the meta-architecture or our reduced architecture. The proposed reduced architecture speeds up model search by retaining only a single convolutional layer as the stem layer and two cell layers, eliminating the majority of layers from the original meta-architecture of NASBench-101 and NASBench-201. The reduced dataset is a representative proxy subset of CIFAR-100, with the aim of accelerating the training process by reducing the number of training images, thus reducing the time needed for each training epoch.Reduced Architecture vs. Meta-Architecture. Table A2 compares the model parameters and FLOPs of predefined meta-architecture and the peoposed reduced architecture, when evaluated on the NASBench-101 and NASBench-201 benchmarks. For the NASBench-101 case, the reduced architecture contains much fewer parameters of 34.4K, in contrast to the 2.3M parameters of the original meta-architecture. This is a significant reduction of 64×. Similarly, the FLOPs for the reduced architecture amount to 18.3M, while the original meta-architecture records 1.0G FLOPs, indicating an acceleration of 435×. In the case of NASBench-201 candidates, the reduced architecture exhibits reductions/accelerations of 15× and 4× compared to the original meta-architecture. These results indicate a superior advantage of the proposed reduction strategy in both memory requirements and computational speed on the two benchmarks. We incrementally introduce the reduced architecture and reduced dataset to assess the performance variations induced by the two strategies. Let T denote the number of training images, e denote the total epochs in search, and f denote the FLOPs of the candidate architecture. The total FLOPs (T-FLOPs) required throughout the training process are calculated as:EQUATIONTable A3 presents the results of the ablation study. In the table, "Acc w/ MA" denotes the early stopping test accuracy (after 4 epochs) from training on the original meta-architecture. When utilized it for candidate ranking, it achieves a correlation of 34.5%. However, owing to the complexity nature of the meta-architecture, it incurs substantial time costs, with T-FLOPs reaching 190.7B and a time cost of 88.2 seconds per candidate. In contrast, "Acc w/ RA" denotes the replacement of the meta-architecture with a reduced architecture, leading to a considerable reduction in time cost to 8.7B T-FLOPs and 29.2 seconds. However, this time reduction comes with a significant performance loss, as the correlation drops to 8.1%, primarily attributed to the sensitivity nature of the test accuracy.Finally, "MOTE + RA" represents our proposed solution that addresses trade-off concerns. Using MOTE's dual objectives, it maintains the same time cost while elevating the correlation to 50.1%, even surpassing the baseline. Subsequently, "MOTE + RA + RD" indicates the further introduction of the reduced dataset, which reduces the number of training images used in the search. The time cost rapidly decreases with the smaller values of the hyperparameter r. At r = 10, it requires only 872.6G T-FLOPs and 13.2 seconds, resulting in acceleration rates of 21.7× and 27.5× compared to the baseline, with a slight decrease in correlation to 46.6%.Despite MOTE consistently delivering impressive performance aided by reduction strategies, it remains only a proxy estimate. A comprehensive NAS method still requires the participation of Figure A1 : The mutation step of the evolutionary process in the search. When the iteration count is 0, as opposed to being greater than 0, the number of samples taken from the search space varies. The sampling size B, increases with the number of iterations.Figure A2 : Encoding a cell into the adjacency and operation matrix. This entire procedure closely resembles the predictor-based NAS.accuracy to evaluate the actual performance of the candidates discovered, similar to NTK-based NAS methods [6, 47, 51] . However, the time cost to obtain test accuracy is exceedingly high, often requiring thousands of GPU seconds. Hence, we employ MOTE to assist the evolutionary process in the search stage to obtain a small subset of potentially promising candidates, and then evaluate them by test accuracy. As illustrated in Fig. A1 , throughout the iterative evolutionary process, MOTE-NAS relies on mutations to quickly transform known promising architectures (in pool P ) into candidates for the next iteration. The detailed steps of the mutation process are outlined as follows:1. Select the top 10% candidate architectures from the pool P based on MOTE.2. Encode candidate architectures S into corresponding adjacency matrices A and operation matrices X.3. Mutate S to obtain the mutated architectures S ′ . The encoded matrices A ′ and X ′ derived fromS ′ satisfy (A ′ -A) 2 + (X ′ -X) 2 2 ≤ 1.4. If the size of S ′ exceeds 10 × k, randomly select 10 × k and add them to the pool P . Otherwise, add S ′ to P .Fig. A2 illustrates the methodology employed in Step 2 to encode candidate architectures. In the cellbased search space [25, 49, 10, 38] , candidate architectures are decomposed into smaller cells with predefined hyperparameters, including the total number of nodes within a cell, the maximum number of edges, and the allowable operations for each node. The fixed nature of these hyperparameters ensures the consistent shape of each cell. This consistency allows to represent edge connections within the cell using an adjacency matrix, as well as using one-hot vectors to represent the selected operations of each node using an operation matrix. Generating these fixed-shape matrices for all candidates enables the representation of structural differences by calculating element-wise distances between corresponding matrices of two candidates and summing the results.We remove the evolutionary search stage in MOTE-NAS and apply Random Sampling (RS) to test the performance of MOTE under various sampling strategies. This hybrid approach, termed MOTE-NAS-RS, initially draws 100 × k candidates from the search space by random sampling. The top-k candidates are then selected based on MOTE, further refined to top-1 based on early stopping test accuracy (after 12 epochs). Our experiments on NASBench-201 involved 10 runs for each experiment, with the results averaged and presented in Table A4 .Although MOTE-NAS-RS naturally lags behind MOTE-NAS due to the inherent simplicity of random sampling compared to the evolutionary algorithm, MOTE-NAS-RS notably outperforms mainstream NTK-based NAS. Achieving a final architecture accuracy of 94.07%, MOTE-NAS-RS surpasses KNAS [47] , Eigen-NAS [51] , and RS+LGA [30] on CIFAR-10. On CIFAR-100, it reaches an accuracy of 72.60%, even outpacing REA+LGA [30] at 72.42%. For ImgNet-16-120, it closely trails RS+LGA. This highlights the effectiveness of MOTE, demonstrating superior performance even when paired with a basic sampling strategy. Future studies may explore the potential combination of MOTE with NTK-based NAS without training, as suggested by [30] .A. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: This paper proposes to use both macro-level loss landscape smoothness and micro-level convergence speed as proxy for efficient neural architecture search, improving the accuracy and cost trade-off. The proposed method is effective in building a good correlation between the proxy and the final accuracy. Overall, the paper is well-written. Most reviewers are satisfied with the authors’ response to their concerns. Therefore, the AC recommends Accept. The authors should incorporate the comments and suggestions from reviewers when preparing the final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel limited training NAS method that is able to rank the candidate architectures after training them for a limited number of epochs. The MOTE metric consists of two terms, the landscape term and the speed term. The landscape term is indicative of the loss landscape and it is the cross-entropy loss of the model $\theta(g)$ obtained by the weighted sum of the initial parameter weights and the current parameter weights. This loss values are summed for a certain number of epochs. As the number of epochs increase, the weight assigned to the initial model weights decreases and that of the current model increases to result in $\theta(g)$. The speed term is used to estimate the convergence speed of an architecture. It is computed by summing the cross-entropy loss at every epoch divided by the time taken to train for that epoch over all the epochs.

Rather than training the original architectures for N epochs, they derive a reduced variant for each architecture that is close enough to the original architecture. This reduced architecture is then trained on a reduced dataset to obtain the training dynamics necessary for MOTE. The reduced dataset is created by first obtaining the logits of a pretrained VGG-16 model and then for each label, k-means and farthest point sampling are applied to yield the reduced dataset.
strength_weakness: strength_weakness: 1. What is the correlation between the accuracies of the (i) reduced architectures and the original architectures (ii) training original architectures on reduced dataset and training the original architecture on the entire dataset (iii) the reduced architectures trained on the reduced dataset and the original architectures. Table A3 shows the correlation between the reduced architecture trained for 4 epochs but that is not the actual setting. The correlation already drops from 34.5 to 8.1%. MOTE when applied on the original architectures and the original datasets already would introduce some errors in ranking the architectures. The reduced architecture and the reduced dataset setting would further cascade the errors as the reduced setting is not 100% correlated to the original setting. So how can this reduced setting be used to begin with? Please include the correlation results for all the search spaces and all the datasets.

2.  The authors should not be comparing their approach against zero-cost methods while MOTE considers training information.  Please evaluate the rest of the baselines also on the trained architectures for fair comparison. However, as mentioned above, it is not clear how the other methods would perform in the reduced setting owing to the concerns stated in 1.  Also, 7 seconds per architecture while the rest of the baselines take only 1 second on an average would become very expensive when the number of architectures in the search space is large. 

3. Please evaluate the efficacy of MOTE on other search spaces as well such as DARTS, ENAS [1],[2]. [3] demonstrated that training-free NAS methods don't generalize to tasks beyond image classification. Please evaluate on TransNASBench-101 [4] too similar to ZiCO.


[1] NAS-Bench-301and the case for surrogate benchmarks for neural architecture search, Siems et al.

[2] On Network Design Spaces for Visual Recognition, Radosavovic et al.

[3] https://iclr-blog-track.github.io/2022/03/25/zero-cost-proxies/

[4] TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search, Duan et al.
questions: questions: 1.  Also, how many architectures did you sample for each search space? Did you run all the other baselines on the same architectures, same data augmentations that MOTE was evaluated on or did you use the results reported in their corresponding papers? Also, if you are evaluating the top-k architectures for some baselines, please change the table to be : (i) evaluation free, top-k=5, top-k=10) for all the baselines and report the accuracies accordingly. Right now the table is inconsistent. 
2. How does a linear combination represent the loss landscape? Given a point $\theta_{B}$ in the loss landscape  and 2 other intermediate points $\theta_{C}$ and $\theta_{D}$, if we draw a line from init to B, it might not actually pass through intermediate points in the loss landscape. So how is it capturing the landscape? The main reason this might be working better than TSE is that it is assigning higher weight to the current weights as the epochs increase and lesser weight to the initial weight. How would it compare against TSE-E and TSE-EMA where the loss values of the initial few epochs are discounted. 
3. This paper used architecture reduction and dataset reduction to accelerate the training so that MOTE can be computed faster. Can you clarify if the k-means and the farthest point sampling is applied to datapoints corresponding to each label seperately to obtain a subset of them for that label? If not, how exactly is the clustering and the sampling done?
 Can you show how well the architectures the reduced dataset performs when compared to the original dataset by computing their correlations?  Why did you use k-means followed by FPS instead of other subset selection methods such as set-cover, facility location or coresets [5]? 
4. Can you please elaborate further what the meta architecture and the reduced architectures are comprised of? It is not clear to me.
5. MOTE uses the landscape term and the speed term. In Table1, the estimation free version of MOTE seems to discover architectures that train much faster than the others. However, once top-k are considered,  the compute cost of training the discovered architecture is not any lesser than those discovered by some of the other baselines. Figure 1 shows that the landscape term performs better than the speed term. Can you also do an ablation and show how the MOTE-landscape and MOTE-speedterm perform in figure 7 and table 1? 

[5] https://cords.readthedocs.io/en/latest/strategies/cords.selection_strategies.SL.html#module-cords.selectionstrategies.SL.submodularselectionstrategy
limitations: limitations: None
paper_summary: paper_summary: The paper introduces MOTE-NAS, a novel approach for efficient Neural Architecture Search (NAS). MOTE-NAS suggests a novel proxy utilizing both macro-level loss landscape smoothness and micro-level convergence speed to predict the performance. By utilizing reduced architectures (RA) and datasets (RD), MOTE-NAS achieves state-of-the-art accuracy on benchmarks such as CIFAR-10, CIFAR-100, and ImageNet-16-120 while significantly reducing computational costs.
strength_weakness: strength_weakness: The design of the skeleton for the reduced architecture seems to require human expert’s guide. It would be better if it were demonstrated that MOTE-NAS is robust across various RA configurations or if there were a general methodology for constructing RA.
questions: questions: Typos
Table 2, UTRE-NAS
limitations: limitations: The authors do mention some of the limitations in the paper.
paper_summary: paper_summary: The paper presents MOTE, a training-based estimate for Neural Network accuracy, as a proxy method to accelerate Neural Architecture Search. The intuition behind MOTE is the non-convexity and non-linearity in the training loss landscape. Consequently, the authors provide a model that characterizes the training process's non-convexity and convergence speed and consider both in a few-shot joint optimization fashion. The proposed model is applied on a reduced architecture and dataset and requires a few training iterations. The evaluation results are promising and show the superiority of MOTE compared to predictor-based and few-shot NAS methods.
strength_weakness: strength_weakness: - Zero-shot estimates like NTK struggle with high sensitivity to the weights initialization method. While the authors have raised this point, there's no detailed study on MOTE's sensitivity to weight initialization . Furthermore, the type of initialization adopted by MOTE is not mentioned in the paper.
- The non-convexity and non-linearity in DNNs is a core reason for the inefficiency of NTK-based methods. The authors should put more emphasis on this point, as it is a significant factor influencing the performance of MOTE. An ablation study on the impact of skip-connection and non-linear operations on the performance of MOTE can help draw more concrete conclusions about the robustness of the proposed approach when the DNN model is highly non-linear. 
- The RD transformation needs to be fully detailed. For CIFAR100, the authors employed a VGG model pretrained on ImageNet1K. However, it's not mentioned whether they perform a fine-tuning for CIFAR100 and the rationale behind the choice of the VGG model specifically or whether VGG should also be employed to perform an RD on any dataset.
- While the comparison in the evaluation section is more focused on predictor-based and few-shot NAS frameworks, it does not specifically compare against zero-shot approaches [1, 2], especially the latest work that considers the feature map correlation [3].
- The search spaces considered are mainly from the NAS-Bench line of work. MOTE should also be evaluated on supernets like P-DARTS and Transformer-based models. 
- The MOTE-NAS and MOTE-NAS-EF are built upon evolutionary algorithms. It's therefore unclear whether the superior performance comes from the evolutionary search or MOTE evaluation. The evolutionary search should be compared w/ and w/o MOTE and also against simple baselines (e.g., random search).
- The conclusion section should discuss MOTE's limitations in more details, especially for multi-objective NAS methods, where efficiency and accuracy need to be optimized in a joint fashion.

**References:**
- [1]: Lin, Ming, et al. "Zen-nas: A zero-shot nas for high-performance image recognition." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
- [2]: Bhardwaj, Kartikeya, et al. "ZiCo-BC: A Bias Corrected Zero-Shot NAS for Vision Tasks." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
- [3]: Jiang, Tangyu, Haodi Wang, and Rongfang Bie. "MeCo: zero-shot NAS with one data and single forward pass via minimum eigenvalue of correlation." Advances in Neural Information Processing Systems 36 (2024).
questions: questions: - To what extent is MOTE sensitive to weight initialization?
- What impact does the non-linearity in DNN architectures have on MOTE’s performances? 
- What is the exact process of the DR transformation? Can the same process be adapted to any type of dataset? And what’s the rationale behind choosing a VGG model?
- What if the search space is a supernet (e.g., P-DARTS) or Transformer-based (e.g., NASViT)? How generalizable is MOTE to these search spaces?
limitations: limitations: - Highly non-linear DNNs are more challenging for MOTE---Thus the need for an ablation study on non-linear operations.
- The limited scope of MOTE's evaluation on NAS search spaces—only on NAS-Bench-related search spaces.
- The RD transformation method is highly specific, hindering its applicability to other tasks/datasets.
- No detailed study on the impact of weights initlization.
paper_summary: paper_summary: This paper proposes Multi-Objective Training-based Estimate (MOTE) for efficient NAS, leveraging landscape view and convergence speed to estimate the performance of neural architectures. It also introduces two reduction strategies for speeding up MOTE generation. Compared to other training-free NAS methods, MOTE achieves better search performance on NasBench201.
strength_weakness: strength_weakness: - Code is not available.
- The experimental results on ImageNet are not very good.
- Regarding the title "Multi-Objective Training-based Estimate for Efficient NAS," on NasBench201, only accuracy is considered as a metric, missing other important metrics like the number of parameters, FLOPs, latency, etc.
- The reduction strategies lack novelty, as many low-fidelity estimation methods have been proposed, including early stopping[1], training with down-scaled models[1], and training on a subset of the data[2].



[1]: B. Zoph, V. Vasudevan, J. Shlens, and Q. Le. Learning transferable architectures for scalable image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

[2]: A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter. Fast bayesian optimization of machine learning hyperparameters on large datasets. In Artificial intelligence and statistics, pages 528–536. PMLR, 2017.
questions: questions: I would like to ask about the results on ImageNet in Table 2. OFA costs 50 days, including the training and search process. Do you consider the training cost of the searched architectures? I think 0.1 days is far from enough to well-train a model on ImageNet.
limitations: limitations: No additional significant limitations. Please refer to the weaknesses. It would be beneficial if the paper could add future research directions for this work and extend this method beyond image classification tasks.

Comments: title: Response to rebuttal
Comments Added After Reading Author Response
comment: Dear Reviewer XpJH , 

Please respond to authors and engage in discussion as soon as possible.
 
AC
We sincerely appreciate the effort you have put into reviewing our paper. We will thoroughly consider your valuable suggestions and carefully reflect on the limitations you mentioned, including computational costs and search space issues. Your insights will significantly contribute to our continued research in the field of NAS. Finally, we would like to express our gratitude once again for your efforts on our paper, and we are also very thankful for your decision to increase the score.
I thank the authors for their response. To begin with, while I understand the idea of the MOTE-NAS paper, i seem to have misunderstood the way the linear combination of \theta and \theta_{init} is computed to arrive at \theta_{g}. Thank you for clarifying and correcting my understanding regarding g and G. In that case, how often do you evaluate \theta_{g} during the training?
While I am familiar with the cell based search space, i now understood what you meant by the term meta-architecture. 

A3. I understand that there are various methods to reduce the dataset. I wanted to know how the proposed k-means method compares to other subset selection methods that I pointed out. I see that you compared RD against random sampling in the ablation study.

A5. Algorithms such as TE-NAS, NASWOT, Zico, Zero-cost proxies for lightweight NAS rely only on the inference of a few batches during the search phase and evaluate the best architecture found. Like I mentioned earlier, the architecture found by MOTE-NAS-EF is worse than TE-NAS in table 1. Zico is also not included in Table 1. 

While all the search spaces require the algorithm to search for a cell, some search spaces are more complicated than others. NASBench 101, 201 etc are all reduced search spaces when compared to the original DARTS search space. TransNASBench 101 is also a challenging search space. That was the reason, it is important to evaluate how well the proposed method performs on these search spaces

The authors do point out that the correlation of the landscape term and the speed term don't deteriorate as much as the accuracy term in the reduced setting, albeit on 1k architectures. It is important to see how well they can discern between the top few architectures that are close in performance. However, computing the landscape term and the speed term is still computationally expensive. 

I thank the authors for answering most of my questions and I would like to increase my score
Thank you very much for your detailed feedback and thoughtful review. We greatly appreciate the time and effort you have invested in our paper. Your positive assessment means a lot to us.
Thank you again for your response. Although the search cost of 50 GPU days for OFA in ZiCO does not make sense to me, your answer has partially addressed my concerns. I will keep a positive score for this work.
Thank you for your feedback. The discrepancy arises due to differences in the search space and other hyperparameter settings. The configuration in our Tab. 2 was based on the experimental setup from ZiCO [1] (refer to their Table 2). Some of the values marked with an asterisk in our Tab. 2 are experimental results that we have reproduced.

[1] Li, Guihong, et al. "Zico: Zero-shot nas via inverse coefficient of variation on gradients." arXiv preprint arXiv:2301.11300 (2023).
Thanks for your answers. According to the Table.1 in OFA paper [1], the search cost is 40 hours,  50 days in table 2 is not consistent with the [1]. 






[1]Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791, 2019.
Thank you for your feedback. NAS methods can generally be categorized into the following two types:

1. **Zero-shot**: These methods rely entirely on performance estimates for ranking and only select the top-1 candidate as final architecture. In other words, the search process for these NAS methods does not require any "training". This is why some methods (CARS-D, ZiCO, and Ours) in Tab. 2 exhibit such impressive search speeds.

2. **Multi-shot**: These methods do not solely depend on performance estimates for ranking. Instead, they involve training (or evaluating) several promising candidates during the search process and then selecting the one with the best actual performance as the final architecture. In other words, these methods require multiple or iterative "training" of several candidates to achieve a more stable final architecture, which explains why their cost appears much higher (OFA).

Therefore, the costs reported in Tab. 2 reflect the search time of each method, providing a fair comparison. The difference in costs arises because some NAS methods incorporate "training" during the search process while others do not.
Thank you for your response. If the searched architecture requires additional training, it may be unfair to compare the search time directly with the OFA in Table 2, as it includes both search and training costs.
Thank you for your kind feedback and for updating your score. We appreciate your suggestions and will make sure to incorporate the explanations into the revised version of our paper. We also plan to publicly share our codebase on GitHub.
Thank you for your reply. The experiments conducted on ImageNet only represent the search time and do not include the training time. Since the MOTE-NAS-EF (top-k=1) version we used in Tab.2 is evaluation-free, we specifically emphasized that this method requires only 0.1 GPU days for the search process. In fact, training this top one architecture found after the search still takes nearly a week on a single V100 GPU.
I appreciate the authors' detailed responses, which have addressed several of my concerns. I encourage the authors to incorporate these explanations into the revised version of the paper and to share their codebase as well. I updated my score accordingly.
Thank you for the further response. In your response to question A1, you mentioned that the training cost of the searched architecture has been considered. However, I am concerned that 0.1 GPU days may not be sufficient for training neural architectures to convergence on ImageNet. Could you please clarify if the 0.1 GPU days include the training time? Thank you.