Title: Per-Architecture Training-Free Metric Optimization for Neural Architecture Search
Venue: NeurIPS 2025
Accepted: yes
Keywords: Neural Architecture Search

Training-Free Metric

Surrogate Model

Bayesian Optimization

Evolutionary Algorithm
Abstract: Neural Architecture Search (NAS) aims to identify high-performance networks within a defined search space. Training-free metrics have been proposed to estimate network performance without actual training, reducing NAS deployment costs. However, individual training-free metrics often capture only partial architectural features, and their estimation capabilities are different in various tasks. Combining multiple training-free metrics has been explored to enhance scalability across tasks. Yet, these methods typically optimize global metric combinations over the entire search space, overlooking the varying sensitivities of different architectures to specific metrics, which may limit the final architectures' performance. To address these challenges, we propose the Per-Architecture Training-Free Metric Optimization NAS (PO-NAS) algorithm. This algorithm: (a) Integrates multiple training-free metrics as auxiliary scores, dynamically optimizing their combinations using limited real-time training data, without relying on benchmarks; (b) Individually optimizes metric combinations for each architecture; (c) Integrates an evolutionary algorithm that leverages efficient predictions from surrogate models, enhancing search efficiency in large search spaces. Notably, PO-NAS combines the efficiency of training-free search with the robust performance of training-based evaluations. Extensive experiments demonstrate the effectiveness of our approach. Our code has been made publicly available at https://github.com/LMZ-Zhuo/PO-NAS.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel method called PO-NAS, which integrates training-free metrics and individually optimizes metric combinations for each architecture. The approach significantly enhances search efficiency and performance by incorporating evolutionary algorithms.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:  \n1. The authors integrate multiple metrics and optimize their combinations through limited training to obtain relatively effective weighting schemes.  \n2. They employ evolutionary algorithms combined with trained metrics predictors to extensively explore the search space.  \n3. The superiority and scalability of PO-NAS are validated across multiple datasets.   \n  \nWeaknesses:  \n1. The authors fail to explain the specific meanings of each metric in Table 4, making certain details of the paper difficult to comprehend.   \n2. While the core contribution focuses on utilizing multiple metrics and searching for optimal weightings through training, there's no discussion about why these metrics make the architecture search effective. This fundamentally limits its extensibility. The authors should instead explore the interpretability of these metrics (similar to works like DARTS-IM [1]), which would be more meaningful for NAS development.  \n  \n[1].Zhang M, Huang W, Yang B. Interpreting operation selection in differentiable architecture search: A perspective from influence-directed explanations[J]. Advances in Neural Information Processing Systems, 2022, 35: 31902-31914.\n\nQUESTIONS:\n1. Do multiple runs with different seeds yield different architectures? Approximately how many attempts are needed to obtain the optimal architecture's best results?  \n2. Could the authors provide either intuitive explanations or rigorous theoretical proofs to demonstrate how these metrics contribute to the search process?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes that prior training-free NAS metrics capture only part of a network’s merit and often fail to generalize across different tasks. Simply combining multiple metrics into a single score with fixed weights also can not transfer well. The authors propose PO-NAS, a NAS algorithm that dynamically learns a tailored combination of training-free metrics for each candidate architecture with a weight vector. Then the score is used as a surrogate model to select the optimal architecture. PO-NAS proposes a four step NAS method to combine train-free and training NAS, comprising initialization, a pre-train stage, a Bayesian stage and an evolution stage. Experiments on multiple NAS benchmarks demonstrate that PO-NAS achieves state-of-the-art performance compared with conventional NAS and other hybrid NAS methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- The paper is generally well-written and organized. It clearly claims its motivation and core ideas.\n- The authors have included a detailed appendix with further experimental details and provided the source codes.\n- The experimental evaluation is comprehensive. The authors test PO-NAS on 20 different tasks across three NAS benchmarks, including a variety of domains (image classification, scene recognition, segmentation, etc. in TransNAS-Bench-101).\n- The proposed method yields state-of-the-art or near-optimal performance on multiple benchmarks.\n\n**Weaknesses**:\n- The proposed approach is quite elaborate, involving multiple components and phases: a graph neural network encoder, a metric predictor, per-architecture weight optimization via a surrogate model, Bayesian optimization guiding architecture selection, and an evolutionary algorithm with custom crossover/mutation. This complexity might make the method hard to understand and tune.\n- The complex pipeline proposed in this paper introduces a significant number of additional hyperparameters, but no discussion is provided on how sensitive the final outcome is to these hyperparameters.\n- Although the results are strong, in some cases the margins over prior art are not very large. The paper would be stronger if it provided a deeper analysis of current results.\n\nQUESTIONS:\n- Could you clarify how the per-architecture weight vector $w_A$ is actually optimized in practice? You mention employing Bayesian Optimization for the surrogate – does this mean a Gaussian Process is tuning the weights for each architecture sequentially? \n- PO-NAS relies on training architectures for a small number of epochs (10 on CIFAR, 3 on ImageNet) to get the true performance feedback used for weight updates. How well does this short-term performance correlate with the final performance of architectures? \n- Ablation on Metrics Used: You integrate six training-free metrics in the surrogate. Did you investigate how important each one is? For example, if you remove a metric or down-weight it, does performance drop noticeably? Conversely, if a new metric is added, is the framework robust to incorporate it? \n- Hyperparameter Sensitivity: The paper introduces multiple components to construct the NAS method, and it is necessary to clarify the sensitivity of the final performance to certain key hyperparameters. How sensitive are the results to the choice of $T_e$ and the population size or number of offspring?\n- When evaluating the time overhead, will the computational costs of introduced components—such as the architecture encoder and Bayesian surrogate model training—be accounted for?\n\nLIMITATIONS:\nThe limitations have been clarified in the weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper notes that individual training-free metrics capture partial architectural features (for e.g. Parameter Count capture just computation, not much about connectivity, etc.). They claim that  combining training-free metrics and adjusting to sensitivities of different architectures to specific metrics is needed. PO-NAS is proposed, combining multiple training-free metrics as scores, and combining them smartly. They further use an EA with predictions from surrogate model to search in large search spaces.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Good idea to combine multiple training-free metrics, train an arch-encoder to predict these metrics and use that for broader NAS. \n\nWeaknesses:\n- Combining such metrics has already been studied in MultiPredict, FLAN, TA-GATES. Comparing with such methods is important, as they report results on NB201 as well. \n- Comparing encoders such as Arch2Vec and CATE might be insightful, it may especially help at the attention-network where distinguishing architectures is important.\n- On macro-spaces, comparing with GENNAPE or relevant discussion is important. \n\n[CATE] Yan, Shen, et al. ‘CATE: Computation-Aware Neural Architecture Encoding with Transformers’. arXiv [Cs.LG], 2021, http://arxiv.org/abs/2102.07108. arXiv.\n\n[Arch2Vec] Yan, Shen, et al. ‘Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?’ NeurIPS, 2020.\n\n[MultiPredict] Akhauri, Yash, and Mohamed S. Abdelfattah. ‘Multi-Predict: Few Shot Predictors For Efficient Neural Architecture Search’. arXiv [Cs.LG], 2023, http://arxiv.org/abs/2306.02459. arXiv.\n\n[TAGATES] Ning, Xuefei, et al. ‘TAGATES: An Encoding Scheme for Neural Network Architectures’. Advances in Neural Information Processing Systems, edited by S. Koyejo et al., vol. 35, Curran Associates, Inc., 2022, pp. 32325–32339, https://proceedings.neurips.cc/paper_files/paper/2022/file/d0ac28b79816b51124fcc804b2496a36-Paper-Conference.pdf.\n\n[FLAN] Akhauri, Yash, and Mohamed S. Abdelfattah. ‘Encodings for Prediction-Based Neural Architecture Search’. arXiv [Cs.LG], 2024, http://arxiv.org/abs/2403.02484. arXiv.\n\n[GENNAPE] Mills, Keith G., et al. ‘GENNAPE: Towards Generalized Neural Architecture Performance Estimators’. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 8, Association for the Advancement of Artificial Intelligence (AAAI), June 2023, pp. 9190–9199, https://doi.org/10.1609/aaai.v37i8.26102.\n\nQUESTIONS:\nAside from evaluations missing relevant work from prior literature, I do not have major concerns, paper is well written, well motivated and does not raise any major red flags.\n\nLIMITATIONS:\nDiscussed in weaknesses above, just needs more evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
PO-NAS is a Neural Architecture Search (NAS) algorithm that improves the accuracy and efficiency of training-free NAS methods. Unlike previous approaches that apply a fixed combination of training-free metrics across the whole search space, PO-NAS dynamically optimizes metric combinations for each individual architecture using limited real training data. It integrates multiple metrics, avoids benchmark dependency, and incorporates an evolutionary search strategy guided by a surrogate model. The method effectively balances the speed of training-free NAS with the reliability of training-based evaluations, achieving strong performance across various tasks.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The proposed PO-NAS introduces a novel method that dynamically weights existing zero-cost estimates.
2. Through a well-crafted system design, PO-NAS significantly outperforms the performance of individual zero-cost estimates.
3. The paper is clearly written and easy to follow.

Minor Concerns:
1. The use of both ELU and GELU activation functions appears inconsistent. Is there a specific reason for this combination?
2. I am also curious whether different permutations or combinations of zero-cost estimates have any impact on performance.
3. It would be valuable to discuss the relationships between different zero-cost estimates, for example, whether SynFlow and SNIP might be complementary. Such analysis could further enhance the contribution of the work.

Comment:
Overall, this paper is well-written, with a clear motivation and strong experimental results. Discussing the potential relationships among different zero-cost estimates could further strengthen the contribution. So I recommend borderline acceptance.

QUESTIONS:
as my concerns

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03