Per-Architecture Training-Free Metric Optimization for Neural Architecture Search
Authors: Mingzhuo Lin
Jianping Luo
Venue: NeurIPS 2025 poster ()
Keywords: Neural Architecture Search
Training-Free Metric
Surrogate Model
Bayesian Optimization
Evolutionary Algorithm
Primary area: deep_learning

Abstract: Neural Architecture Search (NAS) aims to identify high-performance networks within a defined search space. Training-free metrics have been proposed to estimate network performance without actual training, reducing NAS deployment costs. However, individual training-free metrics often capture only partial architectural features, and their estimation capabilities are different in various tasks. Combining multiple training-free metrics has been explored to enhance scalability across tasks. Yet, these methods typically optimize global metric combinations over the entire search space, overlooking the varying sensitivities of different architectures to specific metrics, which may limit the final architectures' performance. To address these challenges, we propose the Per-Architecture Training-Free Metric Optimization NAS (PO-NAS) algorithm. This algorithm: (a) Integrates multiple training-free metrics as auxiliary scores, dynamically optimizing their combinations using limited real-time training data, without relying on benchmarks; (b) Individually optimizes metric combinations for each architecture; (c) Integrates an evolutionary algorithm that leverages efficient predictions from surrogate models, enhancing search efficiency in large search spaces. Notably, PO-NAS combines the efficiency of training-free search with the robust performance of training-based evaluations. Extensive experiments demonstrate the effectiveness of our approach. Our code has been made publicly available at https://github.com/LMZ-Zhuo/PO-NAS.

Per-Architecture Training-Free Metric Optimization for Neural Architecture Search
Mingzhuo Lin', Jianping Luo"
Guangdong Key Laboratory of Intelligent Information Processing, College of Electronic and Information Engineering, Shenzhen University, China
Abstract
Neural Architecture Search (NAS) aims to identify high-performance networks within defined search space. Training-free metrics have been proposed to estimate network performance without actual training, reducing NAS deployment costs. However, individual training-fii metries often capture only partial architectural features, and their estimation capabilities are different in various tasks. Combining multiple training-fiee free metrics has been explored to enhance scalability across tasks. Yet, these methods typically optimize global metric combinations over the entire search space, overlooking the varying sensitivities of different architectures to specific metrics, which may limit the final architectures' performance. To address these challenges, we propose the Per-Architecture Training- Metric Optimiza- tion NAS (PO-NAS) algorithm. This algorithm: (a) Integrates multiple training- free metrics as auxiliary scores, dynamically optimizing their combinations using limited real-time training data, without relying on benchmarks; (b) Individually optimizes metric combinations for each architecture; (c) Integrates an evolutionary algorithm that leverages efficient predictions from the surrogate model, enhancing search efficiency in large search spaces. Notably, PO-NAS combines the efficiency of training frrennggg-eeeeeeee search with the robust performance training based evaluations. Extensive experiments demonstrate the effectiveness of our approach. Our code has been made publicly available at httttss: //gi thub. com LMZ- 2huo/PO-NAS
Introduction
Neural Architecture Search (NAS) has emerged as a method automate architecture engincering. Currently, NAS methods have surpassed many manually designed architectures in various tasks, such as image classification (I), object detection (2), and semantic segmentation
B.. While many training-based based NAS algorithms have achieved state-of-the-ar (SOTA) performance across various tasks, their search costs are often prohibitively high in resource-constrainet scenarios, primarily due the necessity of training deep neural networks during the search process. To address this, numerous training-free metrics have been developed to evaluate candidate architectures' performance 4 5 7 10), eliminating the need for actual training. For instance, SWAP-NAS (10) achieves SOTA performance on the ImageNet dataset within the DARTS (II) search space in just minutes of search time. These metrics are typically computed through a single forward and backward pass using a small batch of data, rendering their computational overhead negligible compared to traditional NAS methods. The efficacy training-fres metrics typically assessed by evaluating the correlation between the rankings they produce and the actual performance rankings of architectures 112. However, significant limitation of these metrics lies in their transferability. Studies have demonstrated considerable variability in the consistency between scores derived from training-free metrics and actual performance rankings across different tasks 55 14 15).
To address the limited transferability of individual training-free metrics in diverse tasks, researchers have explored combining multiple such metrics for architecture evaluation. Depending on the methods
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

of combination, several studies have emerged. Such as EZNAS (16), Auto-Prox (17), Auto-GAS (18)
and Auto-DAS (T8), combine various symbolic regression operators optimize training-free free metrics. These methods optimize the metrics by aligning the combined metrics scores with the performance
rankings across multiple task benchmarks. However, these approaches typically rely on a substantial amount of training data. Moreover, the optimized training-free -ree metrics are often tailored specifically for the tasks in the training benchmarks, thereby limiting their applicability to a broader range of search tasks. Some approaches aim to integrate training-free metrics with training- -baedd search methods. OMNI (19), ProxyBO (20). HNAS (4), and RoBoT (T5) utilize training-five metrics as auxiliary information rather than sole evaluation criteria. During the training process, these methods evaluate the significance of training-free free metrics based on actual performance and refine these metrics through surrogate models. However, these methods typically optimize the metric weights over the entire search space, neglecting the varying sensitivities of different architectures to each metric. As a result, the correlation of training-free metrics varies significantly among architectures with different features, even within the same search space. Our experiments on NAS-Bench-200 (21) across various architecture sets confirm this point, with detailed results in Appendix C.T
To address the aforementioned challenges, we propose PO-NAS. PO-NAS assigns different weights to fraee metrics for each architecture, uses the weighted sum of metrics for scoring, and employs this score as a surrogate model to select the optimal architecture. It also dynamically adjusts the weight allocation based on performance feedback, which enhances PO-NAS's transferability across diverse unknown tasks. Additionally, PO-NAS employs an evolutionary algorithm that efficiently utilizes the surrogate model's ' evaluation capabilities to enhance the exploration of the search space. Our main contributions are as follows:
We propose surrogate modeling method, which assigns training-frre free metric weights to each architecture, uses the weighted sum of these metrics as a surrogate model, and dynamically adjusts the weights according to performance feedback.
We propose an evolutionary algorithm that employs the surrogate model to efficiently explore the search space. With particular parental selection mechanism along with crossover and mutation methods, enhances the balance between exploration and exploitation.
We propose PO-NAS, a NAS method that effectively combines the efficiency of training-free NAS with the robust performance of maainng--bsee NAS. Extensive experiments across various datasets demonstrate PO-NAS's superior performance and scalability.
2 Related Work
Training-free NAS Recent studies have proposed numerous training-free metrics for evaluating the performance of different network architectures. Model-dependent training-free metrics are designed for specific types of network architectures. For example, Zen-NAS (22) uses the Gaussian complexity of linear classifiers within each linear region to measure the distribution of linear regions. On the other hand, inneedennee metries exhibit higher transferability across different neural network structures (23). For example, NASI (22) leverages the ability of Neural Tangent Kernel (NTK) (24) to characterize architecture performance at initialization to develop new training-free metries. However, many metrics still lack scalability across various architectures and tasks. Recent metrics such as Zico (8) SWAP (10), and AZ-NAS (9) have demonstrated excellent performance across various search tasks, but they still struggle to consistently exhibit robust evaluation performance across multiple tasks. Recent research indicates that training-free metrics are generally complementary to one another, an appropriate combination of these metrics can enhance rank correlation across multiple tasks (I3).
Hybrid NAS Some studies have attempted to integrate training- free metrics with rriniig-baseed neural architecture search methods to better balance efficiency and performance. These approaches use the ground-truth performance on the target task a supervisory signal. For example, OMNI (19) accelerates architecture search by combining training-free metrics with Bayesian Optimization, while ProxyBO 200) incorporates these metrics as input to the surrogate model to improve search efficiency. HNAS (14) introduces techniques such as gradient normalization and dynamic weight adjustment to improve the stability and reliability of gradient-based metrics. However, this approach remains limited by the intrinsic constraints of single metries. RoBoT (15) employs weighted linear combination to ensemble multiple training-fres metrics and uses Bayesian Optimization to dynamically adjust their weights based on real-time training feedback. However, these methods
2

---

typically optimize the metric weights over the entire search space, neglecting the varying sensitivities of different architectures each metric. They often settle for suboptimal solutions and struggle to identify the globally optimal architecture. In contrast to these methodologies, PO-NAS analyzes the sensitivity of individual architectures to various training-free free metries, assigning distinct metric weights for each architecture accordingly, which enhances PO-NAS's evaluation capability.
Initialization
Compose
Best
Train
NYw
Archilectures
Computed
N, Architectures
Trined
Pretrain Stage
I Trained Set
Architecture Fnceder E
Metrics Prodicoor P.
Intixalicaion
Initial
Supervisol
Atientiom Network
Archriccture Embeediing
Predicted Metnos
Computed Metric
Bayesian Stage
Model Model M
Metries Embedding
Attention Network
Best
Train
Score
Triined
Initial Pool
Froeen Encoder
Supervised
Evolation
Predieted Metrics
S0 Excellent
Parent
Orlipring
2500 Ollspring
Sunogzic Model
Campute
Inital Pool
Menics P.
M
Motation
Figure I: Overview of PO-NAS. First, we randomly initialize a large number of architectures as the initial pool and select several architectures for training form the trained Pre-training stage, initialize the attention network and optimize the architecture encoder as well as the metrics predictor with training-frree metries to generate embeddings that distinguish between different architectures. Bayesian stage, we use the surrogate model to search for the optimal architecture, and optimize the network which assigns weights metries for each architecture based performance feedback. We combine the surrogate model with the evolutionary algorithm to extensively explore the search space.
Our Methodology
3.1 Algorithm Principle
An appropriate combination of training-free metrics can enhance rank correlation across multiple tasks (13). In this section, we explain how PO-NAS leverages raining-free metrics to select the best architecture while using the true performance feedback to optimize the metric weights. Let the search space be defined as a set containing N candidate architectures, i.e., = [A The available information includes: (a) the architecture graph ((A): (b) the training ffre metries Z(.4) and (c) the true performance f ((A). When true performance data is limited, it is essential to fully exploit the available information. ((AA) can be obtained
prior: ((A) can computed at low cost; whereas f(A) is the most valuable but also the most computationally expensive, making it difficult to serve as large-scale scale supervision. Thus, we employ 2(A) and ((A) to train the architecture encoder, enabling it to generate embeddings that distinguish architectures with different characteristics. This aspect will be discussed in detail Section 3.3 We assign different metric weights for each architecture based on performance feedback ((A). Given the constraints of limited search budgets T, adopt weighted linear combination as the fusion strategy for the metrics. This formulation not only more robust but also more efficient to optimize under limited supervision. Let (AA), (A), (A)

---

be K complementary metrics. We define the scoring function for single architecture A as:
S(A:wa)
WAI (A(
(1)
where WAA E A is the weight vector corresponding to the architecture (satisfying Σ | = 1), and (AA) denotes the normalized value of the i-th metric. To maximize the rank correlation between the true performance f((A) and the combined score S(A; (A) over a validation set, we formulate the following weight optimization objective:
W
A
(++1) arg max
(1S(A)281) AEA,
.L((A))
(2)
where 7-)) denotes the Kendall rank correlation coefficient, and A₃ is the set candidate architectures selected in iteration f. With the optimized weights w evaluate both the entire set of candidates A and the architectures discovered through the evolutionary algorithm.
3.2 Overall Framework
Algorithm 1
Pseudo code for PO-NAS
Require:
True training performance function f, true training- free metrics function 2, the number of the candidate architecture Nivis the number the initial trained architecture Nr. the pre-training epochs T,. the BO search budget T, and the evolution iteration To
1: // Initialization (Appendix [B.3)
2: Candidate architecture
Ao Generate randomly Na architectures
3: Candidate metrics set Zag <- Compute the real training-free metrics 2 2(Aa)
4: Trained 2o Select N₂ architectures from Ag and obtain (Binis ff (Ainss) by training Aint
5: // Pre-training stage (Section 3.3)
6: for step I.... do
7: Train the architecture encoder and the metrics predictor using Zo and Ag
8: Optimize the architecture encoder with the node feature masking reconstruction task.
9: end for
10: Obtain architecture encoder E and metrics predictor Pa
11: // BO search stage (Section 3.4)
for step 1. 1,..,, T, do
13: Update the surrogate model M using Q.-1
14: if >> T. then
15:
// Evolution stage (Section 3.5
16:
Choose the excellent architecture set Aee by MEE(A-1)), 2,-1) from 44-1
17:
Select excellent architecture combinations Cex by Spair (AppendAp (A.4) from Aez
18:
Obtain the offspring set Ach. by applying crossover and mutation operations to Cer
19:
Choose excellent offspring architectures Actus by ((E(Ae)), Pa (A(tt) from Ach Compute the real training-free metrics 2( ((Brchue)
20:
21:
Obtain A, A.-
(Arhher } and Z, 24-1 (Z(Archoa))
22:
end
Obtain the best architecture Abest by M (E(A).. Ze) from ha while Abest E Qr-1 do
24:
25:
Select next best architecture Aneat Abest
26:
end while
Obtain the true performance f(Abest) by training Abest
28: Obtain Q = Q.-1 U ([Abest: (Abest)))
end for
30:
return the best architecture Abrst arg A (())
Bayesian Optimization 255 can effectively refine the surrogate model within limited training budget, making highly suitable for our algorithm. Thus, we employ Bayesian Optimization (BO) to optimize surrogate model. The overall framework of PO-NAS is presented in Figure]

---

Algorithm I describes our proposed method. (a) Initialization (lines 1-4 of Algorithm[] ::: We randomly initialize large number of architectures as the candidate set and select several architectures for training based on the average of all training-free metrics to form the initial trained set ett(( (Implementation details can be found Appendix [B:3]: (b) Pre-training stage (lines 5-10 of Algorithm [T): We optimize the architecture encoder and the metric predictor using the initial architecture set and their training- free metries (Section 3.3): (c) BO search stage (lines 1 1129 of Algorithm D: At each iteration, we first update our surrogate model with the actual performance data of the trained architectures. Once the surrogate model has been trained to a certain loss threshold, we use it to predict the performance score of each architecture, rank the architecture pool based on these scores, select the highest architecture for actual training, and then update the pool of trained architectures (Section 3.4); (d) Evolution stage (lines 15-21 of Algorithm []: We select advantageous offssring based on predicted scores, calculate the operation costs between them, and choose the best combinations for crossover and mutation obtain the offspring set (lines 16-18 of Algorithm D. Additionally, we use the surrogate model to forecast the scores of the offspring and select the top-ranked ones to calculate their training-free frrenngggeeeeaeee metrics and update the initial architecture pool (lines 19-21 of Algorithm (Section 335)
After several cycles of iteration, we identify the optimal architecture.
Pretrain Stage
Rayesian Stuge
Nomal Cell
Rabor Call
Mariss
Archiecaure Eubealing
Rmbedding
GAT
Lincar
Elu
Bla
Libear
Linear
Layeritorn
Linear
GAT
Linear
Geta
Profict Menics
Linear
Getu
Gella
Layeritorm
Getu
38a
Linear
Mron Food
Wean Pool
Multihead Attention
LayerNorn
Linar
Recon Architectane
Geta
Concat
Linear
Arthaeeture Embssiding
Linear
Sigmood
Weight
Figure 2: Model design of PO-NAS. Pre-training stage includes an architecture encoder, well as metric predictor and architecture reconstruction heads for supervised training. Bayesian stage includes an attention network, which is utilized to generate metric weights by integrating architectural embeddings with metric embeddings.
3.3 Architecture Encoder
In this section, we will provide a detailed description of the Architecture Encoder component. The purpose of this component is to leam the structural characteristics of architectures through training-free metrics, thereby generating embeddings that can distinguish between different types of architectures. We employ a multi-head graph attention mechanism (GAT) (26) to perform hierarchical feature learning on the neural architecture. The model design of the architecture encoder is shown in Figure 2 We use two-layer GAT encode graph features. We employ the Exponential Linear Unit (ELU) post the GAT, driven by the nature of node connectivity in the topology. GAT processes connections and operations between nodes, some of which may negatively correlate with predictions. ELU's smooth negative region (a(e--)) -1)) retains these critical negative features. The final graph-level embedding he obtained through global average pooling. To effectively learn the expansion structure of architectures, inspired by GraphMAE (27). we design node feature masking reconstruction task to enhance the robustness of architectural features. Specifically, we randomly mask node features Ie with a probability Pmmsk, replace them with learnable mask tokens m. and reconstruct the original features through a linear decoder. The reconstruction loss is defined as follows:
Lream
Σ VVV...
1 1u
(3)
1/m
5

---

Where Vn represents the set of masked nodes, and Fy is the reconstruction result. We define the metric prediction loss follows:
metricc
1 - (,99)
Here, (hg) represents the i-th metric prediction value, and (g) is the i-th training-freee metric value. During the pre-training phase, the encoder is optimized through the self-supervised recon- struction loss and the training-free metric prediction loss. After the pre-training phase, we freeze the architecture encoder and metric predictor, providing the graph-level embedding hg and the metric prediction values P (hg) for subsequent search stages.
3.4 Surrogate Model
Model Design We design a surrogate model based multi-head cross-attention network (28). aimed at assigning training-fis free metric weights for each architecture and generating architecture scores. The design of the attention network is shown in Figure 2 We utilize the multi-heat cross-attention mechanism to establish associations between architectural embeddings and metric embeddings. Based on this, the network generates metric weights for each architecture, enabling the surrogate model to generate scores for predicting the performance each architecture. We introduce dynamic gating module to regulate the integration ratio between attention outputs and original metric embeddings. To improve cross cross-modal modal interaction efficiency, each metric embedding and architectural embedding equipped with a Linear-GELU-Liness trio, mapping the original embeddings into a higher-dimennionall space. We employ GELU to enhance the representation of high-level topological features. The linearly transformed features require refined nonlinear processing. GELU's probabilistic gating (x ((x)) captures complex node interactions, while its smoothness aids hierarchical feature propagation. Each metric is transformed into a dense vector through an embedding layer, with residual connections incorporated to retain original features and enhance representational capacity. A three-layer MI.P network is employed generate metric weights.
Normalize and Loss Function To fairly evaluate the sensitivity different architectures to various metrics, we normalize the training-free metries and their weights. The specific implementation details can be found Appendix B.3 After obtaining the normalized metries 2, and the weights w; we decompose the normalized weights w into positive activation weights ú = ReLU(()) and negative activation i (eLU((-w) and construct the scoring function follows:
$
Σ
w 2, + wZ (1 -11-2,) 2,))
(5)
Most studies evaluate the performance of training-free frree metrics by calculating the correlation coeffi- cient between the ranking of training-free metrics and the actual performance ranking of architectures. However, in this algorithm, due to the limited availability of real performance data, calculating the correlation coefficient not feasible. Therefore, we use the alignment loss between the distribution of differences in architecture scores and actual performance the main supervisory signal for the surrogate model. Due to the model score being ultimately derived from a weighted sum metrics, challenging fine-tune the weights to ensure accurate ranking among architectures with closely matched performance. Hence, we introduce a difference threshold Tu to reduce the impact of archi- tectures with closely similar performance for correctly ranked architecture combinations. For the trained set O, of iteration I, where the true performance of the architecture has been obtained, the alignment loss is defined as follows:
the optimization difficulty. We calculate the loss only
Spred = S... SAp, Sprue = f(A:) - s(A,) (Vi,j E C(Q. 2))
1, if |Stool < Tin
M
1. if 6pred Stne
0, otherwise
Opeeal M [1opredl) a ((0peea))
M [[omme]] |imeel
Lalign E
(6)
o a(dmeel)

---

B(-) denotes the expectation, a(-) denotes the standard deviation and @ denotes the element multiplication. Additionally. we incorporate the correlation between scores and performance L as an auxiliary loss and add a direction alignment penalty Lair to ensure overall ranking accuracy:
Loor - 1- - ((5A) AEE -I(AA)
where p(.) denotes the Pearson correlation coefficient.
Lir = E [[ReLU(-Oppee Swee)
(8)
At the beginning of each iteration, the surrogate model is updated through the alignment loss, the correlation loss, and the direction alignment loss.
3.5 Evolutionary Algorithm
We develop a novel evolutionary algorithm that employs the surrogate model to efficiently ex- plore the search space. Detailed methods are provided in Appendix A This algorithm employs matrix encoding to map topological structures into operable matrix forms. To effectively mea- sure the intrinsic differences between architectures, we compute the shortest operation path and operation cost between two parent architectures and propose the shortest operation path crossover. We enumerate the neighborhood space of elite archi- tectures using neighborhood mutation strategy and adopt an adaptive selection mechanism to balance ex- ploration and exploitation. As shown in Figure in the early stages of search, it prioritizes offssring com- binations with higher computational costs enhance global search capabilities by increasing architectural diversity. In the later stages, it shifts to itness-based selection, focusing on high-performing individuals to deepen local optimization. The algorithm leverages the efficient evaluation speed of the surrogate model described in Section 3.4 to efficiently explore the search space. Through the evolutionary algorithm, as illustrated in Figure []] we are able focus on a large number of samples in the search space with very limited computational cost, thereby identifying optimal solutions among the search space.
Epoch-11
Epoch=23
Figure 3: These figures represent two- dimensional (2-D) embeddings of high- dimensional search space. Red dots: superior architectures; Blue dots: candidate architec- tures; Orange dots: exploration areas.
4 Experiments
4.1 PO-NAS on NAS Benchmark
To validate the robustness and exceptional predictive performance of PO-NAS across various tasks, we conduct comprehensive experiments on multiple popular NAS benchmarks. These benchmarks include 20 distinct training tasks: NAS Bench-201 (2I). TransNAS CaansNAS--encc----- 101 (34), and DARTS (II). PO-NAS utilizes 6 metrics outlined by (5): grad norm, snip, grasp, fisher, synflow, and jacob cov. Implementation details and more empirical results can be found in Appendix B and Appendix
Table T Table 2 Table 3 and Table present the performance results of PO-NAS NAS-Bench-201, In the large-scale DARTS search space, PO-NAS demonstrates
DARTS and SannsNAS-Bench--e remarkable performance. This attributed to integration of the advantages of both training-free and training-based NAS methods. It optimizes training-free free metrics based on training performance without relying on training benchmarks and leverages the efficient computational speed of training- free metrics. By utilizing training- free metrics and the layer-wise evaluation of the surrogate model, PO-NAS only requires training small number of architectures (25), yet can focus on a large number of architectures (approximately 50000) during the search phase. With a balanced exploration- exploitation search strategy, PO-NAS can rapidly and effectively explore the large-scale search space relying solely on limited training. Moreover, the results on NAS-Bench-200 and MaansNASSBeneede 101 indicate that PO-NAS significantly enhances the evaluation capability of the original metrics. By optimizing the corresponding metric combinations for different types of architectures, PO-NAS gains

---

Table Comparison of various NAS algorithms NAS-Bench-201. Results are reported with the mean + standard deviation of 10 runs. "Training-free (Avg./Best)" represent the average value of metrics and the best individual metrics. The best results are bold, and the second best are underlined.
Test Accuracy (%)
Algorithm
Cost (GPU Scc.)
Method
C10
C100
IN-16
ResNet 29
93.97
70.86
43.63
manual
REA
93.92+0.30
71.84+0.99
45.1540.80
12000
cvolution
RS (w/o sharing)*
93.70=0.36
71.04:.007
44.57+1.25
12000
random
REINFORCE'
93.85+0.37
71.71+1.09
45.24+1.18
12000
RL
ВOHB
93.61#0.52
70.85
44.42+1.49
12000
BO+bandit
DARTS (2nd) (1I
54,30=0.00 00
15.61+0.00
16.32+0.00
43277
gradient
DrNAS 30
93.98+0.58
72.31+1.70
44.02+3.24
14887
gradient
Sharply-NAS 31
94.05+0.19
23.15+0.26
46. 46.25+0.25
14762
gradient
3-DARTS 32
94.00+0.22
72.91+0.43
46.20#0.38
3280
gradient
TE-NAS (4
93.90=0.47
71.24+0.56
42.38+0.46
1558
training-free
NAST 3R
93.55=0.10
71.20#0.14
14.84+1.41
120
training-free
GradSign
93.3140.47
7..3
42.42+2.81
1824
training-free
ZiCo 8
93,5010.18
70.6240.26
42.0440.82
372
training-free
AZ-NAS
93.53
70.75+0.48
45.43+0.29
43
training- faree
HNAS 14
94.0410.21
71.75+1.04
45.91
3010
hybrid
RoBoT 15
94.36+0.00
73.51+0.00
46.34+0.00
3051
hybrid
Training-fir free (Avg.)
92.00
70.66
45.73
training- free training-fre free
Training-fre free (Best)
93.76
71.11
42.60
PO-NAS
94.
14:0.22
73.51+0.00
46.71+0.12
3162
hybrid
Optimal
94.37
73.51
47.31
Reepoted by 21
Table 2: Performance comparison among various NAS algorithms on ImageNet on DARTS search space. The search costs are evaluated on an Nvidia 1080Ti. The best results are bold, and the second best are underlined.
Test Error (%)
Params (M)
Search Cost
Algorithm
(M)
(GPU Days)
Top-1
Top-5
Inception-vl 35
30.1
10.1
6.6
1448
MobileNet (36
29.4
10.5
4.2
569
AmoebaNet-A
25.5
8.0
5.1
555
3150
PNAS (37)
25.8
8.1
5.
588
225
MnasNet-92 (38
25.2
8.0
4.4
388
DARTS |I]
26.7
8.7
4.7
574
4.0
ProxylessNAS (39)
24.9
7.5
7.1
465
8.3
SDARTS-ADV (40
25.2
7.8
5.4
594
1.3
TE-NAS
24.5
7.5
5.4
0.17
NASI-ADA 33
25.0
7.8
4.9
559
0.01
QE-NAS 41
25.5
3.2
0.02
SWAP-NAS 110
24.0
7.6
5.8
0.006
HNAS (4
24.3
7.4
5.1
575
0.5
RoBoT (5
24.1
7.3
5.0
556
0.6
PO-NAS
23.9
7.1
6.3
667
0.64
8

---

an advantage in exploring the best architecture. However, this optimization also correspondingly in- creases the difficulty of surrogate model optimization, thereby affecting the stability of its performance to some extent. Nevertheless, PO-NAS still achieves competitive results.
Table 3:
Performance comparison among various NAS algorithms on CIFAR-10/100 on DARTS search space. The performance of the final architectures selected by PO-NAS is reported with the mean + standard deviation of 5 runs. The search costs are evaluated on an Nvidia 1080Ti. The best results are bold, and the second best are underlined.
Test Error (%)
Params (M)
Algorithm
Search Cost
Search Method
C10
C100
C10
C100
(GPU Hours)
DenseNet-BC 42
3.46*
17.18*
25.6
25.6
manual
NASNet-A (43
2.65
3.3
48000
RL
AmoebaNet--A (1)
3.34+0.06
18.93
3.2
3.1
75600
evolution
PNAS (37
3.41
19.53*
3.2
3.2
5400
SMBO
ENAS 44
2.89
19.43*
4.6
4.6
12
RI.
NAONet (45)
3.53
3.1
9.6
NAO
DARTS (2nd)
11
2.76+0.09
17.54+
3.3
3.4
24
gradient
GDAS (46
2.93
18.38
3.4
3.4
7.2
gradient
NASP 47
2.83+0.09
3.3
2.4
gradient
DARTS- (avg) (11)
2.59+0.08
17.51
3.5
3.3
9.6
gradient
SNAS 48
2.85+0.02
20.09
2.8
2.8
36
gradient
SETN 49
2.69
17.25
3.4
3.4
43.2
gradient
SDARTS-ADV (40)
2.61+0.02
3.3
31.2
gradient
R-DARTS (L2) (50)
2.95+0.21
18.01+0.26
38.4
gradient
TE-NAS 4
2.83+0.06
17.42+0.56
3.8
3.9
1.2
training-free
NASI-ADA 33
2.90+0.13
16.84+0.40
3.7
3.8
0.24
training-free
HNAS (14)
2.62+0.04
16.29:0.14
3.4
3.8
2.6
hybrid
RoBoT (5)
2.60+0.03
16.52+0.10 10
3.3
3.8
3.5
hybrid
PO-NAS
2.5240.03
16.35+0.12
3.8
4.2
3.9
hybrid
4.2
Ablation Studies
We conduct a series of ablation studies on PO-NAS investigate the impact of different components and parameter settings. These factors include ensemble method, pre-training loss, difference threshold Ta and loss threshold, the number of combined training-fire free metrics and evolutionary algorithm. Implementation details and more detailed experimental results can be found in Appendix D
Layout
Scene
Object
Layost
Ca)
Namber Scnchal Axdiedans
(bi
Figure 4: Ablation study of the surrogate model in MaaesNAS-Beechhh 101 regarding the number of searched architectures. All methods are reported with the mean and standard error of 10 independent searches. Implementation details and more empirical results are provided in Appendix D.1
Ablation Studies on Surrogate Model investigate the impact of the surrogate model and its various components the performance of PO-NAS, we conduct series of ablation studies on PO- NAS in TransNAS-Bench-IDe Bench-101. These experiments focus on three main aspects: (a) Ablation studies of global optimization versus per-architecture optimization (GP, w/o encoder); (b) The influence

---

Table 4: Performance comparison of NAS algorithms in TransNAS-Bench--Un All methods search for 100 architectures. The results of RoBoT, HNAS and PO-NAS are reported with the mean standard deviation of 10 runs, while 50 independent searches for REA, RS, and REINFORCE. "Training-free (Avg./eest)" represent the average value of metrics and the best individual metrics.
Space
Algorithm
Accuracy (5))
Loss (x10 2)
mloU (98
SSIM (x10
Scene
Object
Jigsuw
Layout
Segment.
Normal
Autoenco.
REA RS (wlo sharing)
54,63+0. 18
44. 44.8820.31
94.7320. 14
-62.02+0.59
94, 94.6620.02
36.7600.33
56.0020.71
54.53+0.19
44.76=0.45
94.61=0.29
02.1790.98
94.5240.03
56.45+0.25
55.2740 90
REINFORCE
$4.49=0.15 19
44 14.6420.38
94.6920. 16
-61.61#0.91
94,53+0.04
51.0640.33
55.47=0.72
HNAS
54,29+0.09
08000000
94.56=0.21
-64.83+ .69
94.5740.00 00
56, 88.88400000 00
48, 48.66440.00
RoBoT
$4.87+0.00
45.59:0.00
94.82=0.06
-61.1620.86
94.5840.00
57.4420.34
55.42+1.05
Micro
Training-fice (Avg.)
40.85
83,49
-73.79
94,43
53,42
35.56
Training-free (Best)
53.72
41.78
91.08
-70.25
94.53
55,22
41.77
PO-NAS
54.9020.03
45.5920.00
94,9120.09
-61.5520.75
94.60z00001
57.08a0.14
55.6240.41
Optimal
54.94
45.59
95.37
-60.10
94.61
58.73
57.72
REA
56.65+0.31
46.87:0.33
96.75=0.08 08
-60.38+1.12
94. 80+0.03
60.57=0.56
71.39+2.76
RS (wlo sharing)
56 655+0.225
46.60+0.36
96.72=0.24
1.6
76+0.00
60.61+0.51
71.07+2.75
REINFORCE
0.30
60036+1.1
94.7840.04
60.34:0.52
69.21+2.55
HNNS
55.03a0.000
45.00:20.00
96.28a0. 18
-61.40a0.1 1
94.7920.00
59.27a00.0
$7.59:0.000
RoBoT
$7.3520.13
46.9420.09
96.92:0.02
38.8810.70
94. 85.0500.0
61.6620.00
73.53:0.06
Macro
Training free (Avg.)
54.50
43,93
95,20
-6.006
94.54
60.93
64.78
Training (Best)
-63.41
94.70
PO-NAS
$7.41=0.00
47 47.05
96. 96.97:0.04
-58.44.0.22
94.86=00.00
64.35+0.00
74.1920.23
Optimal
57.41
47.42
97.02
-58.22
94.86
64.35
74.88
of different normalization methods (w/o Negative, w/o Normal, Rank); (c) Ablation studies of the surrogate model itself: Eliminate the surrogate model and directly use the average value of all metries (Avg) and the best individual metrics (Best) for prediction. "OPTIMAL" denotes the performance of the optimal architecture on the entire dataset. Partial experimental results are presented in Figure which fully demonstrate the effectiveness of our surrogate model and its various components.
Ablation Studies on Evolutionary Algorithm We analyze the impact on PO-NAS performance from: (a) The evolutionary algorithm itself (w/o Evolution, REA); (b) The evolutionary algorithm components (w/o N, w/o Mutation, w/o Crossover); Figure shows partial experimental results, indicating our evolutionary algorithm's robust and excellent search capability across various initial architecture counts. Implementation details and more empirical results are provided in Appendix D.2
100
1000
10000
Tesl Aesunay
Test decorecy
50
15
101
25 Sumber Soarched Archiistures
+ PO-NAS -a- wio Evolution - wio N- wio Mulation - win Croeuover
REA
Figure 5: Ablation study of the evolutionary algorithm DARTS regarding the number of searched architectures. All methods are reported with the mean and standard error of 10 independent searches.
Conclusion and Future Work
We propose PO-NAS which: (a) Employs surrogate model to assign training- metric weights for each architecture and generate architecture scores, effectively enhancing the capability to select the best architecture using training-fres free metrics; (b) Employs a novel evolutionary algorithm that integrates the surrogate model for rapid architecture evaluation to efficiently explore large-scale search spaces. Extensive experiments have demonstrated PO-NAS's superior performance and scalability. To enhance model stability, we plan to cluster architectures and optimize metrics for each cluster rather than for individual architectures in the future.
10

---

Acknowledgement
This work was supported by the National Natural Science Foundation of China under Grant 62176161, and the Scientific Research and Development Foundations of Shenzhen under Grant ICYJ202208181000 1810000501 and 20200813144831100
References
[1] Real, E.., A. Aggarwal, Huang, Regularized evolution for image classifier architecture search, In Proceedings of the uaai conference on artificial intelligence, vol. 33, pages 4780-4789. 2019.
[2] Zela, A., A. Klein, Falkner, al. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint 1rKiv:1807.0668 2018.
[3] Chen, L.-C., M. Collins, Zhu, et al. Searching for efficient multi-scale architectures for dense image prediction. Advances in neural information processing systems, 31, 2018.
[4] Chen, Gong, Z. Wang, Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. In International Conference on Learning Representations (ICLR). 2021.
[5] Abdelfattah, M. S., Mehrotra, Dudziak, Zero-cost proxies lightweight [nas). International Conference on Learning Representations (ICLR). 2021.
Lin, M., Wang, Z. Sun, nen-nas: A zero-shot nas for high-porformance image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 347-356. 2021.
Zhang, 7., 7. Jia. Gradsign: Model performance inference with theoretical insights. In International Conference on Learning Representations (ICLR). 2022.
Li, G., Y. Yang, K. BBhardwaj, al. Zico: Zero-shot via inverse coefficient of variation gradients. In ICLR. 2023.
Lee, J., B. Ham. Az-nas: Assembling zero-cost proxies for network architecture search. In 2024 JEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 5893-5903, IEEE Computer Society, 2024.
[10] Peng, Y. A. Song, H. M. Fayek, et al. Swap-nas: Sample-wis wise activation patterns for ultra-fast nas. In ICLR. 2024.
[11] Liu, H., K. Simonyan, Y. Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: arrriv:1806.... 880650955, 2018.
[12] Zhang, B., Wang. X. Qin, Boosting order-preserving and transferability for neural architecture search: a joint architecture refined search and fine-tuning approach. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5662-5671. 2024.
[13] Krishnakumar, A., C. White, A. Zela, et al. Maa-bbench-siie suite-zeutez- Accelerating research on zero cost proxies. Advances in Neural Information Processing Systems (NeurIPS), 35:28037-28051,, 2022.
[14] Shu, Y., Z. Dai, Z. Wu, et al. Unifying and boosting gradient-basee training- free neural architecture search. In A. H. Oh, A. Agarwal, D. Belgrave, K. Cho, eds., Advances in Neural Information Processing Systems (NeurlPS). 2022.
[15] He, Z., Y. Shu, Z. Dai, et al. Robustifying and boosting training-free neural architecture search. In The Twelfth International Conference Learning Representations (ICLR). 2024.
[16] Akhauri, Y., J. Munoz, N. Jain, et al. Eznas: evolving zero-cost proxies for neural architecture scoring. Advances in Neural Information Processing Systems (NeurIPS). 1.30459-30470, 2022.
[17] Wei, Z., P. Dong, Z. Hui, et al. Auto-prox: Training-free vision transformer architecture search via automatic proxy discovery. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pages 15814 15822. 2024.
[18] Sun. H. L. Li, P. Dong, al. Auto-das: Automated proxy discovery for training- free distillation aware architecture search. In European Conference on Computer Vision, pages 56-73. Springer, 2024,
[19] White. C., A. Zela. R. Ru, et al. How powerful are performance predictors in neural architecture search? Advances in Neural Information Processing Systems, 14:28454-28444 2021.

---

[20] Shen, Y., Y. Li, J. Zheng, et al. Proxybo: Accelerating neural architecture search via bayesian optimization with zero-cost proxies. In Proceedings of the AAAI conference on artificial intelligence, vol. 37, pages 9792-9801. 2023.
[21] Dong, X., Y. Yang. Nas-hench-201: Extending the scope of reproducible neural architecture search, arXiv preprint arXiv:2001.00326 2020.
[22] Wu, M.-T., C.-W. Tsai. Training-free neural architecture search: A review. ICT Express, 10:213-231, 2024.
[23] Lee, J., L. Xiao, S. Schocnholz, et al. Wide neural networks of any depth evolve as lincar models under gradient descent, Advances neural information processing systems, 32, 2019.
[24] Jacot, A., F. Gabriel, C. Hongler. Neural tangent kemell: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[25] Snoek, J., H. Larochelle, R. P. Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
[26] Veličković, P. G. Cucurull, A. Casanova, et al. Graph attention networks, arXiv preprint arXiv: 1710.10003. 2017.
[27] Hou. Z. Liu, Cen, Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 594-004, 2022.
[28] Vaswani, A., Shazeer, N. Parmar, al. Attention all you need. Advances in neural information processing systems, 30, 2017.
[29] He, K., X. Zhang, S. Ren, et al. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778. 2016.
[30] Chen, X., R. Wang, M. Cheng, et al. Drnas: Dirichlet neural architecture search. arXiv preprint arXiv:2006. 10355, 2020.
[31] Xiao, H.. Z. Wang. Z. Zhu, et al. Shapley-nas: Discovering operation contribution for neural architecture search. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 11892-11901. 2022.
[32] Ye, P. B. Li, Y. Li, et al. b-darts: Beta-decay regularization for differentiable architecture search. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10874 10883. 2022.
[33] Shu, Y., Cai, Z. Dai, al. NASI: Label- and data-agn neural architecture search at initialization. In International Conference on Learning Representations (ICLR). 2022.
[34] Duan, Y.. X. Chen, H. Xu, et al. Transnas-bench-U 101: Improving transferability and generalizability of cross-task neural architecture search. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5251-5260. 2021.
[35] Szegedy, C., W. Liu, Y. Jia, et al. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9. 2015.
[36] Howard, A. G., M. Zhu, B. Chen, al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.0481,, 2017.
[37] Liu, C., B. Zoph, M. Neumann, al. Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV), pages 19-34. 2018.
[38] Tan, M., B. Chen, R. Pang, et al. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2820-20000000 2828. 2019.
[39] Cai, H., L. Zhu, S. Han. Proxylessnas: Direct neural architecture search target task and hardware. arXiv preprint 1rXiv:8812.00: 2018
[40] Chen, X., C.-J Hsieh. Stabilizing differentiable architecture search via perturbation-base regularization.
In International conference on machine learning, pages 1554-1565. PMLR, 2020.
[41] Sun, Z. C. Ge, J. Wang, et al. Entropy-driven mixed-precisioe quantization for deep network design. Advances in Neural Information Processing Systems, 35:21508 -15200, 2022.
12

---

[42] Huang, G., Z. Liu, L. Van Der Maaten, et al. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recugnition, pages 4700-4708. 2017.
[43] Zoph, B., V. Vasudevan, J. Shlens, et al. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8697-8710. 2018.
[44] Pham, H., M. Guan, B. Zoph, et al. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pages 4095-4104. PMLR, 2018.
[45] Luo, R., F. Tian, T. Qin, et al. Neural architecture optimization. Advances in neural information processing systems, 31, 2018.
[46] Dong, X.. Y. Yang. Searching for robust neural architecture in four gpu hours. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1761-1770. 2019.
[47] Yao, Q., J. Xu, W.W. Tu, et al. Efficient neural architecture search via proximal iterations. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pages 6664-6671. 2020.
[48] Xie, S., H. Zheng, C. Liu, et al. arXiv: arXiv:1812.088 10w...999.99 2018.
Snas: stochastic neural architecture search. arXiv preprint
[49] Dong. X., Y. Yang. One-shot neural architecture search via sell-evaluated template network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3681-3691 2019.
[50] Zela, A., T. Elsken, T. Saikia, et al. Understanding and robustifying differentiable architecture search. arXiv preprint arXiv:1909, 09656, 2019.
[51] Sun, Y., B. Xuc, M. Zhang, et al. Evolving deep convolutional neural networks for image classification. IEEE Transactions on Evolutionary Computation, 24:394-407, 4077, 2019.
[52] White, C., W. Neiswanger, Y. Savani. Bananas: Bayesian optimization with neural architectures for neural architecture search. In Proceedings of the AAAI conference on artificial intelligence, vol. 35, pages 10293-10301. 2021.
[53] Mehta, Y. C. White, A. Zela, et al. Nas bas-bench--uing Nas evaluation is (now) surprisingly casy. arXiv preprint XXiv:2201.133900 2022.
[54] Qiu, X., R. Müikkulainen. Shortest edit path crossover: A theory-driven solution to the permutation problem in evolutionary neural architecture search. In International Conference on Machine Learning, pages 28422-28447. PMLR, 2023.
[55] Krizhevsky, A., G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[56] Chrabaszez, P., I. Loshchilov, F. Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint 1707.08777 2017.
[57] Zamir, A. R., A. Sax, W. Shen, al. Taskonomy: Disentangling task transfer lcarning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3712-3722. 2018.
[58] Theis, I.,, Korshunova, A. Tejani, et al. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv: 00,77777 2018.
[59] Turner, J., E. J. Crowley, M. O' O'Boyle, al. Blockswap: Fisher-guided block substitution for network compression on budget. arXiv preprint arXiv: 041 13, 2019.
[60] Lee, N., T. Ajanthan, P. H. Torr. Snip: Single-shot network pruning based connection sensitivity. arXiv preprint 1000:38800 2018
[61] Wang, C., G. Zhang, R. Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint 20x::200. 2020.
[62] Tanaka, H., D. Kunin, D. L. Yamins, et al. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33:6377-6388, 2020.
[63] Mellor, J. J. Turner, A. Storkey, et al. Neural architecture search without training. In International conference machine learning, pages 7588-7598. PMLR, 2021.
[64] DeVries, T. G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 0455000.4552221111
13

---

[65] Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
[66] Mills, K. G., F. Han, J. Zhang, et al. Gennape: Towards generalized neural architecture performance estimators. In Proceedings the AAAI Conference Artificial Intelligence, vol. 37, pages 1190-9199, 2023.
[67] Akhauri, Y., M. S. Abdelfattah. Encodings for prediction-based neural architecture search. arXiv preprint arXiv:2403.0 0244444444 2024.
[68] Falkner, S.. A. Klein, Hutter. Bohb: Robust and efficient hyperparameter optimization scale. In International conference on machine learning, pages 1437-1446. PMLR, 2018.
[69] Yan, S., Y. Zheng, W. Ao, et al. Does unsupervised architecture representation learning help neural architecture search? Advances in neural information processing systems, 33:12486 12498, 2020.
[70] Yan, S., K. Song, F. Liu, al. Cate: Computation-aware neural architecture encoding with transformers. In International Conference on Machine Learning, pages 11670-11681 1681, PMLR, 2021.
[71] Zhang, M., W. Huang, B. Yang. Interpreting operation selection in differentiable architecture search: perspective from influence-directed explanations. Advances Neural Information Processing Systems, 35:31902-31914 2022.
14

---

A More Details about Evolutionary Algorithm
In this section of the appendix. we provide detailed explanation of the evolutionary search algorithm employed PO-NAS. PO-NAS utillizes cell-based search space, similar that found in related work such as DARTS (IT). Regarding the search strategy, PO-NAS selects parent architectures based on operation costs and the scores provided by the surrogate model. The search process involves series of candidate offspring rather than single network at each step. This population-based based approach allows for broader coverage of the search space, thereby increasing the likelihood of discovering pirh-peeformance architectures. Although evolutionary search algorithms are typically resource-intensive due to the need for multiple evaluations, PO-NAS mitigates this drawback by leveraging the low assessment costs of the surrogate model. Thus, our aim is to offer evolutionary algorithm that both efficient and effective, enabling extensive exploration of the search space without incurring excessive computational costs.
Avg
2
Skip 3onnect-3
4
0
0
0
0
Sep
N
0
5
0
0
Sep
Conv 3x3
Conv 5x5
(a)
2
N
0
0 0
2 1
0
0
0
2 1 0
0
0
0
2
0
0
0
4
0
5
0
0 0
0
5
0
0
3
0
0
0
0
2
0
1
0
0
3
1
0
0
3
1
(b)
(c)
Figure 6: Matrix encoding method of PO-NAS. (a) Matrix encoding method of PO-NAS (DARTS). (b) Changes in node operation types correspond to changes in matrix elements. (c) Changes in node connection targets correspond to the swapping of matrix elements.
A.1 Architecture Coding Method
Figure 6 iillustrate the cellular matrix encoding representation of PO-NAS within the DARTS search space. With this encoding, PO-NAS can acquire structural information of specific architectures for subsequent calculations minimal operation paths and costs. As shown in Figure a unique DARTS architecture is encoded as 4x7 matrix, where each index represents type of connection, with specific operations depicted in the figure. The matrix can be adjusted based on the type of search space. A '0' in the matrix indicates the absence of connection, and the matrix is segmented according to the connection patterns. In the case the DARTS search space, since each operational node is only connected to the two preceding nodes, we allocate corresponding matrix elements based on the number connectable nodes for each node, forming an upper triangular matrix. As DARTS includes both normal cells and reduction cells, combining two upper triangular matrices into one represents unique DARTS structure. With this matrix representation, evolutionary searches and calculations of the shortest operation paths can be performed by simply manipulating the matrix. For
15

---

instance, operation types can be changed by altering the values of specific elements, or the connection targets of individual nodes can be modified by swapping the positions of elements within the matrix.
2
1
0
0
0
0
0
2
1
0
0
0
0
0
4
0
6
0
0
0
0
4
0
6
0
0
0
0
0
5
0
7
0
0
0
0
7
0
0
0
0
0
0
0
0
0
[
7
1
0
[70]
10]100][0010011000
7
[7]]1][07][0]]077]
7
|7]1
1
[7]][A]][01][O]][11]]
[76][16]006][06]1000
10
[70][10][00]000100|
0
[70]10][000[000000]
0
[70][1]]000[00]000]
101
[70]100[00][00]100]
101
[70]10][[0][[0]000
Shortest Path Operation
[op], op2, ]
Figure 7: Calculation method of the shortest operation path and the minimum operation cost. This method operates on set of nodes (for example, a node vector of length 5), initially expanding two node vectors into concatenated matrix, followed by the application of Algorithm to carry out the specific computation process.
A.2 Minimal Operation Path and Cost
Research indicates that the predictive performance differences between architectures are positively correlated with their Graph Edit Distance (GED) (55), (52) (53), (54)) The Shortest Edit Path (SEP) effectively encodes the fundamental differences between two architectures. Inspired by these studies, we employ the shortest operation path and minimum operation cost to measure the differences between architectures. The shortest operation path is defined as the minimum set of operations required to transform one architecture into another within set of architectures through matrix-based transformations, which include element changes and clement swaps. This path not only considers the direct changes in the elements of the architecture but also the changes in the relative positions of the elements. In this way, differences between architectures. The minimum operation cost refers to the total sum all operations in the shortest operation path. The minimum operation cost provides method for quantifying structural differences, allowing us to prioritize architectures during the search process that can achieve significant performance improvements with fewer operations and to measure the extent of exploration of the search space in the current search process. The specific calculation method is illustrated in Figure and Algorithm
more accurately assess the similarity and
Specifically, our calculations are based on a set of nodes (for example, node vectors of length 5), expanding two node vectors into concatenated matrix. Using this concatenated matrix, compute the shortest operation path and minimum operation cost, then sum the shortest operation paths and minimum operation costs of all nodes to obtain the overall shortest operation path and minimum operation cost between two different architectures.
In single concatenated matrix, the elements on the diagonal represent the positional information of operations in the two node vectors, while the elements within the node vectors represent the type of operations. Our goal is to align the positional elements of the node vectors (i.e., the two elements on the diagonal are either both or neither 0) and to make the operation types of the node vectors consistent (i.c., the two elements on the diagonal are equal).
16

---

Let the individual element vector of the concatenated matrix be denoted as A., (A. To minimize the operation path, we prioritize positional transformations of elements that have the same operation type but different positions: select elements off the diagonal where A = B, denote the column index of these elements as j. and then swap all elements B that row with all elements B in row (lines 2-9 of Algorithm 2 Next, we align the positional elements of the node vectors: when the number clements on the diagonal where = B is less than 2, select elements where neither A nor B is O, denote the row index as i and the column index as j; if element A at A., is and element B at A is also O, swap all elements in row with all elements B in row (lines 10-19 of Algorithm 2). Finally, elements on the diagonal where A * B, and set B A (lines 20-27 of Algorithm 2). We record all transformation operations and calculate the length of the operation set. thereby obtaining the shortest operation path and minimum operation cost for the architectural combination.
make the operation types of the node vectors the same: select
Algorithm 2 calculation minimal operation path and cost two nodes
Require: Concatenated matrix A, the number of rows in concatenated matrix R 1: C = 0,
2: for each pair (i. j) in combinations of rows from to R do
3: i + j and As [1]
A2) then
4:
Swap every [ with every At, A,,
5:
Record exchange operation as (e,(1,5)
6:
1 (O (1,
7:
C = C
8:
end
9:
end for
10: while the length La vector a {Aij, (i (1= j. A., [1] + 0, A. 0)) < do
11:
for each pair (i, j) in combinations of rows from to R do
12:
+ 0, Ar, [2] * 0, [1] = 0 and A, = 0 then Swap every A El,,, with every A (2,,,, if
13:
14:
Record exchange operation as (i,j)
15:
= 0 (Oa
16:
C C+++1
17:
end
18: end for
19: end while
20: for each pair (i. in combinations of rows from to R do
21: i = and A;, [1] + Ar,(22 then
22:
[t +- [
23:
Record change operation as Oct (As, [1), As,
24: 1 = 0 U OO (A,,, [1), A [2)]
25:
C=C+1
26:
end if
27:
end for
return the operation set O, the operation cost C
A.3 Crossover and Mutation
By leveraging the low-cost evaluation advantages of the surrogate model and training-free metrics, PO-NAS is capable of focusing on and assessing a vast number of architectures without incurring significant additional costs. This approach greatly expands the search scope, allowing the algorithm to effectively explore a broad architectural space. To more efficiently guide the search process for these architectures, we introduce novel crossover and mutation strategy. This strategy is specifically designed to simultaneously explore unknown subspaces and local optima within large search spaces.
Shortest operation path Crossover PO-NAS employs novel crossover method to generate offspring architectures. Specifically, we first calculate the shortest operation path between two parent architectures, representing the minimum set of operations required to transform one parent architecture into the other. We then randomly shuffle the operational steps in this shortest operation path to ensure diversity and exploratory nature in the search process, thereby increasing the coverage
17

---

of the search space. Subsequently, to balance the need to explore new architectural spaces with the utilization of known effective operations, we randomly select half of the shuffled operational steps for application. The chosen operations are applied to one of the parent architectures, thereby generating a new offspring architecture. Shortest operation path crossover is illustrated in Figure
Neighborhood Traversal Mutation After completing the crossover operation, to enhance the exploration capabilities around local optima, we traverse all neighboring architectures that have an operation cost of with respect to the newly generated offssring architectures. By traversing these neighboring architectures, we are able to identify subtle changes that may lead significant performance improvements, without straying too far from the current architecture. This approach achieves balance between maintaining the stability of the solution and exploring new solutions. Neighborhood traversal mutation is illustrated in Figure g
Old1
Crossover
&
Crossover New
News
Shortest Path Operation
Operation
Old2
a
(b)
Figure 8: Shortest operation path crossover and neighborhood traversal mutation
A.4 Search Strategy
To balance the exploration and exploitation of the search space, we introduce an exploration weight N (0, 1). During each iteration, we select a subset of superior architectures based on the surrogate model's scores. These architectures are then paired, and the operation cost for each pair is calculated and normalized, denoted Scost- Additionally, these architecture pairs and normalize it, denoted Spre. The final score for the architecture pair is denoted as follows:
compute the sum of surrogate model scores for
Spair N NSoeet + (1- - N)
In the early stages of the search, we set a higher exploration weight N to enrich the distribution of offsspring architectures and enhance the exploration of the search space. As the search progresses, we reduce the exploration weight focus more on the performance scores of the architecture pairs, aiming to identify the optimal architecture. All offspring architectures generated by the evolutionary algorithm are predicted using the surrogate model, combining the training- -fre metric predictor and the surrogate model weights to jointly predict scores. Given the potential errors in the training- free metric predictor, we select the top-performing offspring based on predicted scores and further calculate their actual training-fret metrics to ensure accurate assessment. This approach significantly reduces the evaluation cost of offspring architectures, allowing PO-NAS to thoroughly explore large search spaces at low training cost.
18

---

B Experimental Details
B.1 Benchmark
NAS-Bench-201 NAS-Bench-201 (2) a widely utilized benchmark for Neural Architecture Search (NAS), with its search space primarily composed of stacked cellular structures. Each cell consists 4 nodes interconnected by edges. The operation on each edge can be one of the following five types: 3x3 convolution, Ixl convolution, 3x3 average pooling, zeroize, skip connection. Consequently, the search space encompasses a total of 5 15,625 distinct neural architectures. These architectures have been evaluated across three datasets: CIFAR-10 (C10) (55), CIFAR- 100 (C100), and ImageNet-16- 120 (IN-16) (56).
- (54) offers a variety of search spaces and benchmark tasks, typically used to assess the transferability of Neural Architecture Search (NAS) algorithms across different tasks. Unlike
TransNAS- MOANNA---ench-00
which primarily focuses on cellular structures, TransNAS-Ban -iI-AA--eec----e 101 explores both micro cellular structures and macro skeleton structures. The cellular search space is similar to that of NAS-Bench-
but includes only four operations (excluding average pooling), resulting in a total of 4 = 4,096 neural architectures. In the macro search space, the cell structure is fixed, while the skeleton is variable. The skeleton consists of 4 to 6 modules, each containing two residual blocks. Each module can choose to downsample the feature map, double the number of channels, or perform both operations. Throughout the entire architecture, downsampling may occur to times, and channel doubling may occur I to times, yielding a total of 3,256 distinct evaluates these architectures all using a single dataset of 120K indoor scene images derived from the Taskonomy project (57). These tasks include scene classification, object detection, jigsaw puzzle solving, room layout analysis, semantic segmentation, surface normal estimation, and autoencoding.
architectures.
seven different visual tasks,
DARTS DARTS (11) is well-known example of an extensive search space in the domain of neural architecture search (NAS). An architecture in DARTS comprises two types of cells: a normal cell and reduction cell. Each cell features two input nodes and four intermediate nodes, where each intermediate node is connected to two randomly chosen preceding nodes via an edge. Across each edge between the nodes, selection of seven different operations can be applied. In contrast to other benchmarks, DARTS boasts an immense search space, encompassing 10 distinct architectures.
B.2 Training-free Metric
Grad _ Abdelfattah et al. (5) introduced a set of training-free scoring functions, drawing inspi- ration from the literature on pruning at initialization. Among these, the gradient norm (Grad Norm) metric computes the aggregate Euclidean norm of the gradients following the propagation of single mini-batch through the network.
The Fisher metric was initially introduced in 588 to quantify the impact of model parameters on loss, facilitating the removal of activation channels with minimal loss impact through channel pruning. Turner et al. (59) aggregated the Fisher metric across all channels within a convolutional primitive to assess the significance of that primitive when substituted with more efficient alternative. It can be formulated as follows:
Fisher
[0C Oz
M
(2)
S, Σ S2() 1=1
(10)
where S2 is the saliency per activation z, and M is the length the vectorized feature map.
SNIP Lee et al. (60) proposed performing parameter pruning based on saliency metric computed initialization using a single minibatch of data. This metric approximates the change loss when specific parameter is removed. SNIP calculates the saliency metric before training with only single mini-batch of input data to estimate the loss variation caused by the removal of particular parameter.
0c
8,
3
(11)
00
19

---

where L is the loss function of neural network with parameters 0 and C is the Hadamard product.
Grasp Wang et al. (61) aimed to enhance the SNIP metric by approximating the gradient norm change upon parameter pruning. Grasp measures the gradient norm alteration during neural network pruning. It can be formulated follows:
Sp(0) = 0L 00 0
(12)
where H is the Hessian, Sp is the per-parameter saliency.
SynFlow Tanaka al. (62) generalized synaptic saliency scores and proposed SynFlow, modified version that prevents layer collapse during parameter pruning. SynFlow computes loss based on the product of all network parameters, eliminating the need for data.
ac 00
(,)
3
(13)
Jacobian Covariance (Jacob _Cvv Jacob Cov (63 specifically designed evaluate neural networks in the context of NAS. To put it simply, it reflects the correlation between activations within neural network when influenced by different inputs minibatch of data. The lower the correlation, the better the network's 's performance, as it indicates the network's 's ability to effectively distinguish between various inputs.
B.3 Implementation Details of PO-NAS
Normalize To equitably assess the sensitivity of various architectures to different metrics, we normalize the training-free training metrics and metric weights using the following normalization function to map them into the interval (0,1). For the i-th training-free metric 2; of single architecture:
Z: Zimin
2,
(14)
Zi,ax Zimin
Zi,aa and mii represent the maximum and minimum values of the i-th metric across all architectures. For the i-th training-fre frre metric weight wy of a single architecture:
W/
(15)
|wll
where ||lm| denotes the L1 norm of the weight vector w including the weights of all training-free metrics for a single architecture. To enable the model to account for both the positive and negative correlations of metries, we retain the positive and negative correlations of model weights.
Surrogate Model In this section, we provide a detailed description of the training setup for the surrogate model in PO-NAS. After series of experimental tests, we identify optimal set of training parameters and apply a uniform training strategy across all datasets. During the pre-training phase, divide the training dataset for the architecture encoder into training and test sets in 4 ratio and randomly mask 20% architectures for the mask reconstruction task. We train the architecture encoder using the initial architectures and their corresponding training-frre free metrics, employing stochastic gradient descent (SGD) over 100 epochs. In the first 5 epochs, the learning rate is initially increased to x 10-3 and then gradually reduced to 0 according to a cosine annealing schedule, with batch size of 64. In the Bayesian Optimization (BO) phase, we set a loss threshold of 0.1 (for some tasks in TransNAS AAA--Bench--00
adjustments are made due the peculiarity of the loss metric values) and maximum number of iterations of 100. Each iteration begins with a training period of 100 epochs, increasing by 10% per iteration, and the model weights are reset after each iteration until the model reaches the loss threshold or the maximum number of iterations is reached, after which the best model weights are saved. The loss difference threshold Teh set to 0.1 (adjustments are made for some tasks in MaassNAS--enchh 101 due to the peculiarity of the loss metric values). We use the Adaptive Moment Estimation (Adam) optimizer to train the surrogate model. In the first 10 epochs, the learning rate is initially increased to 3 x 10-1 and then gradually reduced to 0 according to cosine annealing schedule, with a weight decay of 0.01. Additionally, we employ gradient clipping with max norm 1 to prevent gradient explosion.
20

---

PO-NAS on NAS-Bench-201 and TransNAS-Bench-101 For JAS-Bennh- 201 and TransNAS- Bench- 101, to assess the effectiveness of our method through ablation studies, we adopt the same training-free metrics as those used in the RoBoTT experiment. These metrics include the six training- free metries outlined by (5): grad norm, snip, grasp, fisher, synflow, and jacob cov. To ensure the reproducibility of the experimental results, we utilize and MaanNAAS-Bencc-e Due to the relatively small search space, we utilize only the surrogate model without employing the evolutionary algorithm. For DAS--ench-2201, we maintain the experimental conditions consistent with RoBoT and HNAS, using the CIFAR-10 10 validation performance after 12 training epochs from the table data in as the objective evaluation metric for all three datasets, and calculate the search costs displayed in the same manner (i.e.. the training cost of 20 architectures). However, we report the full training test accuracy of the proposed architectures after 200 epochs. For the training tasks Segmentation, Normal, and Autoencoding in SeeeeNAS--eech--lee
(13) to calculate these
training-free free metrics for both
maintain consistency in experimental conditions, we do not use the training-free metric Synflow and only employ the remaining five training- free metrics. For Trans MAAAS-Been--1111 we only report the validation performance of the architectures identified through the search process.
PO-NAS on DARTS For the DARTS search space, we establish pool of 10000 diverse architec- tures and calculate six training-free free metrics for these architectures: grad norm, snip, grasp. fisher, synflow, and jacob cov. These assessments are conducted
the corresponding datasets. For the CIFAR-10 10 and CIFAR- 1-00 datasets, we allocate a budget of 25 search attempts for PO-NAS, with each optimal architecture identified undergoing 10 epochs of training. For the ImageNet dataset, set a budget of 10 search attempts, with each optimal architecture undergoing. 3 epochs of training. We select three initial architectures to initialize the surrogate model, which are the top three based on the average scores of the training-free metrics. (As the surrogate model predicts architecture scores based on combination of training-free metries, it can still demonstrate a certain level of evaluation performance even with limited number of initial architectures. Therefore, we do not choose to initialize more architectures.) We initiate the evolutionary algorithm at the 10th epoch (3rd epoch on ImageNet) of the Bayesian Optimization (BO) phase. Following the experimental setup of DARTS we construct a 20-layer network architecture based on the identified cell structures. The initial number of channels for these architectures is set to 36, with the auxiliary tower weight set to 0.4 for CIFAR-10, located the 13th layer: for IIFAR-100. the auxiliary tower weight is set to 0.6. We test these architectures on CIFAR- 10 and CIFAR- 100 through 600 epochs of stochastic gradient descent (SGD). The learning rate starts at 0.025, gradually decreasing to 0 for CIFAR- 10 and from 0.035 to 0.001 for CIFAR- 100, using a cosine annealing strategy. Momentum is set to 0.9, weight decay to 3 x 10-4, and batch size to 96. Additionally, we employ Cutout (64) and ScheduledDropPatt as regularization techniques. which are linearly increased from 0 to 0.2 for CIFAR-10 and from 0 to 0.3 for CIFAR-100. For ImageNet, we train a 14-layer architecture from scratch for 250 epochs with a batch size of 1024. In the first five epochs, the learning rate is initially increased to 0.7, then gradually decreases to zero according to a cosine schedule. When using the SGD optimizer, momentum is 0.9, and weight decay is 3 x 10
Average and Best Training-free Metric Performance (Avg.) (Best) We report on the evaluation performance of the training-free metrics employed by PO-NAS. We compute the training- free metrics for all architectures and select the ones with the highest scores for further assessment. Specifically, "Best" refers to the architecture that demonstrated the most outstanding performance among the optimal architectures identified by separate evaluations of different metrics, while "Avg." denotes the architecture with the highest average score across all metrics.
Search Costs of PO-NAS The search cost of PO-NAS comprised of four primary components: First, the training cost of the architecture encoder during the pre-training phase; Second, the evaluation cost of real training performance during the Bayesian Optimization (BO) phase; Third, the prediction and training cost of the surrogate model in the BO phase: Lastly, the cost associated with crossover and mutation operations and the computation of offspring's 's training-free metries during the evolutionary algorithm phase (this cost is omitted for NAS- S---Becch-200 and TransNAS -eeSS--Bh----ee 101 as they do not employ an evolutionary algorithm). For the real training performance evaluation cost of WAS-Bench- 201 and MaanNNAS-Benchl 1001, we directly utilize the data provided in the benchmark tables.
21

---

B.4 Other Implementation Details
Classification Details of Architecture Sets in T The architecture sets are categorized based on the proportion of operations and the magnitude of parameters. Specifically, OpO to Op4 denote architecture sets in which the corresponding operation constitutes at least 50% of the architecture (each set comprises 1,545 architectures). In contrast, the Mix category represents architecture sets with evenly distributed operations (consisting of 1,440 architectures). For the parameter magnitude classification, each category includes 5,208 architectures. The operations are defined as follows: Op0 represents zeroize, Opl represents skip connection, Op2 represents Ixl convolution, Op3 represents 3x3 convolution, and Op4 represents 3x3 average pooling.
Random Search (RS) We randomly sample network architectures from a predefined search space, evaluate their performance, and ultimately select the architecture with the best performance. Random search highly flexible and makes it a common baseline for NAS algorithms.
Regularized Evolutionary Algorithm (REA) 1 Regularized Evolutionary Algorithm (REA) is commonly used baseline for NAS algorithms. REA initializes the population by randomly selecting a batch of architectures and evaluating their performance. In our experiments, we use one-third of the current experimental budget for initialization. Then, the best-performing architecture is selected from the initial population, mutated to produce new offspring, which is evaluated and added back the population. For cell-based search spaces (e.g., NAS-Bench-201 and TransNAS- Bench- 101 101-mi)ro), we change the operation type on an edge as the mutation operation. (For TransNAS MaasNAAA--encc-1000 senchh-01-maerr we randomly add or remove a downsampling/doubiig operation on one residual module.)
REINFORCE (65) REINFORCE is classic policy gradient reinforcement learning algorithm that directly optimizes the policy to maximize the expected cumulative reward. In our comparative experiments, REINFORCE is used as one of the baseline algorithms. We adopt the same configuration as (21) and use the Adam optimizer with a learning rate of 0.01 to update the parameters of the policy network. Additionally, to stabilize the training process, we introduce an exponential moving average with momentum of 0.9 as the reward baseline.
Hybrid Neural Architecture Search (HNAS) 14 HNAS is one of the classic hybrid NAS algorithms, which we use as a benchmark for comparison in our experiments. HNAS employs Bayesian Optimization (BO) to optimize training-frre free metrics based on the true training performance of architectures. Since the training-free metrics used by HNAS are gradient-basede we follow their experimental setup and use the gradient norm as the training-free metric in experiments.
Robustifying and Boosting Training-free Neural Architecture Search (RoBoT) (15) Similar to PO-NAS, RoBoT employs a weighted linear combination to integrate multiple training-free free metries, and we use as benchmark for comparison in our experiments. RoBoT utilizes Bayesian Optimization (BO) to dynamically adjust the weights of these metrics based on real-time training feedback. Following their experimental setup, we use six classic training- free metrics (grad norm, snip, grasp, fisher, synflow, and jacob_coo cov) as the optimized combination.
C More Empirical Results
C.1 Correlation of Different Architecture Sets
We classify architectures in the NAS-Bench-201 (1) dataset based on their structural characteristics and calculate the Spearman correlation coefficients of several classic metrics across these architectural classifications. Implementation details can be found in Appendix B.4
As shown in Table 5 the results indicate that even within the same search space. due to differences in operational composition and parameter volume, the correlation coefficients of different training-free metrics for these architectures can vary significantly. For instance, although Jacob cov (63) generally exhibits strong performance, its correlation drops sharply to 0.33 when evaluating architectures dominated by 3x3 convolutional operations (Op3). Ignoring the differences in the assessment performance of these metrics across various types of architectures during the optimization of training- free metric combinations will inevitably affect the ability of the final metric combination to select the best architecture. Therefore, optimizing different metric combinations for different types of architectures can further enhance the evaluation capability of training- free metric combinations.
22

---

Table 5: Spearman correlation coefficients between several classic training-free free metrics and the actual performance of architectures within different architecture sets on the NAS-Bench- dataset. Experimental details can be found in Appendix B.4
Operation
Params
Metrics
All
Op0
Opl
Op2
Op3
Op4
Mix
Low
Med
High
Grad norm
0.74
0.76
0.06
0.35
0.71
0.48
0.55
0.34
0.22
0.59
Snip
0.76
0.75
0.06
0.36
0.70
0.48
0.56
0.33
0.22
0.60
Grasp
0.73
0.56
0.15
0.34
0.49
0.36
0.46
0.30
0.27
0.52
Fisher
0.69
0.65
0.02
0.30
0.69
0.42
0.46
0.27
0.17
0.51
Synflow
0.63
0.85
0.44
0.76
0.72
0.58
0.56
0.46
0.51
0.73
Jacob_cov
0.88
0.87
0.52
0,33
0.89
0.62
0.86
0.74
0.66
0.75
zen
0.05
0.86
0.39
0.60
0.22
0.50
0.24
0.09
0.27
0.35
Nwo
0.85
0.85
0.48
0.69
0.83
0.63
0.77
0.62
0.60
0.75
Params
0.70
0,42
0.40
0.79
0.71
0.58
0.43
0.42
0.48
0.72
Flops
0.64
0.39
0.32
0.76
0.69
0.56
0.35
0.28
0.42
0.70
C.2 PO-NAS on NAS-Bench-201
We evaluate the search performance of PO-NAS on -2S-Bench--21. Due to the relatively small search space, we utilize only the surrogate model without employing the evolutionary algorithm. and Figure 9 present the performance outcomes on three benchmark
Table Table (6) Table Table Table datasets, with "Optimal" indicating the upper limit of this benchmark, corresponding to the archi- tecture with the highest accuracy within the excellent performance across three datasets. also assess the performance of the training- free metrics utilize, including the architectures with the highest average metric scores (Avg.) and the highest score for a single optimal metric (Best). The results show that PO-NAS significantly enhances the evaluation capability of the original metrics. As PO-NAS optimizes the metric combinations corresponding to different types of architectures, it can more effectively explore superior architectures compared to other training-free free NAS methods that use global optimization metric combinations.
search space. PO-NAS demonstrates
Table 6: Comparison of the number of architectures that must be trained for GENNAPE, FLAN CAZ. and PO-NAS attain the target test accuracy NAS-Bench-22 CIFAR 10.
Algorithm
GENNAPE 66
FLANT
PO-NAS
Trained models
32
20
Test Accuracy (%)
93.27
93.30
94.12
Table 7: The mean and standard deviation of the test accuracy($)) of BOHB (68), arch2vec (69 and PO-NAS under three datasets MMS-Bench-221. All methods search for 20 architectures. All results are reported as the mean + standard deviation over 10 runs
Accuracy (%)
Cost
Algorithm
(GPU Seconds)
C10
C100
IN-16
BOHB
-3.52
72.37-10.90
45.26:1:0.3
12000
arch2vec-RL
0..12
03.15
46.16:10..8
12000
arch2vec-BO
94.18-0.22
0..3
46.27-40.37
12000
PO-NAS
94.12++0.22
0.05
46.71+-0.12
3162
C33 PO-NAS on TransNAS-Bench-100
To verify the robustness of PO-NAS across various training tasks, we conduct a performance evalua- tion on TransNAS MAA-A---Bcch--iee 101. We design ablation studies to demonstrate the superior performance
23

---

C10
95
Test Ranking
95-
Tost
91-
5
10
15
20
10
15
20
C100
2
72
Accuracy
Test Ranking
2
21.
68-
2²
IN-16
48
24
46
Test Accuracy (%0
44
Text Ranking
42
72.
Number Scarchod Architectures
PO-NAS(wel Evo.)
RolBot
-
REA
BEST OPTIMAL
Figure 9: Comparison of various NAS algorithms in NAS-Bench-201 regarding the number of searched architectures. Results are reported with mean and standard deviation of 10 runs, search costs are evaluated on an Nvidia 1080Ti. "AVG" and "BEST" represent the average value of metrics and the best individual metrics. The best results are in bold, and the second best are underlined.
of architecture embedding-bol surrogate model. Due the relatively small search space, we utilize only the surrogate model without employing the evolutionary algorithm. We compare PO-NAS with several other hybrid NAS methods known for their robustness, as well as several benchmark search methods. Table 4 Table 9 Figure [4 and Figure [5 present the experimental results, indicating that PO-NAS achieves excellent performance across 14 different training tasks and outperforms other hybrid NAS methods in most of these tasks. This is attributed to PO-NAS's ability to optimize the corresponding metric combinations for different types of architectures. Notably, PO-NAS demonstrates significant improvements in handling the search space of macro architectures (including chain-like architectures) compared other NAS methods and consistently identifies optimal architectures within the search space across multiple training tasks. This confirms the great potential PO-NAS in search spaces with regular distributions like macro architectures, as the surrogate model can more effectively optimize the metric combinations for each architecture.

---

Table 8: Test performance ranking comparison among various NAS algorithms on MAS-Bench-2011 All methods search for 20 architectures. The results for RoBoT, HNAS, and PO-NAS are reported as the mean + standard deviation over 10 runs, while REA, RS, and REINFORCE are evaluated over 50 runs. "Training-fre (Avg./Best) represent the average value of metrics and the best individual metrics.
Test Ranking
Algorithm
C10
C100
IN-16
REA
2259:1550
2330:1540
2399:1577
RS (w/o sharing)
1346#:413
1194#3777
1493#363
REINFORCE
2512:440
1919=351
2400=330
HNAS
34426
72=20
53+10
RoBoT
3+00
1+0
24+0
Training Training-frre (Avg.)
4822
357
82
Training-fres (Best)
169
173
1780
PO-NAS
21
1:0
7:55
Table 9: Test performance ranking comparison among various NAS algorithms on TransNAS-Bende
101. All methods search for 100 architectures. The results for RoBoT, HNAS, and PO-NAS reported as the mean + standard deviation over 10 runs, while REA, RS, and REINFORCE are evaluated over 50 runs. "Training-fres (Avg./Best)" represent the average value of metrics and the best individual metrics.
Test Ranking
Space
Algorithm
Scene
Object
Jigsaw
Layout
Segment.
Normal
Autoenco.
REA
21=16
13#9
26423
17=11
9+7
32426
16#11
RS (w/o sharing)
30:24
20:12
50+23
19#13
33+22
80#67
37*11
REINFORCE
40:29
2924
29+21
14#11
19:16
13+9
27+22
HNAS
96232
84a15
41+25
374#323
4+0
2243
1329456
Micro
RoBoT
3+0
19#3
7+4
3+0
6t4
29+7
Training-frie friningg-fre (Avg.)
2863
2731
2701
2654
324
2472
3591
Training-free (Best)
500
2372
1928
1904
22
722
3151
PO-NAS
3+2
1+0
17#5
12:4
2+0
12
26+10
REA
9+7
26:15
20=12
10+8
19+14
RS (w/o sharing)
14211
26422
31424
21
32+26
18:15
20+13
REINFORCE
27=21
20=16
13#9
19=12
36+24
34+27
31#22
HNAS
506240
535+41
149+27
5047
1840
202+30
671431
Macro
RoBoT
1=0
6#:3
5#1
4#3
1+0
2+0
6+1
Training-free (Avg.)
823
1278
751
1021
908
8
116
Training- free (Best)
54
117
72
366
138
8
98
PO-NAS
1:0
140
C.4 PO-NAS on DARTS
We present the results of PO-NAS DARTS in Table 2 Table 3and Table TO
DARTS cell
architectures found by PO-NAS are presented in Figure T6 Figure 17 and Figure T8 combined efforts of the surrogate model and evolutionary algorithm, PO-NAS utilizes training- free metrics as auxiliary evaluation tools, effectively exploring the search space and demonstrating excellent performance under low-cost assessment conditions. Furthermore, PO-NAS relies solely on immediate training outcomes as supervisory signals, without the need for extensive training benchmarks, enabling to easily migrate large and unknown search spaces that lack substantial prior knowledge. However, PO-NAS also faces the inherent limitations of training-based NAS methods: there is still a certain gap between short-term real training performance (10 epochs/3 epochs on CIFAR-10/100 1000mmeee/0//mmmeNeett and the final test performance (600 epochs/250 epochs CIFAR- 10/100//mageNet). Thus, PO-NAS is currently primarily limited to quickly searching for optimal architectures within short training periods, and bridging the gap between short-term performance
With the
25

---

and final test performance is a goal that PO-NAS and srining-basse NAS methods need to further improve. Nonetheless, PO-NAS still demonstrates exceptional performance on the DARTS search space.
Table 10: Comparative results of CATE (70,, arch2vec, and PO-NAS DARTS Search Space with CIFAR-10. All results are reported as the mean + standard deviation over 10 runs
Algorithm
Test Error (%)
Params (M)
Search Cost
CATE-DNGO-LS (small budget)
2.55++0.08
3.5
3.3 (GPU days)
CATE-DNGO-LS (large budget)
2.46:10.05
4.1
10.3 (GPU days)
arch2vec-RL
2.65
3.3
8.3 (GPU days)
arch2vec-BO
2.56:10.05
3.6
9.2 (GPU days)
PO-NAS
2.52:10.03
3.8
3.9 (GPU hours)
D More Ablation Studies
D.1 Ablation Study on Surrogate Model
Ablation Study on Ensemble Method In this ablation study, we thoroughly investigate the impact of the surrogate model and its various components on the performance of PO-NAS. The experiments are conducted on the scene, object, jigsaw, and layout tasks in both the TransNAS- 101-rrrr and MaansNAS--enchhh 101-r datasets. We focus on three main aspects: ablation studies of global optimization versus per-architecture optimization (GP, w/o encoder), the influence of different normalization methods (w/o Negative, w/o Normal, Rank), and ablation studies of the surrogate model itself (Avg, Best). The detailed experimental settings are as follows: (a) PO-NAS: We use the default experimental settings on MaansNAS-Benchh 101; (b) GP: We replace the entire surrogate model with Gaussian Process (GP) surrogate model to perform training-fre free metric optimization in the global search space, using the performance of the scarched architectures in each round the supervision signal to optimize the surrogate model; (c) w/o Encoder: We set the output of the architecture encoder to name-dimensionn
vector with all elements equal to (i.e., all architectures output the same architecture embedding), thereby eliminating the influence of the encoder and optimizing the training- free metric weights using only the attention network in the global search space; (d) w/o Negative: We eliminate the influence of weight signs, making all weight outputs positive without considering the potential negative correlation of training-free metrics; (e) w/o Normal: We directly optimize the numerical values of the training-free metrics without normalizing them; (f) Rank: Instead of using the numerical values of the training-free metrics directly, we use the ranking of single architecture among all architectures for particular training- free metric as the training-five metrie; (g) Avg: We eliminate the surrogate model and select architectures using the average value of all training-frre metrics; (h) Best: We climinate the surrogate model and choose the optimal architecture from those selected by individual training-free metries as the result; (i) OPTIMAL: The performance of the optimal architecture on the entire dataset.
Figure 10 presents our experimental results, which fully demonstrate the effectiveness of our surrogate model and its various components. PO-NAS achieves a higher performance ceiling in the global search space by assigning training-free metric weights to each specific architecture, compared the other two methods. As the search progresses, PO-NAS is able to discover architectures with superior performance. Moreover, distinguishing between the positive and negative correlations of training- free metrics allows PO-NAS to leverage potentially negative-correlate Normalizing the metries can effectively reduce the optimization difficulty of the surrogate model. However, since ranking metrics forces uniformity in metric value differences, while the actual differences in training-fre free metric values among architectures with similar rankings may vary, using metric ranking as proxy metric may hinder the surrogate model's 's ability to accurately measure the performance of different architectures. Nevertheless, certain tasks (such as macro-jigsaw), the Rank method exhibits unexpected performance. Therefore, in practical applications, we recommend experimenting with different combinations based on the specific training task, which may lead to unexpected performance improvements.
metrics to enhance performance.
26

---

Micro
Scenc
Object
Jigsaw
Layout
2
244
The %
100 SCAANE
Toe 300250
Ina Woslang
100
25
50
100
25
30
75
100
23
30
100
Macro
Scene
Object
Jigsaw
Layout
233
21.
The Saskie
24
Tea Rocking
Number of Searched Architectures
25
PO-NAS -
- wio Facoder
Wio Negntive
wio Nommal
Rank-
Axg
Best
OPTIMAL
Figure 10: Comparison between different ensemble methods on tasks in MaansNAS-Beechhh 101 regarding the number of searched architectures. Note that all methods are reported with the mean and standard error of 10 independent searches.
Ablation Study on Pre-training Loss To delve into the performance of the architecture encoder and the training-free metric predictor, we calculate the predicted loss of various training-free metrics for scene, object, jigsaw, and layout tasks in TransNAS- SainsNAA--eenc--100 Bench-101-miero and TransNAS- -Bnnchh 101-macro. The results, shown in Table IT and Figure [1] indicate that across different training tasks, the predicted loss of the metric predictor is concentrated between 0.5 10-3 and 1.5 x 10 which corresponds
error margin of 2% to 3%. This demonstrates that our metric predictor possesses a certain level of predictive capability. Therefore, to prevent the impact of prediction errors on the selection of top-level architectures. PO-NAS utilizes the metric predictor for pre-scoring during the evolutionary algorithm phase. It selects outstanding offspring from the pre-scoring and then recalculates the metrics obtain the final scores. Additionally, in our experiments, we find that the predicted loss of training-free metrics also varies significantly across different search spaces and prediction tasks. For instance, Synflow exhibits considerably larger predicted loss in the micro space compared to other metrics but has a very small predicted loss in the macro space. The predicted loss some extent reflects the connection between training-fire free metrics and architectural features, which also affects the predictive accuracy of the subsequent surrogate model in the process. This experiment provides us with insights: For different search spaces and training tasks, we should selectively choose different combinations of metrics (with smaller predicted losses), rather than using fixed single combination of metrics, which may enhance the predictive performance of PO-NAS. We conduct rigorous analysis of the architectural features captured by each metric and employ the sensitivity analysis methods from DARTS-IM 77 to quantitatively assess the complementarity of the metrics.
A Alltionn Study on Architecture Encoder Supervision Signals We investigate the impact of different architecture encoder supervision signals on the performance of PO-NAS. As comparison, we employed recently developed architectural encoders that use actual training as supervision signals. is most comparable PO-
Among them, the experimental setup of arch2 on NAS-Bench-22 NAS. To preserve the original performance of arch2 we retained its official training conditions: obtaining its fixed embeddings through supervision by actual architecture performance, and then feeding these embeddings into the attention network for architecture discrimination, followed by a direct comparison with PO-NAS. The preliminary results are summarized in Table [2] The results indicate that replacing PO-NAS's encoder with existing SOTA architectural encoders yields only marginal improvements on certain datasets, while the overall performance remains virtually unchanged. This further corroborates the excellence of PO-NAS's encoder and demonstrates that
27

---

Table 11: Test loss of the metric predictor for various metrics on 8 tasks in TransNAS- BaA--eeechh-01 with a 1:4 training-to-tedt to-testiig set ratio over 100 epochs. The results are reported as the mean standard deviation over 10 runs.
Mse Loss (x10-9)
Space
Tasks
Grad norm
Snip
Grasp
Fisher
Synflow
Jabcob cov
Total
Scene
0.58#0.05
0.67+0.05
0.63+0.03
0.1 0.16+0.04
1,48=0.38
0,27+0.07
0.63#0.10
Object
0.6720.12 12
0.60:0.91
0.91=0.01
0.1720.04 04
188=0.21
1.83+1.64
1.31
Micro
ligsaw
0.47+0.01
0.67=0.02
0.09=0.00
0.10+0.01
1.46=0.27
0.38+0.17
0.53+0.08
Layout
0.86+0.02
0.78+0.01
0.33+0.03
0.20=0.01
1.47=0.17
0.28+0.16
0.65+0.07
Scene
1.45=0.07
0.74+0.05
2.94=0.03
1.44+0.08
0.10=0.03
2.00#0.09
1.45#0.06
Object
1.12#0.08
0.70=0.07
0.72+0.02
1.8840.13
0.10=0.02
1,08+0.03
0.87=0.06
Macro
Jigsaw
0.39#0.08
0.32+0.07
0.82+0.01
1.6900.07
0.14+0.03
1.34+0.05
0.62+0.05
Layout
2.01+0.05
1.22+0.03
1.73+0.02
1.41+0.01
0.11=0.03
1.64+0.11
1.35+0.04
Mícro
Scene
Object
Jigsaw
I Layout
Me 100000000
Me- LAN D010
Naa 200
Me
25
100
Macro
Scene
Object
Jigsaw
Layout
Man Lant (030"
Me-lon
Pre traintaiiin epoch
Soip
Grasp
Svaflow
Figure 11: Test loss of the metric predictor for various metries on 8 tasks in TransNAS-Bent
with a 1:4 training-to-testing set ratio over 100 epochs. The results are reported as the mean standard deviation over 10 runs.
learning architecture representations via training-free metries not only feasible but also highly efficient, significantly reducing the cost of acquiring topological information.
Table 12: Impact of Different Encoders Supervisory Signals on PO-NAS (reported by test accuracy (%) on NAS-Bench-201), All results are reported as the mean + standard deviation over 10 runs
Encoder CIFAR-10
CIFAR-100
ImageNet-16-1200
Supervisory
arch2vec
94.2510.18
73.51+0.00
46.49
short-term training (10 epochs)
PO-NAS 14.1221.22
73.51
46.7110.12
train-free metrics
Ablation Study on Difference Threshold Ta, and Loss Threshold To investigate the impact of the difference threshold Tn, (Section 3.4) and the loss threshold (Appendix [B.)) on the prediction performance of the surrogate model, we compare the search performance of different parameter settings (with the difference threshold on the x-axis and the loss threshold on the y-axis) across the scene, object, jigsaw, and layout tasks in TransNAS-Benchh -Bench-101-micee and TransNAS-Bende 101-macro. Table [5] presents our experimental results. The results demonstrate that different combinations of thresholds significantly influence the prediction performance of the surrogate model.
28

---

Lower combinations of loss and difference thresholds generally enhance the model's prediction aceuracy but slow down the training speed. A higher difference threshold allows the surrogate model to more easily reach lower loss threshold but increases the number of ranking errors, thereby affecting the final results. Moreover, higher loss threshold, regardless of the difference threshold setting, consistently degrades the model's prediction accuracy. Therefore, to balance model fitting speed and prediction accuracy, we recommend setting lower loss and difference thresholds while maintaining certain fitting speed, and ensuring that the gap between the two thresholds is not too large.
Table 13: Comparison of surrogate models with different parameters (X-axis: Difference Threshold, Y-axis: Loss Threshold) on SaansNAA--Benhh. across 8 tasks regarding the number of searched architectures. All models search for 100 architecture. The results are reported as the mean + standard deviation over 10 runs.
Test Rank
Space Loss/Differ (x10-1)
1
2
3
4
1
2
3
4
Scene
Object
3+2
3+0
3+0
4+3
1+0
1+0
1+0
1+0
2
3+2
3+2
3+2
613
1+0
1+0
110
140
543
5+3
745
745
1+0
4
6#3
5+3
644
7=5
120
1+0
140
140
Micro
Jigsaw
Layout
1745
1745
1745
1745
12+4
12+4
19+11
1244
2
1943
1745
1943
1725
1244
19411
1244
2148
1920
1940
1943
1943
19411
1745
29+12
19411
1943
17+5
1940
19#3
112+81
51+32
29+12
37+16
Scene
Object
1+0
1+0
1+0
1=0
2+0
2+0
2+0
3+0
140
140
1+0
1+0
2+0
240
340
240
3
140
140
140
1+0
2+0
240
2+0
340
1+0
1+0
1+0
1=0
2+0
2+0
3+0
3+0
Macro
Jigsaw
Layout
1
2+1
2+1
3+1
3+2
2+1
2+0
2+0
3+2
2+1
2+1
2+1
3+1
2+0
2+1
2+0
3+2
342
2+1
2+1
2+1
240
2+1
2+0
240
4
3+1
2+1
3+2
3+2
2+0
3+2
2+0
3+2
Ablation Study on Number of Training-free Metrics In this ablation study, we investigate the impact of the number of training-frre metrics the prediction performance of the surrogate model. We compare the search performance of surrogate models with different numbers of training- free metrics across the scene, object, jigsaw, and layout tasks in TransNAS SewesNAA--Bech----e Ben---nnhhhh- 101-micro and TransNAS -AAAA--enc----- 101 -maco.. The specific settings are as follows: (a) 6 Metrics: Consistent with the original setting of PO-NAS (Appendix [B33),
grad_norm, snip, grasp, fisher, synflow, and jacob_cov; (b) 8 Metrics: We add two additional metrics (params and flops) to the original six metrics; (c) Metrics: We remove
six metrics: grad_
metries with lower relevance (grasp and fisher) from the original six metrics; Figure 12 presents our experimental results. The findings indicate that when more training-free metrics are used, the need assign different metric weights for each architecture requires more training samples for surrogate model training. This leads to reduced stability and prediction performance in the early and middle stages of the search. However, more architectures are evaluated during the search, the higher number of metrics provides a greater upper limit on performance. Conversely, when fewer training- -free metrics are used, the surrogate model quickly achieves a certain level of prediction performance and identifies promising architectures the early stages of the search. However, later stages, especially for more complex tasks, its performance drops significantly compared to other methods. Therefore, in practical applications, the number of training-free metries should be chosen based on the complexity of the training task and the search cost budget. Given the fitting challenges of the surrogate model, improving the quality

---

of metrics (e.g., replacing existing metrics with more relevant ones) is more effective for enhancing performance than simply increasing the number of metrics.
Micro
Scene
Object
Jigsaw
Layout
Toa Rakiis
%
%
Macro
Object
Jigsaw
Layout
240
Toa Rokaa
Builing
25
20
75
100
25
100
15
100
0
100
Number of Searched Architectures
Mentrs
Meries
Figure 12: Comparison between utilizing different numbers of training-fiee metries on tasks in Saa-N-nS-Behhh 101 regarding the number of searched architectures. Note that all methods are reported with the mean and standard error of 10 independent searches.
D.2 Ablation Study on Evolutionary Algorithm
In this ablation study, we thoroughly investigate the impact of the proposed evolutionary algorithm and its various components on the performance of PO-NAS. Specifically, we conduct 100 search cycles in the DARTS search space, using our surrogate model in combination with different evolutionary algorithm designs. We evaluate their performance based on the test accuracy after 10 training epochs for comparison. and supplement the initial architecture population, conduct experiments with different initial population sizes (100, 1000, 10000, and 60000). avoid impact of initial population size on the training performance of the architecture encoder during the pre-training phase, the same architecture encoder (trained with 10000 architectures) used for all experiments with different initial population sizes. The evolutionary algorithm is applied in all 100 search cycles, and we ablate different components the evolutionary algorithm while maintaining relatively fair experimental conditions, follows: (a) PO-NAS: In each search cycle,
fully demonstrate the ability of our evolutionary algorithm explore
select the top 20 combinations for crossover operations and perform neighborhood mutation operations (generating 128 offspring), resulting in total of 2560 offspring. Finally, select the top 50 offspring to supplement the initial population; (b) Without Evolutionary Algorithm (w/o Evolution): this approach, we do not use the evolutionary algorithm and rely solely on the surrogate model for prediction. All architectures derived from the initial population; (c) Without Exploration Weight N (w/o N): In this method, we ablate the exploration weight N. All parent combinations selected based solely the surrogate model's scores, ignoring the impact of the architectural combination operation cost; (d) Without Neighborhood Mutation (w/o Mutation): In this method, we ablate the mutation operation. All offspring are generated through shortest-path crossover operations. To maintain fair experimental conditions, select the top 20 combinations each cycle, with each combination generating 128 offspring, resulting in total of 2560 offspring. Finally, we select the top 50 offspring to supplement the initial population; (e) Without Shortest-Path Crossover (w/o Crossover): In this method, we ablate the crossover operation. All offspring are generated through neighborhood mutation operations. To maintain fair experimental conditions, we select the top 20 architectures in each cycle, with each architecture generating 128 offspring through neighborhood mutation, resulting in a total of 2560 offspring. Finally, we select the top 50 offspring to supplement the initial population; (f) Regularized Evolutionary Algorithm (REA) ( We use REA as a benchmark for the evolutionary algorithm.
30

---

Following its experimental settings, we select the best architecture for mutation operations in each cycle. We define the mutation operation randomly changing the operation type and edge object of one operation edge. Like the other methods, we generate 2560 offspring in each cycle and select the top 50 offspring to supplement the initial population.
100
1000
10000
60000
Tou Agarro
140 40000
Anchitectures
PO-NAS
wio Evolution
+
N
wio Mutation
Crossover -+- REA
Comparison between different evolution methods on CIFAR-I0 on DARTS regarding the number of searched architectures. Note that all methods are reported as the 10-epochs test accuracy with the mean and standard error of 10 independent searches.
Figure 13:
Figure 13 presents experimental results. When the initial population size is relatively small, all evolutionary algorithms show significant performance improvements compared to the method without an evolutionary algorithm. Among them, the evolutionary algorithm with crossover operations performs particularly well, outperforming methods that generate offisring solely through mutation operations. This highlights the potential of crossover operations exploring unknown search spaces. The performance of PO-NAS is very close to that of the method without the exploration weight N. Although the exploration weight N sacrifices some stability, achieves a higher search upper limit. When the initial population size reaches certain scale, the performance gap between different methods further narrows. this point, the initial population is sufficiently diverse, covering wide range of architectures. Therefore, exploring the local area through mutation operations alone can still yield good results. Among them, the method with only crossover operations (w/o Mutation) achieves the highest performance upper limit compared to other methods. At this time, PO-NAS does not show a clear advantage. Nevertheless, the complete PO-NAS evolutionary algorithm consistently achieves stable and excellent performance across different initial population sizes, without being limited by the initial population size.
We conducted experiments to investigate the impact of the evolution starting cycle (T.) and the number of parents (N (NParr) and offspring (No11) per cycle on the performance of the evolution method. The results in Tables 14 and Tables 15 demonstrate clear negative correlation between Ta and final performance, as accuracy consistently declines when T. exceeds 10. This indicates the necessity of reserving a certain search budget for the evolution method to ensure adequate exploration. In terms of population size, larger number of Nofs exhibits weakly positive correlation with accuracy, suggesting that increasing offspring diversity is beneficial. In contrast, Nper shows less systematic influence. Thus, the optimal strategy should balance the allocation of Nofs and T. within the given computational budget.
31

---

Table 14: Comparison between different Te of evolution method on CUFAR-10CIIFAR-OO 100 on DARTS. Note that all methods are reported the 10-epochs test accuracy with the mean and standard error of 10 independent searches.
Accuracy (%)
DateSet\T.
0
5
10
CIFAR-10
82.69
82.71 61.26:100.1
82.63-10.33
CIFAR-100
61.31+0.28
61.19-10.27
15
20
25
CIFAR-10
82.48 +0.42
82.23++0228
81.84++0.11
CIFAR-100
60.87+0.44
60.36++0.57
59.27+00377
Table 15: Comparison of evolution method with different parameters (X-axis: Number offspring per epoch (Notr). Y-axis: Number of parent per epoch (Npar)) CIFAR-IO/CIFAR 100 DARTS. Note that all methods are reported as the 10-epochs test accuracy with the mean and standard of 10 independent searches.
Accuracy (%)
N NNoff
640
1280
2560
5120
CIFAR-10
82.26-+0.28
82.43-+0036
82.43+1044 82.47+0.36
10
0..5
82.48
82.52+
82.61+0.26
20
82.39-10.4
82.40+0.37
82.59-0.26
82.66110.32
40
82.46
82.5110.21
82.65
82.6410.22
CIFAR- 100
61.09:1.21
61.08+0.36
61.21
61.1440.40
10
61.00=#044
61.08
61.17=0.29
61.23+00.0
61.07=0.12
61.19:0.24
61.11=0.31
61.26+:0.26
40
61.16.10.33
61.13:1.22
61.23
61.19
32

---

Scene
Tel
Object
0.5
Jigsuw
1500
Layout
0.64
Segment
415
240
Normal
Autocncoder
000
Mumber Searche Archilcturos
+ TONNASuuo Ban) ZuBi +Bh
REA - SENSFORCT
IINAS AVG BEST - OPTMMA
Figure 14: Comparison of various NAS algorithms in TransNAS- -EEAAA---ee-----e Bewch-110-micee regarding the number searched architectures. The results for RoBoT, HNAS, and PO-NAS are reported as the mean and standard deviation over 10 runs, while REA, RS, and REINFORCE are evaluated over 50 runs. "AVG" and "BEST" represent the average value of metrics and the best individual metrics.
33

---

Scene
Ronbre
200-
Object
You Mrazing
Jigsaw
21.0
Soal abaa
Layout
043
Segment
34.00
32.00
Normal
2
Yorkeaing
Autoencoder
6.25
SENNTORCE INAS 180 5057 OPTIMAL
Figure 15: Comparison of various NAS algorithms in TransNAS-Benchh 101-maero regarding the number searched architectures. The results for RoBoT, HNAS, and PO-NAS are reported as the mean and standard deviation over 10 runs, while REA, RS, and REINFORCE are evaluated over 50 runs. "AVG" and "BEST" represent the average value of metrics and the best individual metrics.
34

---

sep connect
2
canw,
(_0k3
220
se0 505
C.(4)
max pow. 3x3
oony 3x3
(k-2)
Mkp connect
s00 com. 3x3
3vg_pool__ 3xd
up comy.
2Pp_02TM__555
( Normal Cell
(b) Reduction Cell
Figure 16: DARTS cell architecture found by PO-NAS on CIFAR- 10 dataset with model size 3.85 MB.
dil 3x3
c. (k-2)
3K3
evg_podl__ 333
sep comy,
(K-1)
sep_conw 5x5
max pcol_3x2 3x2
avg 3x3
303
C.3k)
2ep_corw_5xx
ck(k)
(k-2)
(
sep cony. Ex5
sep, cony 5x5
dil cony 3x3
sep. con..coo....o 5x5
eppcomy 5W5
scp_coTV _x3
(b) Normal Cell
(b) Reduction Cell
Figure 17: DARTS cell architecture found by PO-NAS on CIFAR- 100 dataset with model size 4.22 MB.
ve0_conw55ts 5
scp cony 5xS
skip connect
c O-13
mrg_pco__33
sep comV 5x5"
922. www.oor..... 3x)
snn .ow. 3x3
scp. conv SxS
CanV 5x5
c (x22
B00_WWWWS555 5x5
conv. 5x5
C_4k)
(k2)
conv 5x5
sep conv
conv. 5x5
Normal Coll
Reduction Cell
Figure 18: DARTS cell architecture found by PO-NAS on ImageNet dataset with model size 6.25 MB.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our proposed methods and the paper's contributions have been introduced in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In our analysis of the results from various experiments, we have discussed the limitations of our approach, such as the stability limitation the surrogate model.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
36

---

Answer: [NA]
Justification: This paper does not contain any mathematical proofs, only mathematical demonstrations of relevant concepts.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We conduct experiments using public benchmarks, such as 101, NAS-Bench-201 and DARTS, All hyperparameters and implementation details can be found Appendix B and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
37

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have provided comprehensive experimental details in both the paper and supplementary materials. Our code has been made publicly available at https:// anonymous Sesppennsccennce///---A....
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
hot.eesuuubbsssoooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
guides/ CodeeSumieee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided all the hyperparameters for our experiments, including those used for training the models in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We use common comparison metrics and explain any specialized ones. The experimental results are obtained from multiple independent runs and reported the mean and standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
38

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the type and number of GPUs used, and recorded the time consumed by the entire algorithm in related experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: peattss:/neerrigee http://ruripssccccllllc////// cc//public//thhccsssilele11111
Answer: [Yes]
Justification:
We have read and follow nips rules.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
39

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have fully cited the public benchmarks used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
40

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
41

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this paper does not involve LLMs as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:e///eerrrppccc///offeereeee hctp:///eeeeeeeece////////.... for what should or should not be described.
42


Reviews:
[{"review_id": "SWWBjBIS9P", "reviewer": "Reviewer_SnuL", "summary": "PO-NAS is a Neural Architecture Search (NAS) algorithm that improves the accuracy and efficiency of training-free NAS methods. Unlike previous approaches that apply a fixed combination of training-free metrics across the whole search space, PO-NAS dynamically optimizes metric combinations for each individual architecture using limited real training data. It integrates multiple metrics, avoids benchmark dependency, and incorporates an evolutionary search strategy guided by a surrogate model. The method effectively balances the speed of training-free NAS with the reliability of training-based evaluations, achieving strong performance across various tasks.", "questions": "as my concerns", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed PO-NAS introduces a novel method that dynamically weights existing zero-cost estimates.\n2. Through a well-crafted system design, PO-NAS significantly outperforms the performance of individual zero-cost estimates.\n3. The paper is clearly written and easy to follow.\n\nMinor Concerns:\n1. The use of both ELU and GELU activation functions appears inconsistent. Is there a specific reason for this combination?\n2. I am also curious whether different permutations or combinations of zero-cost estimates have any impact on performance.\n3. It would be valuable to discuss the relationships between different zero-cost estimates, for example, whether SynFlow and SNIP might be complementary. Such analysis could further enhance the contribution of the work.\n\nComment:\nOverall, this paper is well-written, with a clear motivation and strong experimental results. Discussing the potential relationships among different zero-cost estimates could further strengthen the contribution. So I recommend borderline acceptance.", "was_revised": true, "final_justification": "The authors' responses are reasonable to me, and as a result, I have decided to raise my evaluation to Accept.", "rebuttal": "We sincerely thank you for the time and effort you have invested in reviewing our paper. Your comments and questions are very insightful and highly valuable for us to enhance the paper. Your questions are addressed point-by-point in the following.\n\n**Minor Concerns**\n\n> The use of both ELU and GELU activation functions appears inconsistent. Is there a specific reason for this combination?\n\nThe hybrid use of ELU and GELU is a targeted design based on the distinct requirements for processing topological features at different network levels:\n\n- The application of ELU after the GAT layer is motivated by the characteristics of node connectivity relationships in the topological structure. The topological information processed by GAT includes connection objects and operation types between nodes, where certain operation connections may negatively correlate with prediction outcomes and thus require negative-valued features for accurate representation. ELU's smooth negative region property $(a(e^{x} - 1))$ preserves these prediction-significant negative-valued features while preventing the information loss that would occur with ReLU's zero-truncation, which is crucial for accurately modeling the complex relationships between topological structures and training-free metrics.\n\n- The use of GELU after the linear layer serves to enhance the expressive power of high-level topological features. The linearly transformed topological features demand more refined nonlinear processing, where GELU's probabilistic gating mechanism $(x \\cdot \\Phi(x))$better captures the impact of complex interaction patterns between nodes on prediction metrics, while its smooth properties facilitate hierarchical feature propagation. This combination ensures both complete representation of low-level topological features and improved abstraction of high-level features, representing a carefully considered design solution.\n\nThis insightful question has helped us better articulate the rationale behind our activation function choices. We will explicitly address this point in the revised manuscript to improve the interpretability of our design methodology.\n\n> I am also curious whether different permutations or combinations of zero-cost estimates have any impact on performance.\n\nThank you for raising this insightful question.  In conventional training-free NAS, the permutation or combination of training-free metrics inevitably influences the final ranking quality, because each metric is usually tailored to specific scenarios (e.g., SWAP[2] is only effective for ReLU-based networks). Consequently, a fixed combination can exhibit large performance gaps across different tasks. PO-NAS addresses this limitation through its learnable linear weighting: during the search, the learnable weight vector $w_\\mathcal{A}$ dynamically assigns weights to each training-free metric, which is equivalent to implicitly constructing and evaluating a new metric combination at every step without enumerating all possible explicit combinations.\n\nIf you were referring to the six training-free metrics adopted in the paper, we must clarify that, to ensure a fair comparison with baselines, we strictly followed the set specified in Appendix B.3 (grad_norm, snip, grasp, fisher, synflow, and jacob_cov from [1]), guaranteeing that any performance gain originates from the PO-NAS algorithm itself. \n\nNevertheless, incorporating recently proposed training-free metrics (such as SWAP [2] or AZ-NAS [3]) can further improve the average performance of PO-NAS, and we will include these results in the revised manuscript to demonstrate the framework’s extensibility. Finally, as the training budget increases, PO-NAS can explore larger pools of training-free metrics and their combinations, thereby yielding additional performance gains.\n\n> It would be valuable to discuss the relationships between different zero-cost estimates, for example, whether SynFlow and SNIP might be complementary. Such analysis could further enhance the contribution of the work.\n\nWe fully acknowledge the importance of analyzing the complementary relationships among training-free metrics. \n\nDifferent metrics (e.g., NASWOT[4] based on ​​binary activation patterns​​, SNIP[5] relying on ​​gradient distribution​​, and ZenNAS[6] analyzing ​​linear regions​​) exhibit inherent complementarity due to their distinct underlying principles. Combining these metrics enables a comprehensive evaluation of architectural characteristics. \n\nHowever, these relationships demonstrate significant volatility: cross-task correlations vary drastically, and fluctuations occur across search spaces and even individual architectures within the same space. This non-generalizable volatility constitutes the core motivation for PO-NAS's dynamic metric weighting mechanism​​—which adaptively adjusts weights to meet diverse task and architectural demands while ​​consistently achieving robust performance​​. We will strengthen this argument by adding quantitative experiments in the revised manuscript.\n\n---\n[1] Zero-Cost Proxies for Lightweight NAS, ICLR 2021.\n\n[2] Peng, Y., A. Song, H. M. Fayek, et al. Swap-nas: Sample-wise activation patterns for ultra-fast nas. In ICLR. 2024.\n\n[3] Lee, J., B. Ham. Az-nas: Assembling zero-cost proxies for network architecture search. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5893–5903. IEEE Computer Society, 2024.\n\n[4] Mellor J, Turner J, Storkey A, et al. Neural architecture search without training[C]//International conference on machine learning. PMLR, 2021: 7588-7598.\n\n[5] Lee N, Ajanthan T, Torr P H S. Snip: Single-shot network pruning based on connection sensitivity[J]. arXiv preprint arXiv:1810.02340, 2018.\n\n[6] Lin M, Wang P, Sun Z, et al. Zen-nas: A zero-shot nas for high-performance image recognition[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 347-356."}, {"review_id": "RO5jDpSeun", "reviewer": "Reviewer_8Jw8", "summary": "The paper notes that individual training-free metrics capture partial architectural features (for e.g. Parameter Count capture just computation, not much about connectivity, etc.). They claim that  combining training-free metrics and adjusting to sensitivities of different architectures to specific metrics is needed. PO-NAS is proposed, combining multiple training-free metrics as scores, and combining them smartly. They further use an EA with predictions from surrogate model to search in large search spaces.", "questions": "Aside from evaluations missing relevant work from prior literature, I do not have major concerns, paper is well written, well motivated and does not raise any major red flags.", "limitations": "Discussed in weaknesses above, just needs more evaluation.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- Good idea to combine multiple training-free metrics, train an arch-encoder to predict these metrics and use that for broader NAS. \n\nWeaknesses:\n- Combining such metrics has already been studied in MultiPredict, FLAN, TA-GATES. Comparing with such methods is important, as they report results on NB201 as well. \n- Comparing encoders such as Arch2Vec and CATE might be insightful, it may especially help at the attention-network where distinguishing architectures is important.\n- On macro-spaces, comparing with GENNAPE or relevant discussion is important. \n\n[CATE] Yan, Shen, et al. ‘CATE: Computation-Aware Neural Architecture Encoding with Transformers’. arXiv [Cs.LG], 2021, http://arxiv.org/abs/2102.07108. arXiv.\n\n[Arch2Vec] Yan, Shen, et al. ‘Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?’ NeurIPS, 2020.\n\n[MultiPredict] Akhauri, Yash, and Mohamed S. Abdelfattah. ‘Multi-Predict: Few Shot Predictors For Efficient Neural Architecture Search’. arXiv [Cs.LG], 2023, http://arxiv.org/abs/2306.02459. arXiv.\n\n[TAGATES] Ning, Xuefei, et al. ‘TAGATES: An Encoding Scheme for Neural Network Architectures’. Advances in Neural Information Processing Systems, edited by S. Koyejo et al., vol. 35, Curran Associates, Inc., 2022, pp. 32325–32339, https://proceedings.neurips.cc/paper_files/paper/2022/file/d0ac28b79816b51124fcc804b2496a36-Paper-Conference.pdf.\n\n[FLAN] Akhauri, Yash, and Mohamed S. Abdelfattah. ‘Encodings for Prediction-Based Neural Architecture Search’. arXiv [Cs.LG], 2024, http://arxiv.org/abs/2403.02484. arXiv.\n\n[GENNAPE] Mills, Keith G., et al. ‘GENNAPE: Towards Generalized Neural Architecture Performance Estimators’. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 8, Association for the Advancement of Artificial Intelligence (AAAI), June 2023, pp. 9190–9199, https://doi.org/10.1609/aaai.v37i8.26102.", "was_revised": true, "final_justification": "The authors provided a good rebuttal, with several of my concerns addressed. The only reason I am hesitating to assign a `5: Accept` is because I have reservations about \"... moderate-to-high impact on more than one ...\" that is attached with the rating, as several components already existed in NAS literature in similar forms, but this is a solid paper.", "rebuttal": "We appreciate your close reading and constructive feedback. The points you raised are central to our work, and our answers to each follow in the next sections.\n\n**Weaknesses**\n\n> Combining such metrics has already been studied in MultiPredict, FLAN, TA-GATES. Comparing with such methods is important, as they report results on NB201 as well. \n\nThank you for the series of papers you recommended. We have carefully read them and confirmed that their experimental settings are highly comparable to PO-NAS and are of great reference value. \n\nTo this end, we selected those whose experimental conditions, datasets, and reported metrics most closely match PO-NAS, and we have supplemented direct comparisons on both NAS-Bench-201 (Table 1, Table2) and DARTS  (Table 3). The results demonstrate that PO-NAS effectively couples real training with training-free metrics, achieving superior performance at markedly reduced search costs.\n\nThese additional results will help readers gain a more comprehensive understanding of PO-NAS’s true performance and unique advantages, and we will also cite these papers in the revised manuscript for detailed comparative analysis.\n\nTable 1: Comparison of the number of architectures that must be trained for GENNAPE, FLAN$^T_{CAZ}$​, and PO-NAS to attain the target test accuracy on NAS-Bench-201.\n| Algorithm          | GENNAPE[1] | FLAN$^T_{CAZ}$[2] | PO-NAS |\n|---------------|----------|----------|----------|\n| Trained models| 50     | 32     | 20     | \n| Test Accuracy (%) | 93.27     | 93.30     | 94.12     | \n\nTable 2: The mean and standard deviation of the test accuracy(%) of BOHB, arch2vec and PO-NAS under three datasets on NAS-Bench-201.\n| Algorithm          | CIFAR-10 | CIFAR-100 | ImageNet-16-120 | Cost<br>(GPU Seconds) |\n|---------------|----------|----------|----------|----------|\n| BOHB[3]| 93.61±0.52     | 72.37±0.90     | 45.26±0.83     |  12000     | \n| arch2vec-RL[4] | 94.12±0.42     | 73.15±0.78     | 46.16±0.38     |  12000     | \n| arch2vec-BO | 94.18±0.24     | 73.37±0.30     | 46.27±0.37     |  12000     | \n| PO-NAS | 94.12±0.22      | 73.51±0.00     | 46.71±0.12      |  3162     | \n\nTable 3: Comparative results of CATE, arch2vec, and PO-NAS on DARTS Search Space with CIFAR-10\n| Algorithm          | Test Error (%)  | Params (M)  | Search Cost| \n|---------------|----------|----------|----------|\n| CATE-DNGO-LS (small budget)[5]| 2.55 ± 0.08     | 3.5     | 3.3 (GPU days)     |\n| CATE-DNGO-LS (large budget) | 2.46 ± 0.05     | 4.1     | 10.3 (GPU days)    |\n| arch2vec-RL | 2.65±0.05     | 3.3     | 8.3 (GPU days)    |\n| arch2vec-BO | 2.56±0.05     | 3.6     | 9.2 (GPU days)    |\n| PO-NAS | 2.52±0.03      | 3.8     | 3.9 (GPU hours)      |\n\n\n> Comparing encoders such as Arch2Vec and CATE might be insightful, it may especially help at the attention-network where distinguishing architectures is important.\n\nThank you for this suggestion. We recognize that the original manuscript indeed lacks comparative experiments on encoders, thereby limiting a complete justification of PO-NAS’s effectiveness. The encoder in PO-NAS is supervised by training-free metrics, and validating the reliability of this supervisory signal is essential.\n\nWe have carefully examined the encoder papers you recommended; among them, Arch2Vec’s experimental setup on NAS-Bench-201 is most comparable to PO-NAS. To preserve Arch2Vec’s original performance, we retain its official training protocol: supervision by actual architecture performance and obtain its fixed embeddings, and then feed these embeddings into the attention network for architecture discrimination, followed by a direct comparison with PO-NAS. Preliminary results are consolidated in Table 4.\n\nThe results show that replacing PO-NAS’s encoder with existing SOTA architecture encoders yields only marginal improvements on certain datasets, while overall performance remains nearly identical. This confirms the excellence of PO-NAS’s encoder and further demonstrates that learning architecture representations via training-free metrics is both feasible and highly efficient, substantially reducing the cost of acquiring topological information.\n\nTable 4: Impact of Different Encoders and Supervisory Signals on PO-NAS Performance (reported by test accuracy (%) on NAS-Bench-201).\n\n| Encoder          | CIFAR-10 | CIFAR-100 | ImageNet-16-120 | Supervisory|\n|---------------|----------|----------|----------|----------| \n| arch2vec | 94.25±0.18     | 73.51±0.00     | 46.49±0.37     |  short-term training (10 epochs)     | \n| PO-NAS | 94.12±0.22      | 73.51±0.00     | 46.71±0.12      |  train-free metrics     | \n\nOnce again, we appreciate your valuable comment. We will incorporate the above comparison and cite the relevant work in the revised manuscript, while extending the comparison to additional encoders.\n\n> On macro-spaces, comparing with GENNAPE or relevant discussion is important.\n\nWe concur with this point. As noted in our first response, we have already included a comparison with GENNAPE[1], and we will further enrich the revised manuscript with theoretical macro-space analyses between PO-NAS and additional algorithms, accompanied by full citations, to provide readers with a deeper understanding of our contribution.\n\n**Paper Formatting Concerns**\n\n> None, figure captions could be made bigger if possible.\n\nThank you for your suggestion. We will address this issue in the revised manuscript.\n\n---\n\n[1] Mills, Keith G., et al. ‘GENNAPE: Towards Generalized Neural Architecture Performance Estimators’. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 8, Association for the Advancement of Artificial Intelligence (AAAI), June 2023, pp. 9190–9199.\n\n[2] Akhauri, Yash, and Mohamed S. Abdelfattah. ‘Encodings for Prediction-Based Neural Architecture Search’. arXiv [Cs.LG], 2024, arXiv.\n\n[3] Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In ICML, 2018.\n\n[4] Yan, Shen, et al. ‘Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?’ NeurIPS, 2020.\n\n[5] Yan, Shen, et al. ‘CATE: Computation-Aware Neural Architecture Encoding with Transformers’. arXiv [Cs.LG], 2021, arXiv."}, {"review_id": "eDjavnmqDE", "reviewer": "Reviewer_G3GX", "summary": "This paper proposes that prior training-free NAS metrics capture only part of a network’s merit and often fail to generalize across different tasks. Simply combining multiple metrics into a single score with fixed weights also can not transfer well. The authors propose PO-NAS, a NAS algorithm that dynamically learns a tailored combination of training-free metrics for each candidate architecture with a weight vector. Then the score is used as a surrogate model to select the optimal architecture. PO-NAS proposes a four step NAS method to combine train-free and training NAS, comprising initialization, a pre-train stage, a Bayesian stage and an evolution stage. Experiments on multiple NAS benchmarks demonstrate that PO-NAS achieves state-of-the-art performance compared with conventional NAS and other hybrid NAS methods.", "questions": "- Could you clarify how the per-architecture weight vector $w_A$ is actually optimized in practice? You mention employing Bayesian Optimization for the surrogate – does this mean a Gaussian Process is tuning the weights for each architecture sequentially? \n- PO-NAS relies on training architectures for a small number of epochs (10 on CIFAR, 3 on ImageNet) to get the true performance feedback used for weight updates. How well does this short-term performance correlate with the final performance of architectures? \n- Ablation on Metrics Used: You integrate six training-free metrics in the surrogate. Did you investigate how important each one is? For example, if you remove a metric or down-weight it, does performance drop noticeably? Conversely, if a new metric is added, is the framework robust to incorporate it? \n- Hyperparameter Sensitivity: The paper introduces multiple components to construct the NAS method, and it is necessary to clarify the sensitivity of the final performance to certain key hyperparameters. How sensitive are the results to the choice of $T_e$ and the population size or number of offspring?\n- When evaluating the time overhead, will the computational costs of introduced components—such as the architecture encoder and Bayesian surrogate model training—be accounted for?", "limitations": "The limitations have been clarified in the weaknesses and questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- The paper is generally well-written and organized. It clearly claims its motivation and core ideas.\n- The authors have included a detailed appendix with further experimental details and provided the source codes.\n- The experimental evaluation is comprehensive. The authors test PO-NAS on 20 different tasks across three NAS benchmarks, including a variety of domains (image classification, scene recognition, segmentation, etc. in TransNAS-Bench-101).\n- The proposed method yields state-of-the-art or near-optimal performance on multiple benchmarks.\n\n**Weaknesses**:\n- The proposed approach is quite elaborate, involving multiple components and phases: a graph neural network encoder, a metric predictor, per-architecture weight optimization via a surrogate model, Bayesian optimization guiding architecture selection, and an evolutionary algorithm with custom crossover/mutation. This complexity might make the method hard to understand and tune.\n- The complex pipeline proposed in this paper introduces a significant number of additional hyperparameters, but no discussion is provided on how sensitive the final outcome is to these hyperparameters.\n- Although the results are strong, in some cases the margins over prior art are not very large. The paper would be stronger if it provided a deeper analysis of current results.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your meticulous evaluation. We address your concerns as follows:\n\n**Weaknesses**\n\n> The proposed approach is quite elaborate, involving multiple components and phases: a graph neural network encoder, a metric predictor, per-architecture weight optimization via a surrogate model, Bayesian optimization guiding architecture selection, and an evolutionary algorithm with custom crossover/mutation. This complexity might make the method hard to understand and tune.\n\nYou have precisely identified the higher complexity of PO-NAS. PO-NAS contains multiple components, but each module is designed to address core challenges in Training-Free NAS:\n\n- Architecture Encoder​​: Introduces architectural topological information to resolve stability issues of training-free metrics across different architectures within the same search space.\n- Metric Predictor​​: Serves as a supervisory signal for the architecture encoder and avoids extensive metric calculations for offspring in evolutionary algorithms.\n- Transformer Surrogate Model​​: Integrates advantages of real training and training-free metrics.\n- ​​Evolutionary Algorithm​​: Combined with preceding components, significantly improves generalizability of existing training-free NAS in large search spaces.\n\nAlthough PO-NAS has complexity, it is flexibly adjustable and easily implementable. In small search spaces (e.g., NAS-Bench-201), benefiting from the high-speed computation of training-free metrics, PO-NAS can directly enumerate all architectures without evolutionary algorithms, greatly reducing complexity. Nevertheless, complexity remains a non-negligible issue, and we will further optimize this limitation in future work.\n\n> The complex pipeline proposed in this paper introduces a significant number of additional hyperparameters, but no discussion is provided on how sensitive the final outcome is to these hyperparameters.\n\n> Hyperparameter Sensitivity: The paper introduces multiple components to construct the NAS method, and it is necessary to clarify the sensitivity of the final performance to certain key hyperparameters. How sensitive are the results to the choice of $T_{e}$ and the population size or number of offspring?\n\nThank you for raising the hyperparameter sensitivity concern. \n\n- We must clarify that: While the training of the surrogate model and architectural encoder contains many hyperparameters (e.g., learning rate, batch size), these parameters are constrained by the ​​loss threshold mechanism​​ during training. Moreover, the surrogate model training time constitutes ​​a negligible portion of PO-NAS's total computational cost​​, making its fluctuations in overall time and performance negligible. The hyperparameters are exhaustively documented in Appendix B.2 solely to ensure the reproducibility of PO-NAS; in practice, the vast majority of them have negligible impact on the final performance. For critical parameters (e.g., loss threshold, difference threshold), Appendix D.1 already provides comparative experiments and impact analyses.\n\n-  Additionally, regarding evolutionary algorithm parameters (e.g., $T_{e}$, parent count and offspring size), we fully acknowledge the lack of discussion in the paper and will supplement comparative experiments with results analysis for these hyperparameters in the revised version.\n\n> Although the results are strong, in some cases the margins over prior art are not very large. The paper would be stronger if it provided a deeper analysis of current results.\n\nWhile PO-NAS exhibits marginal performance improvements in some tasks, this precisely underscores its ​​cross-task robustness​​—across ​​20 differentiated tasks​​ spanning multiple search spaces, PO-NAS consistently ​​maintains stable performance gains which substantially remedies the shortcomings of existing training-free NAS​​. Due to space constraints, we were unable to demonstrate PO-NAS's robustness advantages over more comparable algorithms across diverse task scenarios. In the revised appendix, we will supplement comparative experiments with similar algorithms on TransNAS-bench-101 to better illustrate PO-NAS's core strengths.\n\n**Questions**\n\n> Could you clarify how the per-architecture weight vector $w_\\mathcal{A}$  is actually optimized in practice? You mention employing Bayesian Optimization for the surrogate – does this mean a Gaussian Process is tuning the weights for each architecture sequentially? \n\n- As shown in Table 1, we demonstrate the optimization effectiveness of the per-architecture weight vector $w_\\mathcal{A}$ by tracking how the global correlation between architectural evaluation scores and true performance on NAS-Bench-201 evolves across search epochs. While we cannot guarantee theoretical optimality for every individual architecture's weights, the significant improvement in overall correlation indicates that our weight allocation strategy performs effectively in most cases, which we hope addresses your concern. \n\n- Regarding the implementation details of Bayesian optimization, PO-NAS employs not Gaussian processes but rather the Transformer-based surrogate model proposed in Section 2.1 to dynamically generate per-architecture weight vectors. Serving dual roles as both the surrogate model and acquisition function in Bayesian optimization, this model learns to adaptively determine optimal weight allocation strategies for distinct architectural features through end-to-end training.\n\nTable 1: PO-NAS on NAS-Bench-201 Kendall rank correlation coefficient versus search epoch reported as mean ± standard deviation over 10 runs.\n| Dataset          | 10 epochs | 20 epochs | 30 epochs | 40 epochs | 50 epochs |\n|---------------|----------|----------|----------|----------|----------|\n| CIFAR-10| 0.276±0.062     | 0.537±0.036     | 0.584±0.022     |  0.611±0.020     |   0.641±0.017     | \n| CIFAR-100 | 0.311±0.051     | 0.617±0.041     | 0.632±0.019     |  0.639±0.019     |   0.647±0.012     | \n| ImageNet16-120 | 0.228±0.082     | 0.494±0.052     | 0.541±0.039     |  0.574±0.035     |   0.612±0.026     | \n\n\n> PO-NAS relies on training architectures for a small number of epochs (10 on CIFAR, 3 on ImageNet) to get the true performance feedback used for weight updates. How well does this short-term performance correlate with the final performance of architectures? \n\nWe fully acknowledge the validity of your insightful comments regarding the correlation between short-term training and final performance, which indeed constitutes a key performance bottleneck for PO-NAS, as thoroughly discussed in Appendix C.4. \n\nHowever, this design stems from our careful consideration of the trade-off between computational efficiency and generalization capability across different search spaces: PO-NAS incorporates both topological structures and training-free metrics rather than relying solely on short-term performance; while in contrast to purely training-free approaches, the true target-space training is crucial for achieving cross-space generalization. The consistently superior performance of PO-NAS across multiple tasks demonstrates that, although the short-term training correlation presents a theoretical limitation, research findings confirm it still maintains a strong correlation with the architecture's final performance [1, 2]. This deliberate trade-off enables PO-NAS to achieve better performance while maintaining both efficiency and generalizability.\n\n> Ablation on Metrics Used: You integrate six training-free metrics in the surrogate. Did you investigate how important each one is? For example, if you remove a metric or down-weight it, does performance drop noticeably? Conversely, if a new metric is added, is the framework robust to incorporate it? \n\n- Regarding metric importance: Our preliminary experiments across multiple datasets revealed significant variability in each metric's impact - the contribution of individual metrics to predictor loss and surrogate model performance fluctuates considerably across different datasets, search spaces, and even among architectures within the same search space. This observation motivated the core design of PO-NAS, as we believe dynamic adaptation of metric importance during search is crucial for achieving robust generalization in training-free NAS.\n\n- Concerning metric quantity: As demonstrated in Appendix D.1, the quantity  of metrics substantially influence PO-NAS's performance. We provide guidelines for adjusting metric quantity based on available computational budget.\n\n- About metric extensibility: To ensure fair comparison with baselines, we deliberately adopted identical metric sets (Appendix B.3) (i.e., grad_norm, snip, grasp, fisher, synflow, and jacob_cov from [3]), isolating the performance improvement to our algorithmic contribution rather than metric advantage. However, preliminary results with enhanced metrics show PO-NAS can achieve further improvements, which we will include in the revised manuscript to demonstrate the framework's full potential.\n\n> When evaluating the time overhead, will the computational costs of introduced components—such as the architecture encoder and Bayesian surrogate model training—be accounted for?\n\nWe have thoroughly documented PO-NAS's time evaluation protocol in Appendix B.2, where we account for the computational costs of all components in our time assessment. \n\nIt is particularly noteworthy that the overhead associated with both the architecture encoder and Bayesian surrogate model training represents only a negligible fraction when compared to the predominant computational expense of short-term architecture training.\n\n---\n\n[1] Mellor J, Turner J, Storkey A, et al. Neural architecture search without training[C]//International conference on machine learning. PMLR, 2021: 7588-7598.\n\n[2] Dong X, Yang Y. Nas-bench-201: Extending the scope of reproducible neural architecture search[J]. arXiv preprint arXiv:2001.00326, 2020.\n\n[3] Zero-Cost Proxies for Lightweight NAS, ICLR 2021."}, {"review_id": "ga9pXUNnd2", "reviewer": "Reviewer_KFhC", "summary": "The authors propose a novel method called PO-NAS, which integrates training-free metrics and individually optimizes metric combinations for each architecture. The approach significantly enhances search efficiency and performance by incorporating evolutionary algorithms.", "questions": "1. Do multiple runs with different seeds yield different architectures? Approximately how many attempts are needed to obtain the optimal architecture's best results?  \n2. Could the authors provide either intuitive explanations or rigorous theoretical proofs to demonstrate how these metrics contribute to the search process?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:  \n1. The authors integrate multiple metrics and optimize their combinations through limited training to obtain relatively effective weighting schemes.  \n2. They employ evolutionary algorithms combined with trained metrics predictors to extensively explore the search space.  \n3. The superiority and scalability of PO-NAS are validated across multiple datasets.   \n  \nWeaknesses:  \n1. The authors fail to explain the specific meanings of each metric in Table 4, making certain details of the paper difficult to comprehend.   \n2. While the core contribution focuses on utilizing multiple metrics and searching for optimal weightings through training, there's no discussion about why these metrics make the architecture search effective. This fundamentally limits its extensibility. The authors should instead explore the interpretability of these metrics (similar to works like DARTS-IM [1]), which would be more meaningful for NAS development.  \n  \n[1].Zhang M, Huang W, Yang B. Interpreting operation selection in differentiable architecture search: A perspective from influence-directed explanations[J]. Advances in Neural Information Processing Systems, 2022, 35: 31902-31914.", "was_revised": true, "final_justification": "please see response", "rebuttal": "We deeply value your thorough assessment and insightful feedback, which precisely target the key facets of our method; the corresponding clarifications are provided in the succeeding sections.\n\n**Weaknesses**\n\n> The authors fail to explain the specific meanings of each metric in Table 4, making certain details of the paper difficult to comprehend.\n\nWe sincerely apologize for not providing detailed explanations of each metric's specific meaning in Table 4, and we truly appreciate you highlighting this issue which is indeed crucial for our paper's readability. In response to your valuable feedback, we will supplement the fundamental definitions of each training-free metric along with relevant literature references in the appendix of the revised manuscript. \n\nAdditionally, regarding the specific implementation methods for operation subset classification, we have already provided detailed explanations in Appendix B.3 and have added corresponding appendix index markers in the table title. We believe these supplementary explanations will effectively enhance the paper's readability.\n\n> While the core contribution focuses on utilizing multiple metrics and searching for optimal weightings through training, there's no discussion about why these metrics make the architecture search effective. This fundamentally limits its extensibility. The authors should instead explore the interpretability of these metrics (similar to works like DARTS-IM [1]), which would be more meaningful for NAS development.\n\nWe appreciate your insightful comments, which have accurately identified a critical theoretical gap in our work. While PO-NAS has demonstrated promising results through dynamic combinations of multiple training-free metrics, we acknowledge the need to strengthen the theoretical foundation explaining why these particular metrics are effective. \n\nWe fully recognize the value of the theoretically-driven analysis presented in DARTS-IM [1], particularly their seminal findings regarding the relationship between second-order information and architectural performance. Additionally, we found the sensitivity analysis framework of DARTS-IM to be indispensable for understanding how training-free metrics influence the ultimate performance of PO-NAS. It will help readers better understand our approach. Consequently, in the revised manuscript, we will systematically elaborate on the theoretical principles underlying each employed training-free metric. This will include a rigorous analysis of the architectural characteristics captured by individual metrics and the application of sensitivity analysis methods from DARTS-IM to quantitatively assess metric complementarity. These enhancements will substantially improve the theoretical depth and extensibility of PO-NAS, while appropriately acknowledging the foundational theoretical contributions of DARTS-IM through comprehensive citations at relevant junctures.\n\n**Questions**\n\n> Do multiple runs with different seeds yield different architectures? Approximately how many attempts are needed to obtain the optimal architecture's best results? \n\nWe appreciate your concern regarding the stability of PO-NAS, especially the potential for performance variability across different task settings. \n\nTo fully address this, every experiment on TransNAS-Bench-101 and NAS-Bench-201 is repeated with multiple independent runs, each initialized with a distinct random seed (for instance, seeds 0–9 are cycled whenever ten or more replications are performed), and results are reported as mean ± standard deviation. In several search tasks, PO-NAS consistently identifies the theoretically optimal architecture, i.e., identical structures are recovered across runs. In larger search spaces such as DARTS, different seeds can yield non-identical architectures accompanied by minor performance fluctuations; nevertheless, the architectures discovered by PO-NAS retain competitive and stable performance across all evaluated datasets.\n\n> Could the authors provide either intuitive explanations or rigorous theoretical proofs to demonstrate how these metrics contribute to the search process?\n\nThank you for your incisive observation. We apologize for not explicitly clarifying the adopted metrics in the manuscript. \n\nIt should be emphasized that the core contribution of PO-NAS lies in the dynamic composition mechanism of training-free metrics and the investigation of their generalization capability, rather than in the theoretical novelty of any\n single metric. The interpretability of each metric and its mechanism for facilitating the NAS search process have already been rigorously proven in the foundational literature [2, 3, 4, 5]. \n\nWe fully agree on the necessity of strengthening theoretical exposition; therefore, we will supplement the revised manuscript with detailed descriptions of the precise meaning and theoretical basis of every training-free metric, citing the relevant seminal works. We hope these additions will help readers better understand our approach.\n\n---\n\n[1] Zhang M, Huang W, Yang B. Interpreting operation selection in differentiable architecture search: A perspective from influence-directed explanations[J]. Advances in Neural Information Processing Systems, 2022, 35: 31902-31914.\n\n[2] J. Turner, E.J. Crowley, M. O’Boyle, A. Storkey, G. Gray, BlockSwap: Fisher-guided block substitution for network compression on a budget, 2019, arXiv preprint arXiv:1906.04113.\n\n[3] N. Lee, T. Ajanthan, P.H. Torr, SNIP: Single-shot network pruning based on connection sensitivity, 2018, arXiv preprint arXiv:1810.02 340.\n\n[4] C. Wang, G. Zhang, R. Grosse, Picking winning tickets before training by preserving gradient flow, 2020, arXiv preprint arXiv:2 002.07376.\n\n[5] H. Tanaka, D. Kunin, D.L. Yamins, S. Ganguli, Pruning neural networks without any data by iteratively conserving synaptic flow 33 (2020) 6377–6389."}]

Author rebuttal:

Meta-review: The reviewers all acknowledged the interest of combining multiple metrics in a NAS framework, the effectiveness of the method, and the clarity of the paper. They nevertheless raised some questions regarding design choices for the approach and additional empirical evaluations. The authors' feedback addressed most of the reviewers' concerns, thus resulting in a consensus for acceptance. The AC nonetheless encourages the authors to incorporate their feedback in the camera-ready version of the paper.
Decision: Accept (poster)