Revolutionizing Training-Free NAS: Towards Efficient Automatic Proxy Discovery via Large Language Models
Authors: Haidong Kang
Lihong Lin
Hanling Wang
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Training-Free NAS
Automatic Proxy Discovery
Primary area: deep_learning

Abstract: The success of computer vision tasks is mainly attributed to the architectural design of neural networks. This highlights the need to automatically design high-performance architectures via Neural Architecture Search (NAS). To accelerate the search process, training-free NAS is proposed, which aims to search high-performance architectures at initialization via zero-cost proxies (ZCPs). However, existing zero-cost proxies heavily rely on manual design, which is often labor-intensive and requires extensive expert knowledge. In addition, these crafted proxies often suffer from poor correlation with final model performance and high computational complexity, severely limiting NAS efficiency in real-world applications. To address those issues, this paper proposes a novel Large Language Models (LLMs)-driven $\underline{A}$utomatic $\underline{P}$roxy $\underline{D}$iscovery ($\textbf{APD}$) framework, which revolutionizes the design paradigm of ZCPs by leveraging LLMs to automatically discover optimal ZCPs for Training-Free NAS. Moreover, we utilize actor-critic based reinforcement learning to optimize prompts, enabling to generate better ZCPs in the next generation. We conduct extensive experiments on mainstream NAS benchmarks, demonstrating APD excels in both performance and efficiency. Besides, we firmly believe that our APD will dramatically benefit the deep learning community through providing novel paradigm of design algorithms via LLMs.

Revolutionizing Training-Free NAS: Towards Efficient Automatic Proxy Discovery via Large Language Models
Haidong Kang Lihong Lin² Hanling Wang"
Northeesternn University, China Pengcheng Laboratory, China hdkangesture tumall, neu. edu. en, linlh@mails nwu.nuuu.cco en, wangh1038pc1.
Abstract
The success of computer vision tasks is mainly attributed to the architectural design of neural networks. This highlights the need to automatically design high- performance architectures via Neural Architecture Search (NAS). To accelerate the search process, training-free NAS is proposed, which aims search high- performance architectures at initialization via zero-cost proxies (ZCPs). However, existing zero-cost proxies heavily rely manual design, which is often labor- intensive and requires extensive expert knowledge. In addition, these crafted proxies often suffer from poor correlation with final model performance and high computational complexity, severely limiting NAS efficiency in real-world appli- cations. To address those issues, this paper proposes a novel Large Language Models (LLMs)-drivee Automatic Proxy Discovery (APD) framework, which revolutionizes the design paradigm of ZCPs by leveraging LLMs automatically discover optimal ZCPs for Training- Free NAS. Moreover, we utilize actor-critic based reinforcement learning to optimize prompts, enabling to generate better ZCPs in the next generation. We conduct extensive experiments on mainstream NAS benchmarks, demonstrating APD excels in both performance and efficiency. Besides, we firmly believe that our APD will dramatically benefit the deep learning community through providing novel paradigm of design algorithms via LLMs.
Introduction
Neural networks plays indispensable role in computer vision tasks due to its superior performance, which raises a trend to deploy neural networks (i.e., ResNet He et al.) (2016) on resource-ing -inturcce scenarios. However, conventional neural networks designed by human experts suffer from Out-Of- Memory (OOM) problems due to dramatically limited resources Xie et al. 2023)) Kang et al. [2025] Therefore, this highlights the need design lightweight architectures, liberating the handicraft neural networks from the OOM bottleneck.
Recently, Neural Architecture Search (NAS) Liu et al. [0188 Ye et al. [2022] Kang has emerged as a promising paradigm for its searching high-performance architectures in an automatic manner, while disrupting the conventional paradigm of manually designed architecture. Despite its potential,
NAS still suffers from a key bottleneck of huge computational budgets Ma et al.
2024 To tackle this, training- free NAS Mellor et al.) [2021] Abdelfattah et al. (2021). Wang al.
2020 Chen et al.)
2022], [ee and Ham 22024), 1,24),,1 Peng et al. [2024] is proposed to liberate NAS from the computational
bottleneck via the lens of without gradient descent. Essentially, the training-free NAS leverages ZCPs, predicting the accuracy ranking of architectures in training-free manner. Specifically, the 7CPs rely
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prompts
Input
Thought
Thought
Evaluate
Proxies
Controller
Roward
Code
Code
2
At arortsntionaaa
Sampling
Actor
Critic
(a) Manual Design
(b) Naive method
(c) Ours
Figure 1: A comparison of the designed way of ZCPs. (a) Manual design relies on expert knowledge. (b) A naive method via LLMs. (c) Our method proposes an LLM-driven APD framework to automatically discover optimal ZCPs for Training-Fre NAS.
JIGSAW
66.26
IMNET Are
0LO Ace
SCENE Aor
JISSAW Acc
54
SP
AUTOENC
C100 Ace
AUTOENC SSIM
GrxSP
ZanNAS
SWAP
AFflivrrs
Figure 2: Spearman (SP) ranking correlations & accuracies (Acc) of zero-cost proxies on NAS- Bench-201 (Left) and Trans-Bench- 101 (right).
on some statistical theoretical properties (i.e., the number of parameters, Neural Tangent Kernel) of neural architectures to assess their expressivity.
1.1 Challenges
Although these approaches have taken the first step towards ZCPs tailored for training-free NAS Haidong et al.] [2025), current ZCPs still suffer from several fatal drawbacks: (1) they require extensive
expert knowledge through hundreds of time-consuming trial-and-error processes (as shown in Fig.
7a and Table [. increasing the labor costs and making it hard design new ZCPs; (2) these crafted ZCPs incur poor correlation with the final accuracy of searched architecture (as shown in Fig. andd 3. making the users do not know why those ZCPs failed and hard to deploy real-world applications.
This is identified with ZiCO Li et al. [2023), which points out that the performance of simple proxies (i.e.. #Params, FLOPs) excels most of the crafted ZCPs. Those limitations highlight the need to rethink the design paradigm of AZPss
Our New Observation. Different from existing methods for training-free NAS, depicted in Fig. Db. and Fig. [e. we observe a new way to automatically design ZCPss via Large Language Models (LLMs) in this work, which can effectively address the aforementioned drawbacks by revolutionizing the traditional manual design manner. More details are shown in Section
1.2 Contributions
In this work, we attempt to analyze and address the above drawbacks. To fulfill our goal, we first conduct an in-depth analysis by rethinking the design of ZCPs, and experimentally confirm their limitations (as shown in Section 2). To tackle those backs, motivated by powerful large language models (LLMs) Radford al. 20188, Brown et al. 2020). Achiam al. [2023. Chang et al.) 2024), Liu et al. 2024 for generating new ideas and knowledge, we provide an affirmative answer by proposing a novel way automatically design 7CPs via L.LMs, dubbed APD. Specifically, we first explore naive method (as depicted in Fig. ID)) using simple prompt input to LLMs, however,

---

Spearman (SP) ranking correlation of ZCPs searched by the naive method is very poor. This raises the challenge and need of how to seek new strategy, improving the effectiveness of LLM--divven ZCPs. To reveal the root cause, we conduct an in-depth analysis and observe that the naive method lacks thought between the prompt and tasks, which could be a potential reason of poor correlation. Inspired by GPT-4o and Deepseek R1 Guo et al. [2025]. we leverage actor-critic based reinforcement learning (as shown in Fig. [ as the reasoning engine, enabling it to generate intermediate reasoning steps before the final answer. This plays critical Chain-of-Thouuge (CoT) role in enhancing APD performance, especially in NAS tasks requiring multi-step reasoning. We summarize the main contributions of this work as follows:
New ZCPs paradigm. To the best of our knowledge, we are the first to propose novel ZCPs paradigm by leveraging LLMs, providing a new perspective for understanding and designing training-free free NAS.
CoT driven strategy. Beyond the limit of poor correlation of simple prompts for designing 7CPs, we first reveal that the root cause the absence of positive reward signals for these proxies during search. To address this, we propose actor-critic based reinforcement learning to build reasoning engine, achieving improvements akin to Chain-of- -hhought reasoning in GPT-4o and DeepSeek-RI and yielding much stronger proxy-to-performance correlation.
Numerical Verification. Extensive experiments validate the superiority of our APD, outper- forming previous methods on mainstream search spaces and datasets.
2 Rethinking the design of Zero-cost Proxies
The primary goal of AZPs is to accurately pre- diet the ranking of architectures without train-
Method
Corresponding formula
Human expert
ing on a given search space. As shown in
SNIP
k 6.0
Table [T, the representative AZPs (i.e., SNIP,
Fisher
Exea (
SWAP) leverage heuristics, statistical, or gra- dient properties to measure expressibility of ar-
SynFlow
(%) I 6 0,0, R- 1 (Tee 10/11
V
chitectures. However, those AZPs heavily rely on human expertise, which may be suboptimal for new search spaces or datasets. In addi- tion, designing new ZCPs is time-consuming.
AZ-NAS
Eme log RuA(sMM))
V
SWAP
APD
AND
(Ei
x (Ene
x
Table 1: Comparison of existing ZCPs. Here, sPD is the hest-performing ZCP discovered by APD on NAS-Bench-2011. In APD, B and repre- sent batchnorm and convolutional layers, while M and W denote their outputs and weights.
Notably, ZiCO et al. [023] highlights that simple proxies such as the number of parameters and FLOPss often outperform many hand-crafted ZCPs in predicting neural architecture perfor- mance. As shown in Fig. [3, hand-crafted ZCPs (i.e.. Grasp, AZ-NAS, SWAP) suffer from a significant poor correlation issue. Those experimentation with hand-crafted ZCPs motivate us to seek a new ZCPs paradigm for designing training-free NAS.
Inspired by the knowledge generation capabilities of LLMs, we raise a new question: can LLMS automati- cally generate zero-cost proxies (ZCPs) for NAS tasks? To test this, we designed a simple prompt (App. A) that asks the LLM to output 7CP, based the assumption that I.MMs have internalized rich knowledge of neural networks during pretraining. On NAS- -0S-een the generated ZCP achieved 60.51% % spearman correlation (as shown in Table 6) on NAS-Bench-25 in CIFAR-10 dataset, lower than hand-crafted methods, yet encouraging for LLM--diven ZCP discovery. Fur- ther analysis reveals that the lack of reasoning and feed- back the key limitation, as prompt-based generation is a black-box process. To address this, we propose an actor-critic reinforcement learning framework, inspired by Chain-of-TT Chain-of-Thought (CoT) prompting in GPT-4o. This
Mettrch
55
0 Oreo
50
Insion
SSM
search space
0.0
0.1
022
0.3
0.4
0.5
Spearments
Figure 3: A comparison of 7CPs on Au- toEncoding of
3

---

design introduces feedback loop between the LLM and the NAS task, enabling the LLM to refine ZCPs iteratively and significantly improving both the performance and efficiency of proxy generation (as shown in Fig. 3
APD: A Resourceful Adviser for Training-Free NAS
3.1 Automatic Proxy Discovery
To fulfill the goal of designing ZCPs for the training-free NAS, APD utilizes LLMs to automatically generate proxies by evolving both natural language descriptions and corresponding code. addition, we propose actor-critic RL controller sampling appropriate prompt strategies to guide the evolution, aiming to optimize the correlation between proxies and final model performance. The overall framework of APD is depicted in Fig. 4 which consisting of three main components follows:
Population
Proxies
valuaae
Mutation prompt
apu
AI
Code
Search spacer
Motric roquirement
N
Thought
Crossover prompt
LLMs
Search space:
Metric requirement...
Score
Ground Truth
Inttialization prompt
Controller
Task:
Search
Crossovet
10%
Fittness
Metric requirement...
Samplins
Actor
Crifcc
Reward
Figure 4: Overview of APD. APD utilizes large-scale pre-trained models automatically search for the optimal zero-cost proxies tailored for training-free NAS.
Proxy Candidate Generator. LLM in APD
proxy candidate generator. Carefully structured prompts enable it to synthesize new ZCPs or refine existing ZCPs. Let P denote the set of valid proxies that satisfy fixed I/O contract. During time step f, with the current proxy population P.(P, S P). the LLM receives both a structured prompt π E II that specifies the requested operation II(intiiaizaation, mutation, crossover) and bounded context window C. C P, containing existing proxies together with their natural language rationales and codes. This pair (π, C.) induces context-conditioned distribution:
UFrC. P ा. C.).
where Hrr,C represents the probability distribution over candidate proxies induced by prompt π and context C, with LLM L. Composing over all prompts and admissible contexts yields an implicit, context-aware search space:
F U-U supp(ur.c), LEN TEneeP.
where supp(px.c) is the support set of distribution Pr,c. Obviously, we have F C P. Relative to context-free pairs, this construction enlarges support while preserving ergodic reachability and allows the generator to reuse salient patterns already discovered P. and thereby drives the realized search space A to lie as close as possible to P.
In APD, three categories of context-conditionas pairs employed, each of which gives rise to distinct distribution:
Initialization /imit, provides only the task description, I/O contract and ample prior knowl- edge that depicts input architectures and data.

---

Mutation f conditions on a single proxy f to perform local perturbations. enabling fine-gre -nn-grained exploitation.
Crossover Mcross, ff furnishes at least two parent proxies and asks the LLM to identify their common principles and fuse complementary components, encouraging recombination of useful motifs.
The context-aware Proxy Candidate Generator endows APD with flexible exploration granularity: during single evolutionary step, the algorithm elect either to densely sample previously unvisited regions of the proxy space or to perform ane-grained, local refinement around promising candidates, thus supporting both broad coverage and deep exploitation as needed.
Fitness Evaluator. The fitmess evaluator swiftly quantifies each candidates ZCP by computing its Spearman correlation with ground-truth accuracies on the given NAS benchmark A (e.g., NAS- Bench-201). For a benchmark set B = {(a,,P:) p:)] of architectures and ground-truth accuracies under dataset D. The fitness assigned to is
Φ()) (((()), (a),) p) (eost (/).
Where p is Spearman or Kendall correlation, computed on randomly sampled subset of provide an unbiased estimate of ranking fidelity. cost(f) is the average runtime required to evaluate one architecture of B. a hyperparameter that controls the trade-off between predictive quality and computational efficiency.
RL Evolution Scheduler. To render the proxy evolution strategy learnable and capable of converging efficiently toward optimal ZCPs, APD introduces an actor-critic module that serves as the evolutionary decision-maker. The actor-critic treats the fitness score returned by the evaluator its reward and learns a policy that maximizes this signal, thereby jointly optimizing both the evolving set of ZCPs and the evolution strategy itself.
In a given search space A candidate architectures and their ground-trute performance p(a), the objective of APD is to learn a proxy f A + IR whose scores preserve the ranking induced by p. Therefore, we aim to maximize the expected correlation:
max Eacalp(( p(a)), JEF
where P is Spearman Pa. Each proxy f is represented as a tuple (T,C) of T natural language thought describing the proxy's 's principle, and C an executable code that returns a scalar score, ensuring interpretability and deterministic replay throughout the evolutionary process.
To accelerate convergence toward high correlation proxies while maintaining effective exploration in search space F, APD introduces evolutionary step. At generation f, the controller observes a compact state vector with fixed-width histogram of fitness values and strategies, sampling an action a, E II from a categorical policy Ta(a)s)). Executing a yields candidate proxies P/, whose fitness o> ( is evaluated by the Fitness Evaluator: The scheduler then receives a clipped reward r E(O(P))
Actor-Critic as evolution scheduler that governs every
(P))) that promotes substantive improvements and updates the Actor-Criti by standard advantage-actorr-rinng gradient steps:
0 + 7 V log No (at 8x) + - Vy())))
(5)
+ サレ The V (r: + Vo((s-1) (s(s)))
Where 0 and denote the actor and critic parameters, y) and T/ are the actor and critic learning rates, VU is the value baseline parameterized by vh, and y is the discount factor. The critic minimizes (r + (s (s, and the actor maximizes the advantage-weighted log likelihood.
3.2 Evolution Framework
Based the components outlined in Section 3.1 we integrate the Proxy Candidate Generator, Fitness Evaluator, and the RL Evolution Scheduler into evolutionary loop. As shown Algorithm [ APD unfolds through the following mutually dependent steps:
Step Initialization In time step 0, an (initialivation, 0) issued to the LLM,, producing the seed Po (fi.... fN
5

---

Step Generation At generation t the actor samples an action (op, Ct) (T(-(sr) where op E {init, mut, cross). The corresponding prompt and context window S P. are fed to the LLM, yielding candidate proxies P.
Step 2 Evaluation The candidates are scored in B by the fitness function 6(-). If the proxy fails the contract check, is assigned Φ -0.
Step 3 Policy update The scheduler receives reward rz = [B((()) by calculating the mean & and updates (0, x5) by the rule in (5)
Step 4 Population replacement The union P. U (P,) is sorted by fitness. The worst proxies dropped to keep Pt-1| = N. A tie-breaking rule that favors younger proxies prevents age-related stagnation. The loop then returns to Step 1.
The loop terminates after Tmaa generations. Empirically, the framework produced proxies whose Spearman correlation on NASBench201 exceeds 0.80 within 30 generations roughly one GPU-hour on a single RTX4090.
Algorithm Evolution Framework
Require: LLM C, benchmark B, population N, budget Moa., actor-critic (Ta, Vi)) Po t first N proxies from Minit,0. 80 + {0, 0}
for t = 1 to Tmax do
op ~ T-Iss),
P - [(Hop.)) s0 x o(P') E(p), update (0, 3,,11)
P (PUP)) (/PP)//wworss (ww
end for
return arg max (f))
4 Experiments
4.1 Experimental Settings
We evaluate APD on 5 representative search spaces (e.g.. NAS-Bench-201 Dong and Yang (020)] MAS-Bench-100 101 Ying et al. 2019), DARTS Liu et al. [2018), MaasNAAS-Bench-101--11
Duan et al. [2021), OoD-ViT-NAS Bai et al. [221I) across 4 tasks: image recognition, autoencoding, scene classification, and supe-super jigsaw puzzles. We transfer the zero-cost proxy identified search space to all datasets in both NAS- TAS-Bench-200 and
on CIFAR-10 10 within the MASS-Bench-22 NAS-Bench- 101. However, due to the substantial search space disparities between TransNAS-Bennn 101, OoD-OoDD-iiii ViT-NAS, and MAS--Benc--00 2011 search for new zero-cost proxies directly within these spaces to ensure optimal performance. In addition, we train APD with 7 mainstream LLMs (i.e., GPT44 Achiam et al. (2023) Claude 3.7 Anthropii 20255 Deepseek V3 Liu al. (2024. Gemini flash Google Cloud [2025] Ilama 4 Meta Al 2025
Grok 3 xAl 2025
Full experimental details
provided in App. B.
4.2 Performance on NAS-Rench-201.2 101 search spaces
Table compares APD with state-of-the-ard training- free NAS methods on NAS- -AS-Bench--01 (CIFAR- 10/100, ImageNet16-120) and NAS- PAAS-Bench-101 (CIFAR-10) 10). APD consistently achieves the highest test accuracy and ranking correlations (SPR/KT) across all datasets. Compared to the best existing methods, APD improves test accuracy by +0.07% on CIFAR-10 (vs. ZiCo), +0.52% on CIFAR- 100 (vs. Synflow), and +0.69% on 1- (vs. AZ-NAS), while reducing runtime cost by over 50%. On ImageNet 6- 120, APD also outperforms Grasp by + +14.18% 18%, ZenNAS by +7.85% ZiCo by 13.61%, and AZ-NAS by +0.69%. These results not only highlight the efficiency and robustness of APD across diverse search spaces, but also demonstrate the strong potential of M----erieen 7CP design, advancing the development of training-free free NAS.

---

Table 2: Performance comparison NAS-Bench-200 (NB-201) search space with CIFAR- 10/1000 and ImageNet16- 120 datasets, and We report Kendall's r (KT) and Spearman's 's p (SPR) computed with all candidate architectures. In addition, we report the average independent runs results obtained from our method. The red, blue, and orange indicate the best, second-best, and third-best results, respectively.
101 (NB-101) search space with CIFAR- 10 dataset.
Melhad
XPR
CIBAR-INNI-3RI
KT
The aV:
62R
KCT
Tra rcc
-152
Tiet [V0
3091
CIFAREBUN--OU
Texl A2 (5)
Bustine
3.57g
0.526
0.550
1005 0.33
0.519
41.07
12
155
FRAINN
0.733
3.541
M1.81
0.318
11:31
1p00 11337
04611
0.517
43.07
-064
13.77
12-30 18
1.55
Seat Sokifeteestal
0.903
0:215
18:01
0.470
13320
$1,41
0.35
0.367
31.85
1.00
57.00
1.00
MS TD
0.204
3.871
M187 0.200
0.198
11263
1120 1180
R766
D.83
03:30
3:21
0.38
194 03 2M7
756 Th
0:030
Zesh
0.263
10.55
0.338
13.315
54.00
D.273
0.361
77
3.09
01.
31.00
Zec
1878 core
0.784
3.509
08:00
0.513
0.620
1863 1.6
0834
0604
41.43
009
0.68
02 64 0.50
30
0.510
0/611
48.06
12.40
07.13 1.85
17734
DA10
25.
58.53 316
APD
0,83
0.435
193.70
32 0039 (00.00)
0.543
0.656
77233 100.2)
0817
0433
44308
70060000000
0.75
(00.40
12 0634 (0033))
16:1 20)
Opinaal
1A.35 6:
10.90:21..00
40.97 9.09
10.80 011
Table 3: Comparison the DARTS search space using CIFAR- (left) and ImageNetlk (right). "C10", "C100", and "Img" indicate search conducted on CIFAR-10, 10, CIFAR-100 100, and ImageNet respectively. All models retrained using official code from Liu et al. [018 for fair comparison. "Paper" shows results from original publications; denotes results from released training code means unavailable or unreleased data.
Zheng al. 2021]:
TIPORTER 1/11
Torees
GPT Diti-Dus
Tay HP6) Tap-
FEUIS
Mobattil 5311
15.0
11.3
413)
3153
$7:11
1/1
04000
$1:
$2.0
Chideet
369
210
67.8
$2.8
15.8
05.3
DARTS
15/113
DAE Prete
044
Oedert
151
05.
ELA
Ordon
15.5
12.3
377
Unden
Caden:
157
E-BASTE
A222
5-DAKE
Godee
15.35
13.5
114
201
NAR
13
150
103
43
390
Thanteg tnd
352 352
4.3 Performance on DARTS Search Space
CIFAR-10 and CIFAR-100 Datasets. As shown in Table 3. APD achieves state-of-the-art perfor- mance with sigmificantly reduced search costs. On CIFAR- 10, APD attains a top-1 accuracy of 97.63% % using merely 0.004 GPU-days, surpassing representative gradient- gradient-based methods such as DARTS and advanced training-free -rre methods like TENAS. Similarly, on CIFAR-100, APD achieves 84.83% accu- racy, outperforming leading competitors including DARTS+ and FP-DARTS. These results clearly demonstrate that APD sets new benchmark in both accuracy and search efficiency across datasets.
ImageNetlk Dataset. Table 3 (right) summarizes results on ImageNet demonstrating the strong generalization capability of our method. APD achieves 76.9% top-1 accuracy, improving over the leading training-free method (e.g. NASI-ADA and SWAP) and the exist- ing best one-shot method. Importantly, APD achieves these improvements with minimal search cost of only 0.004 GPU-days, which is 1.5x less than SWAP. These findings clearly demonstrate APD's excellent performance and generalization efficiency across different datasets.
Table 4: Results on TransNAS-Bench- Meeeechh-11-MaM 101-roc
Methud
Autoencuding SSIM
Scene Classiiictiion
Jigsuw
Accuracy (%)
Accuracy (%)
Ground Abdolaatta
$4.9
Grad norm
2021
0,361 0.03
48.7 720.7
30.310.33
SNIP Andeh in
0.33++0.04
48.7+1. 1.1
80.3++0.1
0.33:10.06
10.2:11.
91. 1:0.3
Fisher 24r
0.49:00.01
48.7=006
83.5t:1.2
Synflow Atrdellaaaa
0,46+0.07
53.7=1.2
90.9+-0.4
NWOT
0.43:0.02
33.22:0.
92.3100.3
Zea-soore Lin
2021
0.46 01
53.7=0.2
87.5+0.4
GradSign Zhang and 2021
0.35
53.610.4
93.1
Params
0.46
53.7
$5.9
FLOPS
0.46
51.7
85.9
Lietal
2023
0.48
$3.740.4
93.2110:4
SWA
Frrnng
0.42+0.02 02
45.0= 10.9
89.8+5.
((006)
$40:006 NO.3)
91.2:01 W23)
7

---

4.4 Results on TransMAS-Bench-001-MMM
As summarized in Table our method consistently achieves state-of-the-a performance across all three tasks on TransNAS -eeNNech---e------- Bench-101-Micro highlighting its strong generalization capability heterogeneous downstream tasks. In particular. for the Autoencoding task, our approach attains an SSIM score of 0.54, surpassing leading methods such as ZiCo (0.48), NWOT (0.43), and SWAP (0.42). Similar performance improvements are observed in the Scene Classification and Jigsaw tasks, demonstrating APD's robustness and effectiveness across diverse downstream tasks.
4.5 Generalizability on OoD-ViT-NAS-Ti
To further evaluate the generalizability of method on vision transformers under Out-of-Distributicn (OoD) conditions, we conduct experiments on the DoD-ViT-NAS-T search space Ho et al. (2025] across five datasets: ImageNetlk, ImageNet-A,, ImageNet-R, ImageNet-D/Texturre and ImageNet- D/Matcrial, All results are averaged over 5 independent runs. As shown in Table S APD consistently outperforms prior methods across all OoD settings. Specifically, APD achieves +0.04 higher p than NWOT on ImageNetIk, and +0.01 over DSS on ImageNet-D/Textur surpasses all methods with a highest p of 0.88, outperforming AutoProx by +0. 10 and DSS by +0.07. These results highlight the strong generalizability of APD under distribution shifts, and further confirm the effectiveness of leveraging LLMs to design accurate and robust zero-cost proxies for transformer-basee NAS.
On ImageNet-R, APD
Table 5: Correlation on OoD-ViT-NAS-Ti search space.
Method
ImageNetkk
ImageNet-A
ImageNet-R
ImageNe-I//eextnrg
ImageNat-IMMateria
SNIP
0.38
0.51
0.35
-0.l6
0.11
Grasp
-0.03
-0.06
.0.07
-0.01
0.03
MeCo
CroZe
0.40
0.54
0.60
0.01
0.12
DSS
0.62
0.82
0.81
0.02
0.17
AutoProx
0.67
0.82
0.78
0.05
0.15
NWOT
0.75
0.76
0.74
0.11
0,12
(.794)
0.82
0.121(0.01)
4.6
Ablation studies
The impact of various LLMs: To scru- tinize the impact of various LLMs (i.e., GPT4o, Llama 4), we perform an abla- tion study of LLMs 201 search space in the CIFAR-10 dataset (As shown Table [6] and the left panel of Fig. (5) and the MaansNAS-Benchh 101 Micro search space (as shown Fig. right). Specifically, we conduct experi- ments by averaging 4 independent runs to keep a fair comparison. From Table 6 and Fig. we draw two conclusions: (I) Our APD demonstrates strong robust-
Table 6: Comparison of APD with different LLMs on NAS AAS-Bench-201 search space in CIFAR- 10 dataset.
the NAS-Bench-
Method
LLMs
Run
Run 2
Run
Ron 4
Average (p)
Naive
GPT4o
60.37
59.74
60.52
61.40
60.51
APD
Claudo 3.7
80.90
80.91
82.72
81.91
81.14
APD
Deepseek V3
79.77
80.78
79.77
79.77
80.24
APD
Gemini Bash
75.47
73.31
73.50
74.66
75,22
APD
Llama 4
73.04
72.70
72.32
72.95
72.87
GPT4o
80.96
APD
Grok
72.23
79.77
84.51
79.32
ness across different LLMs. Specifically, APD achieves highly consistent results under different LLMs, showing minimal performance fluctuations across models. Moreover, APD is stable to noise caused by different training settings, further enhancing its reliability in real-world deployment. (2) Compared to Naive method, our APD significantly improves the performance in terms of accuracy. For example, the accuracy of APD with Claude 3.7 21 21.26% higher than the Naive method. The performance improvement can be attributed to our proposed actor-critic reinforcement learning framework, which provides key reasoning and feedback for proxy generation.
The impact of various components: We conduct ablation studies on NAS- Bench-201 (CIFAR-10/100) to evaluate the effectiveness of key components in APD. As shown in Table 7 the naive ver- sion without Evolution and Actor-Critic
Table 7: Ablation study of various components.
Naive
Evolution
Actor-Critic CIFAR-10NN-201)
(IFAB-00/NB-200)
83.53
95.76
72.22
modules performs poorly (82.04% in CIFAR- 10 and 47. 47.62% in CIFAR- 100). Introducing Evolution
8

---

GPT
Qwen plas
7640
Cluude 3.7
Qwen plds
Claude 3.7
20:60
22
Liama Mavertck
DeebseekVi
Liama Miverick
Deepseek V3
Grok 3
thehimi2.44 O Blah
Grok
Ochimn12. flash
Figure 5: A comparison of ranking correlations of our method designed by 7 mainstream LLMs including GPT4o, Claude 3.7, Deepseek V3, Gemini2.0 flash, Grok 3, Llama 4 Maverick, and Qwen plus. Specifically, conduct the empirical validation NAS-Bench-22 (Left) and TransNAS- Bench-101 (Right) search spaces in the CIFAR-10 dataset.
alone significantly boosts performance, yielding +6.49% and +13.54% improvements, respectively, In addition, we observe that there is a joint contribution between Actor-Critic and Evolution. Specifically, obtains 93.76% CIFAR-10 and 72.22% in CIFAR-100. Those results validate the effectiveness of our APD.
4.7 Discussion
We are surprised by the remarkable performance of zero-cost proxies designed by LLMs for training- free NAS, in this section, conduct in-depth analysis of APD. APD achieves impressive performance primarily due to the effectiveness of the reinforcement learning strategy. The RL policy enables adaptive selection and refinement of zero-cost proxies, guiding the LLM toward generating more task mess-relevant strategies. This interaction mechanism significantly enhances the synergy between LLMs and NAS, ensuring that the discovered proxies are both generalizable and high nigh-performing.
Despite involving LLMs, our framework mitigates the typical "black- concerns by embedding the LLM within an RL-guided optimization loop. This interaction provides structured feedback signal, reducing the opaqueness of the overall process and making the optimization more transparent and controllable. Moreover, our approach redefines the traditional NAS design paradigm by shifting from handcrafted heuristics to LLM-guided discovery. This not only introduces new learning framework for NAS but also offers transferable methodology for other deep learning domains.
4.8 Additional Evaluation
Due to the page limit, provide more experimental results in App. A-H. O Detailed Prompt Engineering of APD are presented in App. C. @ More ablation studies are presented App. D. O Limitations are presented in App. E. O Visualizations of designed AZPs presented in App. F. @ Visualizations of searched architectures are presented in App. G.
Related Work. The related work is provided in App. H.
5 Conclusion and Future Works
In this work, we observe that existing zero-cost proxies in training-free NAS are manually crafted, inefficient, and poorly correlated with final performance. To address this. we propose APD, an LLM-driven framework that automatically discovers high-quality proxies without human effort. By incorporating actor-critic reinforcement learning strategy. APD iteratively refines prompts to generate better proxies. The proposed method achieves state-of-the-an art performance and efficiency on multiple NAS benchmarks. We believe APD offers a novel design paradigm and will inspire broader applications of LLMs automated machine learning. the future, we plan to further improve the performance of APD,

---

References
[1] Mohamed S. Abdelfattah, Abhinav Mchrotra, Lukasz Dudziak, and Nicholas Donald Lane. Zero- cost proxies for lightweight NAS. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint S.XXi:233330877440 2023.
[3] Yash Akhauri, J. Pablo Munoz, Nilesh Jain, and Ravi Tyer. Eznas: Evolving zero cost proxies for neural architecture scoring. arXiv preprint 2x11122211.. 2022. URL. httth://p/:/rii
[4]
Anthropic. Claude 3.7 Sonnet. Sttp://feee/e../e.aattttttato//a 2025.
[5] Haoyue Bai, Fengwei Zhou, Lanqing Hong, Nanyang Ye, S-H Gary Chan, and Zhenguo Li. Nas-ood: Neural architecture search for -uttof-distributioon generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages $320-8329, 2021.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 33:1877-1901, 2020.
[7] Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. ArXiv, abs/1812.00333 2018.
[8] Yupeng Chang, Xu Wang. Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
[9] Angelica Chen, David Dohan, and David So. Evoprompting: Language models for code-level neural architecture search. In Advances in Neural Information Processing Systems, volume 36, pages 7787-7817. 2023.
[10] Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Wanli Ouyang, et al. Glit: Neural architecture search for global and local image transformer. In ICCV, 2021.
[11] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching trans- formers for visual recognition. In International Conference on Computer Vision, 2021.
[12] Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. ArXiv, abs/2102.11535, 2021.
[13] Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song. Zhangyang Wang, and Denny Zhou. Auto-scaling vision transformers without training. In International Conference on Learning Representations, 2022.
[14] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1294-1303, 2019.
[15] Xiangxiang Chu, Xiaoxing Wang, Bo Zhang, Shun Lu, Xiaolin Wei, and Junchi Yan. Darts-: robustly stepping out of performance collapse without indicators. ArXiv, abs/2009.01022 2020.
[16] Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair darts: Eliminating unfair advantages in differentiable architecture search. In Computer -CCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XV, pages 465- 480. Springer, 2020.
[17] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. Arxiv preprint 2102. 10882, 2021. URL httpt:tt//////tt //tp///a.vvvvv// wttpoooro///p///1222100/////
10

---

[18]
Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In 8th International Conference on Learning Representations, ICLR 2020,
Addis Ababa, Ethiopia, April 36-00, 2020, 2020.
[19]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[20] Yawen Duan, Xin Chen, Hang Xu, Zewei Chen, Xiaodan Liang, Tong Zhang, and Zhenguo Improving transferability and generalizability of cross-task neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5251-5260, 2021.
Li.
[21] Google Cloud, Gemini 2.0 Flash. venerative-ai/0on att////ccs//mmeeeaaaaaa////////a
http:://clood..ggglee...oooeveee coo/eeeeooo///eeeeeeeeeeeeeeeee. ai/
2025.
[22] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1244 2025.
[23] Kang Haidong, Ma Lianbo, Chen Pengjun, Yu Guo, Wang Xingwei, and Huang Min. Beyond the limits: Overcoming negative correlation of activation-based training-free nas. In International Conference Computer Vision, 2025.
[24] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. In NeurIPS, 2021.
[25] Hongyi He, Longjun Liu, Haonan Zhang, and Namning Zheng. Is-darts: Stabilizing darts through precise measurement candidate importance. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 12367-12375, 2024.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[27] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang. Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In CVPR, 2019.
(28) Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In ICCV, 2021.
[29] Sy-Tuyen Ho. Tuan Van Vo, Somayeh Ebrahimkhani, and Ngai Man Cheung. Vision transformer neural architecture search for of-iitribbttion generalization: Benchmark and insights. arXiv preprint 20Xx:22501.03822, 2025.
[30] Kun Jing, Luoyu Chen, and Jungang Xu. An architecture entropy regularizer for differentiable neural architecture search. Neural Networks, 158: 118:111-1221 1-120, 2023.
[31] Haidong Kang. Revisiting neural networks for few-shot learning: A zero-cost nas perspective. In Forty-second International Conference on Machine Learning.
[32] Haidong Kang, Lianbo Ma, Guo Yu, and Shangce Gao. Where and how to enhance: Discovering bit-width contribution for mixed precision quantization. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, 2025.
[33] Junghyup Lee and Bumsub Ham. Az-nas: Assembling zero-cost proxies for network archi- tecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5893 5893-5903, 2024.
[34] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity, In International Conference on Learning Representations.
[35] Guihong Li, Yuedong Yang, Kartikeya Bhardwaj, and Radu Marculescu. 7ico: Zero-shot nas via inverse coefficient of variation gradients. arXiv preprint arXiv:2301. 1300, 2023,

---

[36] Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping. ArXiv. 19011999906055, 2019.
[37] Ming Lin, Pichao Wang. Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot NAS for high-performance image recognition. In 2021 IEER/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17. 2021, pages 337-346. IEEE, 2021. doi: 10. 10.11099/CC 200400
[38] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint S.Ii:22412.1447,, 2024.
[39] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European Conference on Computer Vision (ECCV), pages 19-34, 2018.
[40] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. ArXiv, abs/1 808.000.990 0905,0 2018.
[41] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[42] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimiza- tion. Advances in Neural Information Processing Systems, 2018.
[43] L. Ma, H. Kang, G. Yu, Q Li, and Q He. Single-domain generalized predictor for neural architecture search system. IEEE Transactions on Computers, 13(05)):400-1413, 2024. ISSN 1557-9956.
[44] Joe Mellor, Jack Turner, Amos Storkey, and Elliot Crowley. Neural architecture search without training. In International Conference on Machine Learning, pages 7588-7598 PMLR, 2021.
[45] Meta AI. The llama 4 herd: The beginning of new era of natively multimodal ai innovation. nttps://at meta.. hatmm//log/lllaaaaaa wtt.mmmmmmmmmmmemeeelleaeee May 16, 2025.
2025, Accessed:
[46] Sajad Movahedi, Melika Adabinejad, Ayyoob Imani, Arezou Keshavarz, Mostafa Dehghani, Azadeh Shakery, and Babak Nadjar Araabi. A-darts: Mitigating performance collapse by harmonizing operation selection among cells. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
[47] Muhammad Usama Nasir, Samuel Earle, Julian Togelius, Steven James, and Clayton Cleghorn. Llmatic: Neural architecture search via large language models and quality diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1110-1118 July 2024.
[48] Yameng Peng, Andy Song, Haytham M Fayek, Vie Ciesielski, and Xiaojun Chang. Swap-nas: Sample-wise activation patterns for ultra-fast nas. arXiv preprint MXXi::2403.0416: 2024.
[49] Yameng Peng, Andy Song, Haytham M. Fayek, Vic Ciesielski. and Xiaojun Chang. SWAP-NAS: Sample-wise activation patterns for ultra-fast NAS. In The Twelfth International Conference on Learning Representations 2024.
[50] Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, pages 4095- 4104. PMLR, 2018
[51] Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15111. 2024. URL.tttps //arxiv. orgg abs/ /alp/2241...m 15115 Includes the Qwen-Plus Mixture-of-Expene variant.
[52] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training, 2018.
12

---

[53] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. In AAAI Conference on Artificial Intelligence, volume 33, page 4780-4789, 2019.
[54] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2018
[55] Yao Shu, Shaofeng Cai, Zhongxiang Dai, Beng Chin Ooi, and Bryan Kian Hsiang Low. NASI: Label- and data-agnostic neural architecture search initialization. In International Conference on Learning Representations, 2022.
[56] Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Vision transformer architecture search. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 23-27, 2022, Proceedings. Part XXI, page 139-157, 2022.
[57] Haosen Sun, Lujun Li, Peijie Dong, Zimian Wei, and Shitong Shao. Auto-das: Automated proxy discovery for training-fres distillation-aware architecture search. In Computer Vision ECCV 2024, Lecture Notes in Computer Science, pages 56-73. Springer Cham, 2024. doi: 10.10077978-3-031-7766 1L..
[58] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-6114, PMLR, 2019.
[59] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML. PMLR, 2021.
[60] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376. 2020.
[61] Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Re- thinking architecture selection differentiable nas. ArXiv, abs/2108.04399 2021.
[62] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[63] Wenna Wang, Xiuwei Zhang, Hengfei Cui, Hanlin Yin, and Yannnig Zhang. Fp-darts: Fast parallel differentiable neural architecture search for image classification. Pattern Recognition, 136: 109193, 2023.
[64] Zimian Wei, Peijie Dong, Zheng Hui, Anggeng Li, Lujun Li, Menglong Lu, Hengyue Pan, and Dongsheng Li. Auto-prox: Training-fres vision transformer architecture search via automatic proxy discovery. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38,
pages 15814-15822, 2024.
[65] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet de- sign via differentiable neural architecture search. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10734-10742, 2019.
[66] XAI. Grok 3: The age of reasoning agents.
grops/// 2025.
[67] Guorui Xie, Qing Li, Zhenning Shi, Hanbin Fang, Shengpeng Ji, Yong Jiang, Zhenhui Yuan, Lianbo Ma, and Mingwei Xu. Generating neural networks for diverse networking classification tasks via hardware-aware neural architecture search. IEEE Transactions on Computers, 2023.
[68] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi. Qi Tian, and Hongkai Xiong. Pe-darts: Partial channel connections for memory-effiiint architecture search. ArXiv, abs/1907.05757 2019.
13

---

[69] Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. CARS: continuous evolution for efficient neural architecture search. In 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 1826-1835. IEEE, 2020.
[70] Peng Ye, Baopu Li, Yikang Li, Tao Chen, Jiayuan Fan, and Wanli Ouyang. Abeta-darts: Beta- decay regularization for differentiable architecture search. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10864-10873. IEEE, 2022.
[71] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pages 7 7105-7114 14. PMLR, 2019.
[72] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang. Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021.
[73] Zhihao Zhang and Zhihao Jia. Gradsign: Model performance inference with theoretical insights. arXiv preprint arXiv:2110.086 2021.
[74] Xiawu Zheng, Rongrong Ji, Yuhang Chen, Qiang Wang, Baochang Zhang, Jie Chen, Qixiang Ye, Feiyue Huang, and Yonghong Tian. Migo-nas: Towards fast and generalizable neural architecture search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):
9936-2952. 2021.
[75] Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. Econas: Finding proxies for economical neural architecture search. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 11393 11401. IEEE, 2020.
[76] Qinqin Zhou, Kekai Sheng, Xiawu Zheng, Ke Li, Xing Sun, Yonghong Tian, Jie Chen, and Rongrong Ji. Training-free free transformer architecture search. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10894-10903, 2022.
[77] Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations (ICLR), 2017.
[78] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable archi- tectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8697-8710, 2018.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly and accurately reflect the paper's contri- butions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Our limitations are in section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
The full set of assumptions and complete proofs are provided in section 3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper provide sufficient information reproduce the main experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We provided runnable code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
cessiinnooicyyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuiii wwwiiccgguie//eeeeeeeeeis CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All experimental details and settings are in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The error bars reported in our paper are appropriately and clearly defined.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Sufficient information is included the paper to reproduce all experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: btctps::/neuripttt http://rrrrpppppppppppeeeeeeeeea httpp//biic//hiicecceeeeeeeeeeee
Answer: [Yes]
Justification:
Our research fully adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: We mention the license in the README file in the code of the provided supplementary material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We describe our prompt engineering workflow using LLMs compo- nent in section 3
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

A A Simple Prompt
To empirically validate the hypothesis presented in Section we employ the prompt depicted in Figure 6 to assess the capability of LLMs in generating effective ZCPs.
Figure 6: representative minimal prompt used in our naive experiments.
Prompt for Naive
Please help me design five novel Kero-Cost Proxies evaluating the expressive performance of neural network on a given batch of image data using training-free metrics.
These proxies should require no training, and must be computationally cheap to evaluate, similar to existing zero-cost proxy methods such as NWOT, SynFlow, Jacobian Covariance. However, you should not copy or imitate these methods - your goal creatively propose original ideas that align with the following requirements:
For each of the five proxies, do the following: Firstly, provide one-paragraph description of the idea, including how the score is computed. Secondly, Implement the proxy in Python, as function named 'ovaluate(model, data)' that: Takes a neural network object ('model') (pytorch model) and image batch ("data' (tensor)) as input. Returns scalar score ('score') reflecting expressive performance.
Note: not use training. Avaid random sampling or stochastic components. The heuristic must work on purpose image data and standard neural networks (e.g.. CNNs). Do not provide any additional explanation outside of the required description and code.
B Experimental settings
B.1 Evolution Settings
Table summarizzes the default haper--yrrannnn parameter configuration used in APD evolutionary experiments. Searches conducted single RTX4090 GPU with fixed random seed of 0. Each candidate proxy's performance is averaged over 5 independent forward passes using a batch of 16 random samples. We adopt CIFAR-10 the primary benchmark and report transfer results on CIFAR-10 and ImageNet-16 The actor-critic controller is a two layer MI.P with 256 hidden units. The actor and critic learning rates are 1e-3 and 1e-2, respectively, with discount factor 7 0.9. During search we run 10 episodes of 100 steps search and test every individual for 200 architectures. The controller retains history window of 5.
Table 8:
for Automatic Proxy Discovery.
Parameter
Value
Parameter
Value
Episodes
10
Steps
100
History windows
Discount factor
0.9
Actor learning
1e-3
Critic learning rate
le-2
Input batch size
Repeats
5
Hidden size
256
Number of layers
2
Population size
B
1
B.2 DARTS Settings
We list the training syper-parameters adopt for all experiments DARTS. Figure shows the optimal architecture obtained by APD in the DARTS search space.
As detailed in Table 9 models are trained for 600 epochs with fixed random seed of 42. We employ SGD with momentum 0.9, an initial learning rate of 0.025 that is decayed to zero via cosine annealing, weight decay of 5c-4, and gradient clipping at maximum norm of 5, Architectures begin with 36
22

---

Table Blyper-parameters for DARTS training.
Parameter
Value
Parameter
Value
Sced
Batch size
CutOut length
Initial channels
36
Cells / layers
20
Aux. loss weight
0,4
Learning rate
0.025
Momentum
0.9
Weight decay
5c-4
Grad clip (norm)
DropPath prob,
0.2
Lr scheduler
CosinnAnnealing
channels and stack 20 cells. An auxiliary classifier is attached with loss weight of 0.4. We apply CutOut regularization (1ength=16) and linearly increase DropPath probability 0.2 over the course of training. All runs use batch size of 96.
B.3
Autoformer Settings
For all ImageNet-Ik Ik training runs, we adhere to the core setup prescribed by AutoFormer 69]
which
is listed in Table T0
The subnet discovered by APD are reported in Figure 28 Figure 29 and Figure
30
We retrain discovered AutoFormer architectures for 300 epochs on ImageNet- 1k using AdamW with an initial learning rate of 5e-4, cosine decay to le-5, and 5 epoch warm-up. The network consumes 224x224 images split into 16x16 patches, and optimization runs with batch of 128. Regularization follows the AutoFormer baseline. RandAugment (9 transformations, magnitude 0.5), mixup-cutmix blending (a 0.8, switch probability 0.5), random erasing (probability 0.25, pixel model), and linearly ramped drop-path of 0.1. All experiments use the retrain mode, thereby only the chosen subnet's weights are updated while the supernet is frozen.
Table 10: Hyper-parameter configuration used retrain the subnet on ImageNet- IK.
Category
Parameter
Value
Notes
Input resolution
224 x 224
Data
Patch size
16
VIT patch size
Batch size
128
Optimizer
AdamW
8 (0.9. 0.999), 10-8 Scaled by batch size 512 Decoupled
Initial LR
5e-4
Optimiser
Weight decay
0.05
LR schedule
Cosine
5 warm-up + 300 epochs
Drop-path rate
0.10
Linearly increased
Regularisation
RandAugment Mixup/ CutMix
rand-m9-metdO.. 1at--m99mmttt.--iinmeee 5-inc1 a a-0.8, prob. -
Timm default
Switch prob. 0.5
Model
Training mode Max rel. position
retrain
Only subnet weights updated
14
Bias radius
B.4
LLMs Settings
For proxy generation, we interface with GPT-4o via the Chat Completion API 2 and evaluate APD across multiple LLMs, including Claude 3.7 [4), Grok 3 [66), Gemini 2.0 Flash 21 Deepseek V3 38), and Qwen Plus [51]. To balance diversity and syntactic correctness, APD is performed with temperature of 0.5 and the output length is capped at 8192 tokens. We supply a specialized system prompt that constrains LLM to emit JSON object with two fields: thought + code.
C Prompt Engineering
In this section, we outline the guiding principles underlying the design of our prompts. Given that the effectiveness of LLM--rrieen heuristic discovery critically hinges on prompt quality, our prompts are carefully crafted to maintain clarity, specificity, and structured output. Particularly, we ensure clarity and specificity by precisely defining the computational boundaries and structural properties of
23

---

candidate proxies within each prompt, clearly delineating the allowable search space of the metrics facilitate efficient exploration by the LLM. Inspired by previous auto proxy discovery research, we prompt the L.M to construct compositional proxies represented as directed acyelic graphs that integrate seworkkddervved signals (e.g., gradients, activations, or weights) with basic statistical and arithmetic operations. Collectively, this rigorous prompt engineering strategy significantly enhances the efficiency, validity, and innovativeness of the ZCPs by APD.
Moreover, through extensive empirical experimentation, we observe that the initial code representation of generated ZCP typically fails to fully realize its underlying conceptual potential. Specifically, the initial instantiation of the proxy, directly produces from single prompt, often exhibits suboptimal correlation with the final performance metries. is only after applying evolutionary strategies such as mutation and crossover operations that iteratively refine the proxy structure and composition that the intrinsic effectiveness of the proxy becomes evident. This empirical insight underscores the necessity of iterative exploration and refinement within the APD, highlighting mutation and crossover as critical mechanisms for achieving robust and effective ZCPs.
C.1 Prompt For NAS-Bench-201
We provide the detailed prompts employed in our experiments on NAS-Bench-201 prompts serve as explicit instructions guiding the LLM to systematically generate novel ZCPs. Specifically, we design three variants of prompts as depicted in Figure 7 and 9 Each prompt variant progressively introduces structured definitions, constraints on computational operations, and explicit guidelines ensure that the generated proxies adhere to our framework requirements, thereby facilitating effective exploration of the proxy search space and enhancing the quality of the discovered ZCPs.
18 These
C.2
Prompt For TransNAS-Benchh-100
We detail the specific prompts utilized for proxy discovery experiments on TransNAS-Bent 101 These prompts are carefully crafted to instruct the large language model (LLM) to produce novel zero-cost proxies tailored specifically for their diverse tasks, including JIGSAW, autoencoding. scene classification, and object classification. To accommodate these distinct downstream tasks,
20
designed three prompt variants as shown in Figure 10 and 12
C.3 Prompt For AutoFormer
We present the detailed prompts employed for proxy discovery experiments conducted on the AutoFormer architecture search space. Given AutoFormer's s specific structural characteristics and its transformer- design for vision tasks, we carefully tailor three distinct prompt variants following
previous designs in Figure 13 14 and 15
D Extended Experimental Results
D.1 Additional Ablation Studies
Analysis on Mutation and Crossover operations To figure out the roles of the mutation and crossover operations, we evaluate the individual contribution the mutation crossover operation by selectively disabling each component during proxy search and observing the resulting performance changes. Figure illustratts that omitting the mutation prompt reduces the proxy's s correlation with ground-truth metrics while removal of the crossover operation narrows the architectural exploration space and weakens proxy accuracy. These findings confirm that mutation and crossover provide distinct but complementary functions. collectively enabling the effective of high-quality zero-cost proxies.
Analysis on population size To understand how the size of the evolving population affects the quality and convergence of our zero-cost proxy search, we perform an ablation study over three different population sizes: 1, 2, 3, 4, 5 and 10. A larger population can offer richer diversity and robustness but incurs greater computational overhead and makes it harder for the LLM to follow context, thereby reducing ZCPs quality.In contrast, smaller population accelerates each generation

---

Figure 7: Prompt used for intialization in NAS-Bench-22
Prompt Initialization
Your Task:
Please design 5 Zoro-Cost Proxies to evaluate the representation capability of different convolutional network architectures given dataset. The final goal of each proxy is provide scalar score that reflects the performance of given network. You should generate Zero-Cost Proxies according to the following specifications, which include: proxy requirements, description the proxy search space (defining the structure of how the proxy is computed), the search space of the networks being evaluated, the given dataset, and the output format, described below:
Proxy Requirements:
Training-free No gradient descent, weight updates, or learned parameters
Efficient: Low computational cost, suitable for carly-stage model selection Dxeterministic: No stochastic elements or randomness.
Input-aware: The proxy should utilize batch of input image data The proxy should reflect meaningful differences between models.
Model-sensitive
Proxy Search Space:
The computation of the proxy can be represented as directed acyelic graph (DAG). For any given node, its in-degree must satisfy 0 I degrec.. S 2.
Nodes with only outgoing edges and incoming edges are called inputs. Inputs consist of two parts: the input modulc and the input property. The input module refers to different types network layers (c.g., convolutional layers, BatchNorm, activation layers, etc.), and the input property is one of the following: gradient G, weights W, output feature maps Z. An imput is defined as combination of these components. For example, (convolutional layer gradient G) refers the gradients of the convolutional layer's 's parameters. proxy can have multiple inputs.
Nodes with both incoming and outgoing edges are referred to as operations, which are further divided into two types:
When degree.. the node typically performs statistical operation such computing the mean or standard deviation of the preceding node's ' output.
When degree,. 2, the node typically performs binary operation, such addition, subtraction, division between two inputs.
Nodes with only incoming edges are outputs. There must be only one output node in the DAG. The input to this final node must be scalar.
Network Search Space: Networks drawn from NASBench-201, varying only in cell structure. Macro architecture: input - conv - cell_n _n - residual (stride=2) - cell_n - residual (stride=2) - cell_*n + global average pooling. Cells: DAGs of four nodes with operations: zeroize, skip-connection, 1x1 conv, 3x3 conv, 3x3 avg-pool.
Given Dataset: CIFAR-10 classification with input shape (3x32x32) and 10-dimensional output.
Output: For each proxy, provide Description paragraph and Code demo implementing evaluate (model, inputs, targets) PyTorch. Ensure numerical stability, deterministic outputs, and move any returned tensors to CPU.
Note:
Your Zero-Cost Proxies must not be the same any existing ones (such as nwot, snip, etc.), though you may draw inspiration from them. Ensure numerical stability in your functions (avoid inf or nan during computation). Do not provide any additional explanation outside of the required description and code.
but risks premature convergence. We fix all other settings and report the average Spearman correlation between proxy scores in Table I2
Analysis on hyperparameter To assess how the discount factor and the history window size affect proxy quality, we conduct two separate ablations, one varying y over (0.5, 0.7, 0.9, 0.99}. and the other varying the history window over (1,3, 10). All other settings remain fixed. Results reported in Table [3 and Table 14
Analysis on actor-critic To evaluate the effect of varying the number of hidden layers in the actor- critic controller on proxy search performance and convergence speed, we conduct an ablation over depths of 2, 3, 4, and 5 layers. Table 15 reports the final Spearman correlation achieved after
25

---

Figure 8: Prompt used for mutation in NAS- MAS-Bench-200)
Prompt for Mutation
Your Task: Building upon the initial set of Zero-Cost Proxies. you will now perform mutation step to generate novel and distinct proxies. You should introduce creative variations into existing proxies through systematic modifications. Follow the mutation rules and constraints defined helow design 5 distinct mutated proxies:
Mutation Rules: When mutating proxy, you must choose at least one and up to four of the following mutations apply:
Input Mutation: Replace, add, or remove one type of input (e.g.. switch from convolutional layer gradients (G) to activation outputs (Z)). Consider using less commonly utilized inputs such BatchNorm statistics residual block outputs.
Operation Mutation: Change statistical operations (mean, variance, max, min) to another statistic. Substitute binary operations -.,/ with different arithmetic operations, cnsuring numerical stability.
Adjust the computational graph by altering the connections (edges) be- tween nodes, ensuring the DAG property remains valid. Introduce an additional intermediate node for richer representations.
Structure Mutation:
Aggregation Mutation: Modify how multiple layer values are aggregated (e.g., sum, mean, weighted average).
Mutated Proxy Search Space:
Each node has an in-degree 0 S degreein S 2.
Exactly one output node producing scalar value.
Inputs: Combinations of modules (conv, BatchNorm, activation layers, residual block outputs)
and properties (gradients G, weights W, oulputs Z).
Operations: Statistical (mean, variance, max, min, sum) or binary (+, (t.-;.))
Network Search Space: Networks drawn from NASBench-201, varying only in cell structure. Macro architecture: input cony cell residual (stride=2) cell n - residual (stride=2) cell. -> global average pooling. Cells: DAGs four nodes with operations: zeroize, skip-connection, Ixl conv, 3x3 conv, 3x3 avg-pool.
Given Dataset: CIFAR-10 classification with input shape (3x32x32) and 10-dimensional output.
For each proxy, provide a Description paragraph and Code demo implementing evaluate (model, inputa, targets) in Py Porch. Ensure numerical stability, deterministic outputs, and move any returned tensors to CPU.
Output:
Note: not use training. Avoid random sampling or stochastic components. The heuristic must work on general purpose image data and standard neural networks (e.g.. CNNs). Do not provide any additional explanation outside of the required description and code.
Existing Proxies:
<Existing Proxies>
convergence and the generation at which the search stabilized for each depth. The results indicate that evolution planning itself is relatively straightforward. A two-layer actor-critic controller already achieves convergence within 50 generations. Employing additional layers results in an increased number of ineffective generations, incurring unnecessary computational overhead without further improving performance (e.g., in the later stages of evolution when proxy quality improvements become minimal, continuing with fixed initialization procedures can result in diminishing returns, incurring unnecessary computational overhead).
26

---

rules and Figure 9: Prompt used for crossover in
Prompt for Crossover
Your Task: Expanding upon the previously defined Zero-Cost Proxies, your next step is to implement crossover operation to create novel proxies. The crossover process involves combining aspects of parent proxies produce distinct erossover proxies. Adhere the constraints defined below:
Crossover Rules:
To generate crossover proxies, select two parent proxies from previously defined proxies. Apply at least one and up two of the following crossover techniques:
Input Crossover: Exchange input nodes between parent proxies, such as combining gradients (G) from one proxy and outputs (Z) from another.
Operation Crossover: Merge operation nodes by replacing a statistical or binary operation from one parent with an operation from the other parent.
Structure Crossover: Blend the computational graph structures of two proxies by inter- changing node connections, ensuring the resulting DAG remains valid.
Aggregation Crossover: Combine aggregation strategies (e.g., sum, mean, weighted average) from the two parent proxies.
Crossover Proxy Search Space:
Nodes must have in-degree 0 degree_in 2.
Exactly one output node producing a scalar value.
Inputs: combinations of modules (conv, BatchNorm, activation layers, residual block outputs) and properties (gradients G, weights W, outputs Z).
Operations: statistical (mean, variance, max, min, sum) or binary ".....
Network Search Space: Networks drawn from NASBench-201, varying only in cell structure. Macro architecture: input - cony - cell_ _n - residual (stride=2) cell_n n - residual (stride=2) - cell_ an + global average pooling. Cells: DAGs of four nodes with operations: zeroize, skip-connection, 1x1 conv, 3x3 conv, 3x3 avg-pool.
Output: each proxy, provide a Description paragraph and Code demo implementing evaluate (model, inputs, targets) in PoTorch. Ensure numerical stability, deterministic outputs, and move any returned tensors to CPU.
Note: Do use training. Avoid random sampling or stochastic components. The heuristic must work on general-purpose image data and standard neural networks (e.g.. CNNs). Do not provide any additional explanation outside of the required description and code.
Existing Proxies:
<Existing Proxies>
D.2 Detailed Cross-LLM Performance
We track the evolution of average Spearman correlation over 20 generations for each LLM backbone under. Table TobleeT6)eee reports the per-generation correlation on NAS-Bench-200 for seven models.
D.3 AutoFormer Accuracy Results
We evaluate the architectures discovered by APD in the AutoFormer search space on ImageNet- 1k classification. Table repports Top-1 accuracies model achieves 76.1 Z. the small model 81.5 %. These results demonstrate that our search not only produces proxies with high correlation to ground truth also yields architectures that deliver competitive performance on a large-scale vision benchmark.
the tiny, small, and base variants: the tiny
27

---

Figure 10: Prompt for initialization in
Prompt for Initialization
Please design 5 novel Zero-Cost proxies to evaluate the representation capability of given downstream task. The final goal of each proxy to provide scalar score that reflects the performance Zero-Cost Proixes according to the following specifications, which include: proxy requirements, description of the proxy search space (defining the structure of how the proxy is computed), the search space of the networks being evaluated, the given dataset, and the output format, as described below:
Your Task:
different convolutional network architectures
given network. You should generate
Proxy Requirements:
Training-free: No gradient descent, weight updates, or learned parameters.
Efficient: Low computational cost, suitable early-stage model selection.
Deterministic: No stochastic elements randomness.
Model-sensitive: The proxy should reflect meaningful differences between models.
Proxy Search Space:
The computation of the proxy can be represented a directed acyclic graph (DAG. For any given node, its in in-degree must satisfy I degrees.. 2.
Nodes with only outgoing edges and no incoming edges are called inputs. Inputs consist of two parts: the input module and the input property. The input module refers to different types of network layers (e.g., convolutional layers, BatchNorm, activation layers, etc.), and the input property is one the following: gradient G, weights W, output feature maps Z. An input defined a combination of these components. For example, (convolutional layer gradient G)) refers the gradients of the convolutional layer's parameters. A proxy can have multiple inputs.
Nodes with both incoming and outgoing edges are referred to as **operations** which are further divided into two types:
When degreesn = 1, node typically performs statistical operation such as computing the mean or standard deviation of the preceding node's output.
When degrees. 2. the node typically performs binary operation, such addition, subtraction, or division between two inputs.
Nodes with only incoming edges are outputs. There must be only one output node in the DAG. The input this final node must be scalar.
The networks to evaluated come from TransNAS-Beinn 101, which includes variations in both cell-level and macro-level structures.
Network Search Space:
Macro architecture: img - searched backbone (composed of stacked cells) - task-specific decoder: The macro structure consists of three stages, each containing modules made of residual blocks. After each stage, downsampling and channel doubling occur. For example: Stage (Module 1) residual blocks + Stage 2 (Module 2, 3) - residual blocks - Stage 3 (Module 4) + residual blocks + task-specific head.
Cell structure: Fach cell is modeled directed acyelic graph (DAG) with six nodes. For any Vi Vy V, if j. then ey Each node represents latent feature tensor, and each edge represents a candidate operation from the following set:
zeroize
skip-connection
1 x convolution
convolution
Each cell forms the base unit of the backbone and is repeatedly stacked according to the macro-level configuration. This enables flexible network designs across different tasks such object classification, semantic segmentation, surface normal estimation, and more.
Given Downstream Task: <Given Downstream Task>
28

---

Figure 11: Prompt for mutation in TransNAA-Benchh-Il
Prompt for Mutation
Your Task: Building upon the initial set of Zero-Cost Proxies, you will now perform a mutation step to generate novel and distinct proxies. You should introduce creative variations into existing proxies through systematic modifications. Follow the mutation rules and constraints defined below design 5 distinct mutated proxies:
Mutation Rules: When mutating a proxy, you must choose at least one and up to four of the following
mutations to apply:
Input Mutation: Replace, add, or remove one type of input (e.g., switch from convolutional layer gradients (G) to activation outputs (Z)). Consider using less commonly utilized inputs such as BatchNorm statistics or residual block outputs.
Operation Mutation: Change statistical operations (mean, variance, max, min) another
statistic. Substitute binary operations (t, /) with different arithmetic operations, cnsuring numerical stability.
Structure Mutation:
Adjust the computational graph by altering the connections (edges) an additional intermediate
between nodes, ensuring the DAG property remains node for richer representations.
Aggregation Mutation: Modify how multiple layer values are aggregated (c.g., sum, mean, weighted average).
The computation of the proxy can be represented a directed acyclic graph (DAG. For any given node, its in-degree must satisfy S degrees. Nodes with only outgoing edges and no incoming edges are called inputs. Inputs consist of two parts: the input module and the input property. The input module refers to different types of network layers (e.g.. convolutional layers, BatchNorm. activation layers, etc.), and the input property is one of the following: gradient G, weights W, output feature maps Z. An input defined a combination of these two components. For example, (convolutional layer gradient G') refers the gradients of the convolutional layer's 's parameters. A proxy can have multiple inputs.
Proxy Search Space:
Nodes with both incoming and outgoing edges are referred to as **operations*** which are further divided into two types:
When degreesn = 1, node typically performs statistical operation such as computing the mean or standard deviation of the preceding node's output.
When degrees, 2. the node typically performs binary operation, such as addition, subtraction, or division between two inputs.
Nodes with only incoming edges are outputs. There must be only one output node in the DAG. The input this final node must be scalar.
The networks to be evaluated come from TransNAS-Bench 101, which includes variations in both cell-level and macro-level structures.
Network Search Space:
Macro architecture: img - searched hackbone (composed of stacked cells) -+ task-specific decoder: The macro structure consists of three stages, each containing modules made of residual blocks. After each stage, downsampling and channel doubling occur. For example: Stage (Module 1) residual blocks - Stage 2 (Module 2, 3) - residual blocks - Stage 3 (Module 4) - residual blocks - task-specific head.
Cell structure: Each cell is modeled directed acyelic graph (DAG) with six nodes. For any My V, if < j. then ey Each node represents latent feature tensor, and each edge represents a candidate operation from the following set:
zeroize
skip-connection
1 x convolution
3 x convolution
Each cell forms the base unit of the backbone and is repeatedly stacked according to the macro-level configuration. This enables flexible network designs across different tasks such as object classification, semantic segmentation, surface normal estimation, and more.
Given Downstream Task: <Given Downstream Task>
29

---

Figure 12: Prompt for crossover in TransNAS-Bench-100
Prompt for Crossover
Your Task:
Expanding upon the previously defined Zero-Cost Proxies, your next step is to implement crossover operation to create novel proxies. The crossover process involves combining aspects of two parent proxies produce 5 distinct crossover proxies. Adhere to the crossover rules and constraints defined below:
Crossover Rules: To generate crossover proxies, select two parent proxies from previously defined or mutated proxies. Apply at least one and up two of the following crossover techniques:
Input Crossover: Exchange input nodes between parent proxies, such as combining gradients (G) from one proxy and outputs (Z) from another.
Operation Crossover: Blend the computational graph structures of two proxies by inter- changing node connections, ensuring the resulting DAG remains valid.
Structure Crossover:
Blend the computational graph structures of two proxies by inter- changing node connections, ensuring the resulting DAG remains valid.
Aggregation Crossover: Combine aggregation strategies (e.g., sum, proxies.
The computation of the proxy can be represented directed acyclic graph (DAG. For any given node, in-degree must satisfy S degree,.. S
Proxy Search Space:
Nodes with only outgoing edges and no incoming edges are called inputs. Inputs consist of two parts: the input module and the input property. The input module refers to different types of network layers (e.g., convolutional layers, BatchNorm, activation layers, etc.), and the input property is one of the following: gradient G, weights W, output feature maps Z. An input defined combination of these two components.
example, (convolutional layer gradient G) refers to gradients the convolutional layer's ' parameters. A proxy can have multiple inputs.
Nodes with both incoming and outgoing edges are referred to as **operations** which are further divided into types:
When degree, = 1, the node typically performs statistical operation such as computing the mean or standard deviation of the preceding node's 's output.
When degreein 2, the node typically performs binary operation, such as addition, suhtraction, or division between two inputs.
Nodes with only incoming edges outputs. There must be only output node in the DAG. The input this final node must scalar.
Network Search Space:
The networks to be evaluated come from TransNAS-Bench.. which includes variations in both cell-Ievel and macro-level structures.
Macro architecture: img searched backbone (composed of stacked cells) task-specific decoder. The macro structure consists three stages, each containing modules made residual blocks. After each stage, downsampling and channel doubling occur. For example: Stage 1 (Module 1) - residual blocks + Stage 2 (Module 2, 3) - residual blocks - Stage 3 (Module 4) - residual blocks + task-specific head.
Cell structure: Each cell is modeled a directed acyelic graph (DAG) with six nodes. For any us, Ml V, if < j, then es Each node represents a latent feature tensor, and cach edge represents a candidate operation from the following set:
zeroize
skip-connection
1 x convolution
3 x convolution
Each cell forms the base unit of the backbone and repeatedly stacked according to the macro-level configuration. This enables flexible network designs across different tasks such as object classification, semantic segmentation, surface normal estimation, and more.
Given Downstream Task: <Given Downstream Task>
30

---

Figure 13: Prompt for initialization in AutoFormer.
Prompt for Initialization
Your Task:
Please design 5 novel Zero-Cost Proxies to evaluate the representation capability of different transformer network architectures given dataset. The final goal of each proxy to provide scalar score reflects the performance of given network. You should generate zero-cost proxies according to the following specifications, which include: proxy requirements, description of the proxy search space (defining the structure of how the metric is computed), the search space of the networks being evaluated, the given dataset, and the output format, as described below:
Proxy Requirements:
Training-free: No gradient descent, weight updates, or learned parameters.
Efficient: Low computational cost, suitable early-stage model selection.
Deterministic: No stochastic elements randomness.
Model-sensitive: The proxy should reflect meaningful differences between models.
Proxy Search Space:
The computation of the proxy can be represented a directed acyclic graph (DAG). For any given node, its in-degree must satisfy
degreein
Nodes with only outgoing edges and no incoming edges are called inputs. Inputs consist of two parts: the input module and the input property. The input module refers to different types of network layers, and the input property is one of the following: gradient G. weights W, output feature maps Z. An input is defined combination of these two components. For example. (convolutional layer gradient G) refers the gradients of the convolutional layer's parameters. proxy can have multiple inputs.
Nodes with both incoming and outgoing edges are referred to as operations, which are further divided into two types:
When degreess = 1, node typically performs statistical operation such as computing the mean or standard deviation of the preceding node's 's output.
When degrees. 2, the node typically performs binary operation, such as addition, subtraction, or division between two inputs.
Nodes with only incoming edges outputs. There must be only one output node in the DAG. The input this final node must be scalar.
Search Space:
The search space defines a set of possible architectural configurations derived from the supemet. Instead of fixed parameters. it provides discrete choices or ranges for each architectural hyperparameter, enabling the selection evaluation of optimal subnetworks.
typical search space includes the following parameters with explicit candidate values:
EMBED DIM: A set of embedding dimensions (D E (D1, D2,..., Dat), egg. (D (528, 576, 624)))
NUM HEADS: Choices attention heads (H E (Hi, Hz...., (m.)),,,m))),, eg. (H E
(9, 103).
MLP RATIO: Options for the MIP expansion ratio (Rmip (Ri, R2...., Ra)), e.g. (Rmtp 3.5, 4.0)).
DEPTH: Number of transformer blocks (L E (L1, Lz...., L.)). e.g. (L E (14, 15, 16)).
Formally, the search space A can defined as Cartesian product of discrete hyperparameter sets: A = {(D, H, Reutps L) E (528, 576, 624), H E {9, Rmut E 3.5, 4.0), L E {14, 16])
Given Dataset: The architectures will be evaluated on image classification task using IMAGE-NET. The input image shape is (3, 224, 224), and network outputs a 1000-dimensional vector.
Note:
Your Cero-Cost Proxies must not be the same any existing ones (such nwot, snip, etc.), though you may draw inspiration from them. Ensure numerical stability in your functions (avoid inf or nan during computation).
31

---

Figure 14: Prompt for mutation in AutoFormer.
Prompt for Mutation
Your Task: Given 5 existing Zero-Cost proxies. mutate each of these proxies to create 5 new ones (one mutated version per original proxy). Each mutated proxy should still be training-free, computationally efficient, deterministic, input-aware, and model-sensitive The final goal is produce updated or extended training-free proxies that remain valid for early-stage model selection but differ in approach from the original set.
When generating each mutated proxy, you may:
Slightly alter or combine existing nodes the computation DAG
Introduce new operations input types (still abiding by the in-degree constraints)
Change the nature the final scalar computation
Ensure numerical stability (avoid division by zero, log of non-positive numbers, etc.)
Proxy Requirements:
Training-free: Do not any form of parameter updates.
Efficient: Maintain low overhead in computation.
Deterministic: Must produce consistent outputs for the same model and data.
Model-sensitive: Should vary meaningfully with different subnetworks sampled from the supernet.
Proxy Search Space: All mutated proxies should still form directed acyelic graph (DAG) in their computation. For any given node:
In-degree Constraint: 0 degreesn 2.
Consist network module. Tied to property (W, G. 2). may now choose to introduce new input properties that are still relevant to a training- free scenario (e.g., a logarithm of weights. or combined dimension of feature maps), but be sure to keep them deterministic and valid.
Input Nodes:
Operation Nodes (single-operand); Perform statistical transformations (c.g., mean, vari- ance, norms, etc.).
Operation Nodes (binary): Perform pairwise operations (e.g., addition, multiplication, ratio).
Output Node: Must be single scalar value (with outgoing edges).
Search Space:
The search space defines a set of possible architectural configurations derived from the supemet. Instead of fixed parameters, provides discrete choices or ranges for each architectural hyperparameter, enabling the selection evaluation of optimal subnetworks.
typical search space includes the following parameters with explicit candidate values:
EMBED DIM: A set of embedding dimensions (D E (Di, D2.... D.)....DD)). e.g., (D E (528, 576, 624)).
NUM HEADS: Choices attention heads (H E (H1, Hy...., Hm)),, eg. (H E
[9, 103).
MLPP RATIO: Options for the MLP expansion ratio (Rmip (Rs, Rz..... R&)), e.g., (Rmtp 3.5,, 4.0)).
DEPTH: Number of transformer blocks (L E (L1, Lz...., L.)), e.g. (L E (14, 15, 16)).
Formally, the search space A can defined as Cartesian product of discrete hyperparameter sets: A = {(D, H, Reutps L) E (528, 576, 624), H E {9, Rmut E (3.0, 4.0), L e {14, 16]}
Given Dataset: The architectures will be evaluated image classification task using IMAGE-NET. The input image shape (3, 224, 224), and network outputs a 1000-dimensional vector.
Note: Your Zero-Cos Proxies must not be the same any existing ones (such nwot, snip, etc.), though you may draw inspiration from them. Ensure numerical stability in your functions (avoid inf or nan during computation).
Existing Proxies:
<Existing Proxies>
32

---

Figure 15: Prompt for crossover in AutoFormer.
Prompt for Crossover
Please design 5 novel Zero-Cost Proxies to evaluate the representation capability of different transformer network architectures on given dataset, using crossover-based approach. This means each of your new proxies should be created by comhining at least two distinct ideas, components, or sub-nodes from hypothetical or existing parent proxies. For instance, you have two parent proxies A and with specific inputs or operations, you must form a new child proxy by merging relevant parts from each parent, ensuring the final proxy remains logically consistent and provides scalar score that reflects the performance of given network. You should generate Zero-Cost Proxies according to the following specifications, which include: proxy requirements. description the proxy search space, the search space the networks being evaluated, the given dataset, and the output format, as described below:
Your Task:
Proxy Requirements:
Training-free: Do not use any form parameter updatcs.
Efficient: Maintain low overhead in computation.
Deterministic: Must produce consistent outputs for the same model and data.
Model-sensitive: Should vary meaningfully with different subnetworks sampled from the supernet.
bossoverr-bseed Each proxy must inherit or merge conceptual "genes" (e.g., particular input sources operations) from least two parent proxics, forming new, valid proxy design.
Proxy Search Space: All mutated proxies should still form directed acyclic graph (DAG) in their computation. For any given node:
In-degree Constraint: 0 S degreeso S 2.
Consist of network module. Tied property (W, C. 2). You may choose to introduce new input properties that are still relevant to training- free scenario (e.g.. a logarithm weights, or combined dimension of feature maps), but be sure to keep them deterministic and valid.
Input Nodes:
Operation Nodes (single-operand); Perform statistical transformations (e.g., mean, vari- ance, norms, etc.).
Operation Nodes (binary): Perform pairwise operations (c.g. addition, multiplication, ratio).
Output Node: Must be a single scalar value (with no outgoing edges).
Search Space:
The search space defines of possible architectural configurations derived from
the supemet. Instead of fixed parameters, provides discrete choices or ranges for each architectural hyperparameter, enabling the selection or evaluation of optimal subnetworks.
A typical search space includes the following parameters with explicit candidate values:
EMBED DIM: A set embedding dimensions (D (D1, D2. ..... D.))) egg. (D (528, 576, 621))-
NUM HEADS: Choices for attention heads (H (Hi, Ha,..., ..... H,))). egg. (H E (9, 103).
MLP RATIO: Options for the MLP expansion ratio (Rmup (Ri Re...,,aa RR)), e.g., (Rmip {3.0, 4.0)).
DEPTH: Number of transformer blocks (L E {11, L2. L.)). e.g., (L E {14, 15, 16)).
Formally, the search space A can be defined Cartesian product of discrete hyperparameter sets: {(D, H, Rmmp, L) E (528, 576, 624), {14, 15,16111)) 16}}
10,, Rmii (3.0,3.5, 3.,, 4.0), L e
Given Dataset: The architectures will be evaluated on image classification task using IMAGE-NET. The input image shape (3, 224, 224), and the network outputs 1000-dimensional vector.
Note:
Your Zero-Cosi Proxies must not be the same as any existing ones (such as nwot, snip, etc.), though you may draw inspiration from them. Ensure numerical stability in your functions (avoid inf or nan during computation).
33

---

Table 11: Average proxy correlation over 20 generations for different prompt operations on NAS- Bench-201. Full denotes the complete operation configuration, incorporating both mutation and crossover.
Generation No Crossover
No Mutation
Full
0.444
0.193
0.246
0.533
0.237
0.470
0.533
0.349
0.617
0.533
0.439
0.628
0,541
0,463
0.662
0,541
0,463
0.662
0.541
0.466
0.705
0.572
01,463
0.705
0.627
0.547
0.754
10
0.640
0.547
0.794
11
0.640
0.547
0.800
12
0.640
0.669
0,800
0.640
0.713
0.800
14
0.761
0.800
15
0.640
0.761
0.800
16
0.680
0.780
0.800
17
0.714
0.780
0.803
18
0.739
0.780
0.803
19
0.739
0.780
0.813
20
0.758
0.780
0.813
Table 12: Effect of population size on the average Spearman correlation of the full prompt configura- tion over 20 generations.
Population Size
Avg. Spearman P
0.6023
0.6837
3
0.7541
0.8013
5
0.8124
10
0.7895
Limitations
While APD markedly advances training-free NAS, several caveats remain. We group them here for clarity.
Black-box optimization Because APD delegates the generation of every candidate proxy to LLM, the token-level reasoning that maps a prompt to executable code is not observable, raising black-box concerns. We acknowledge that the LLM itself remains opaque. Nevertheless, APD is designed so that its outer optimization loop is inspectable. The LLM's outputs are evaluated by a public fitness function, and the aitor-critic scheduler is trained only this scalar reward. All decisions that influence the search trajectory therefore pass through a measurable, task-specific signal rather than hidden logits. Thus, while some inner L.LM reasoning remains inaccessible, the information APD exposes is already adequate for transparency and troubleshooting, so the residual opacity of L.M is unlikely to undermine the method's value.
Need for a small ground-trure -trun subset Computing fitness requires baseline accuracies for roughly 2-3% of each benchmark, which is standard in NAS studies and inexpensive compared with full training. Although small ground-truth subset is indispensable for APD, we have empirically shown that the ZCPs obtained with no significant differences. Consequently, even for previously unseen NAS tasks, the overhead introduced by APD remains minimal and well within acceptable bounds.
such subset continue generalize when transferred to search space
Prompt-context length growth on very large searches The generator receives most C, proxies per step, and both prompt and context are explicitly bounded to fit the LLM window. Empirically the
34

---

Table 13: Effect discount factor y on average Spearman correlation. Best value is highlighted in yellow.
Discount Factor y
Avg. Spearman p
0.50
0.7124
0.70
0,7823
0.90
0,8137
0.99
0.8015
Table 14: Effect history window size on average Spearman correlation. Best value is highlighted in yellow.
History Window
Avg. Spearman P
0.6987
0.8042
0.8259
10
0.7791
combined length stayed under 6k tokens specific population size, and we have to down-sample the context if future domains require longer code.
Security and stability when executing LLM-generated code Each proxy is arbitrary code and could, in principle, issue unsafe or resource-hungry calls. Therefore, code has to be run inside shource-capped sandbox.
Discovered AZPs
F.1 Proxy for NAS-Bench-201
To demonstrate the effectiveness of our approach, present three representative proxies that achieved strong performance on MAS--ench-2201, which are illustrated in Figures 16 17
and 18
F.2
Proxy for TransNAS-Bench-
We present the zero-cost proxies discovered on TransNAS-Bench-UU by APD in Figure 19 20 and
2
F.3
Proxy for AutoFormer
In Figure 22
and 24
we present the zero-cost proxies discovered through APD on AutoFormer.
G Searched Architectures
In the DARTS search space, the optimal architecture discovered by APD is shown in Figure 27 with its normal and reduction cells visualized in Figures (3 [6] respectively. In the AutoFormer setting, the architectures found for the tiny, small, and base variants are presented in Figure 29
and 30 respectively.
H Related Work
H.1 Neural Architecture Search
Neural Architecture Search (NAS) aims to seek nigh-performance neural networks an automatic manner. Early works cast the problem as black-box reinforcement learning or evolutionary optimiza- tion. NASNet [77] and AmoebaNet (53) trained thousands of architectures from seratch on large GPU clusters, demonstrating the promise of search but at prohibitive cost. Subsequent one-shot methods
35

---

Table 15: Impact of A2C network depth on proxy search quality and convergence.
Hidden Layers
Final Spearman
Convergence Generation
2
0.8216
42
3
0.8367
79
4
0.8175
96
0.8230
103
Table 16: Evolution of average Spearman correlation over 20 generations on NAS-Bench-201 for different LLM backbones.
Gen.
Claude 3.7
Gemini 2.0 Flash
GPT-4o
Deepseck V3
Qwen Plus
Grok
Llama4
0.0853
0,0228
0.1293
0.0000
0.5040
0.4290
0.1091
0.1720
0.0712
0.1293
0.1319
0.5040
0.5027
0,1816
0.1720
0.1292
0.1293
0.1319
0,5040
0.5560
0,3909
0.1720
0.3230
0,1293
0.1319
0.6553
0,5560
0.4312
0.2126
0.3807
0.1293
0.1319
0,7087
0.6033
0,4312
0.2126
0.3807
0.1293
0.1319
0,7304
0.6212
0.5036
0.3189
0.3962
0.2669
0.1319
0,7304
0.6714
0.6278
0.4969
0,5518
0.2669
0.1319
0.7304
0,6806
0.6531
0.6109
0.4071
0.1605
11
U,7713
0.6869
0,7411
0.6531
0.6303
0.6933
0.1605
12
0,7713
0.6869
0,7411
0.6331
0.6303
0.6933
0.1605
0,7713 14
0.6869
0.7744
0.7944
0.6303
0.6933
0,2809
0,7713
0.6988
0.7898
0.7944
0.6303
0.7081
0.2809
0.7944
0.7395
16
0,7728
0.7317
0,8090
0.8024
0.6439
0.7395
0.4225
17
0.8114
0.7317
0,8090
0,8024
0.6994
0.7395
0.4348
18
0,8114
0.7517
0,8090
0.8024
0,7212
0,7395
0.5598
0.8114
0.7517
0,8090
0.8024
0.7212
0.7697
0.6495
20
0.8114
0.7522
0.8110
0.8024
0.7212
0.7697
0.6973
reduce cost by sharing parameters into a single supernet. ENAS 75
introduces weight sharing with
recurrent controller, while DARTS 20 relaxes the discrete search space differentiable mixture, enabling gradient- updates but inheriting optimization bias and collapse issues that spawned a line of robust variants. However, performance estimation remains the key throughput bottleneck (44).
H.2 Zero-Cost Proxies
NAS traditionally relies on partial or full training to judge candidate models, making large-scale exploration prohibitive. Zero-Cost Proxies (ZCPs) address this bottleneck by estimating an archi- tecture's potential from its randomly initialized weights in significantly less time and constitute standard component of training- free NAS. The earliest ZCPs exploit gradient saliency. SNIP [3 evaluates weights with the first-order loss sensitivities, GraSP extends the idea second- information, and SynFlow [I] removes data dependence by propagating an all-ones input, yielding strong correlations deep convolutional spaces. A second strand replaces gradients with cheap statistics of activations or parameter layouts. NWOT [44 measures neural-tangent-kemee overlap, Zen-score [37] evaluates Jacobian log-determinants, while ZiCo (35) shows that even simple norms such parameter count FLOPs can outperform many hand-crafted heuristics once appropriately normalized.
Despite steady progress, two limitations persist. Firstly, all existing proxies are manually specified. Their functional form rarely transfers intact across tasks (e.g., from MAS-Beech-221 CIFAR1O to TransNAS Been---B--h---- 101 autoencoding). Secondly, correlation improvements have largely plateaued, and gains often come at the cost of additional forward passes or Jacobian computations.
H.3 Automatic Proxy Discovery
Automatic Proxy Discovery aims to relieve experts from hand-crafting ZCPs by treating the proxy itself as the search object. The most systematic effort so far is Auto-Prox (44 It encodes proxy as a computation graph whose nodes are primitive arithmetic or operations (e-g., exp, log).
evolutionary algorithm with elitism preservation mutates these graphs to maximize joint correlation
36

---

Table 17: The comparison results on the Autoformer search space in the ImageNet dataset. denotes
the results reported by [28
Models
#Param (M)
FLOPS (B)
(o)
(8)
Model Type
Design Type
Years
GPU Days
Tiny search space
ResNer- 18
26
11.7
1.8
12.5
CNNs CNNs
Manual
2015
MobileNet-V
Deit:11
59
5.7
1.2
72.2
91.1
Trunsfommer
Manual
2015
TNT-T:
14
73.9
91.9
Transformer
Manual
2015
CPVT-I: [V]
6.0
74,9
92.6
Transformer
Manual
2015
PVT-Tiny
75.1
Transformer
Mannal
2015
AutoFornee-
74.7
92.6
Transformer
Auto
CVPR2021
GLiF-Ti [0
7.2
14
76.3
Hybrid
Auto
1CCV2021
N/A
56
Transformer
BCCV2022
TT-TAS-Ti
75.3
92.8
Trunsformer
Auto
CVPR2022
Auto-Prox
64
75.6
Transformer
Auto
AAA12024
0.1
CVPR2024
CET-TAS (Ours)
8.4
1.9
76.1
93.1 Small search space
Transfommrr
Auto
0.25
ResNecrS0" 26
25.6
4.1
80.2
CNNs CNNs
Manual
2015
RegNetY-IGF Finl
20.6
79.4
Manual
2015
DeT-S 50
22.1
79.9
Transformer
Manual
2015
VIT-S/16 [19
22.1
4.7
78.8
Transformer
Manual
2015
PVT-Small
24.5
3.8
79.8
Transformer
Manual
2015
Swin-T
29.0
81.3
Transformer
Manual
2015
TNT-S 24
23.8
5.2
81.5
95.7
Trunsformer
Manual
2015
CPVT-S
Transformer
Manual
T2T-VIT 1-14
12
21.5
81.7
Trinsformer
Mannal
2015
AutoFormer-S
22.9
5.1
81.7
95.7
Transformer
Auto
CVPR2021
24
Hyhrid
VITAS-F 13
27.6
6.0
80.5
95.1
Transformer
Auto
BCCV2022
32
TF-TAS-S
95.8
Transformer
Auto
CVPR
AZ-NAS
23.8
5.1
82.2
Transformer
Auto
CVPR2024
CET-TAS (Ours)
30.9
6.3
81.5
95.3
Transformer
Auto
0.25
objective. Auto-Prox, however, specialized to ViTs, Its search space hard-codes transformer- specific primitives. Similar variants, such as EZNAS K and Auto-DAS [57] follow identical graph enumeration but remain tied to fixed settings, respectively, limiting cross-domain portability and placing heavy reliance human expertise.
H.4
LLM for Neural Architecture Search
Recent research has explored leveraging large language models to assist or even automate the neural architecture search process. EvoPrompting (9] pioneers this direction by treating NAS as code-level program synthesis task, where LLMs evolve architecture generation programs through an evolutionary prompt-tuning mechanism. By iteratively mutating and refining code snippets that specify archi- tectural design choices, EvoPrompting enables LLMs to autonomously discover performant neural architectures without explicit gradient-based optimization. Building this paradigm, LLMatic 47] combines LLM-driven architecture generation with quality-diverse optimization, allowing the search process balance exploration and exploitation more effectively. Through LLM-guided mutation and recombination, LLMatic produces diverse yet high-performing architectures across vision and reinforcement learning benchmarks. Collectively, these works demonstrate that L.LMs hold substantial potential for application in neural architecture search, suggesting that language-driven reasoning can serve as viable mechanism for exploring the search space.
explorati
However, a critical limitation these approaches lies in their reliance on evolutionary operations and crossover progressively refine architectures. This paradigm inevitably necessitates the instantiation, training, and subsequent evaluation of each generated model to ascertain its performance, a process that is notoriously resource-intensiv.. The iterative cycle of generating population of architectures, training them for even few epochs, and measuring their fitness creates a significant computational bottleneck, rendering the search process both time-consuming and expensive. Consequently, the practical application of such methods limited, particularly in scenarios demanding efficient and lightweight architecture discovery.
37

---

Figure 16: zero-cost proxyl discovered by APD in the WMS-Bench-200
import torch
import torch.
def proxyl (model, inpute, targets):
bn ranks = []
ratios []
hooks []
def bn bn_hook (module, inp, out):
if isinstance (out, torch Tensor):
B, C, Н, N shape mat out (Be C, -1) permuutt (0, 2, reshape (-1, frob_norm torch.
(mat, ord=' fro') **2
spec_nore forch.linalg.natrii...
(mat, ord=2) **2
stable_rank = frob_nors: (spec_norm + 1e-6) bn_ranks append(stablee
for layer in model.modules
if isinstance (layer, mn. Bm..ccchhooo2d):
hooks append
forward_hool booo (bn_hook))
elif isinstance (layer, Convodd)
weights layer boyer.weight
1i_norm = neights.abs ()s sum (dim= (1,2,3)) mean ()
12_norm = weights norm (p=2, dim= (1 (1,=11,,,55 (1) mean ()
ratios append ((11_norm / 12_norm) ittm ())
with torch no_grad :):
model (inputs)
for hook in hooks: hook. remove))
bn sum torch. stack (bn_rrbar (bb_ranks)) sum () ittm () if ba_ksnks else 0.0
ratio_ = sum (ratios) if ratios else 0.0
return bn_ sum ratio_sur sum
38

---

Figure 17: zero-cost proxy2 discovered by APD in the MAS-Bench-201.
import torch
import torch.m. nn as nn
def proxy2 inputs, targets):
ratios 0
apatial_stds 0
activations 0
def hook fn (module, input, output):
activations append (moouul... weight detach output detach ))))
hooks = []
for layer in if isinstance (layer, mn. Conv2d)
hooks append (layer registerrfoorardd.eee... forward_hoold (hook_fn)
with torch no_grad
model (inputs)
for hook in hooks:
hook. remove)
for weight, act activations:
peight_norm torch, norm (weight, p='fro')
act_norm torch norm(act, p='fro') if act_norm != 0:
ratioa. append ((weight_norm act_norm) itte
spatial_std act act (dim-11 keepdim-True) spatial_stde (spatial
mean () item ())
avg_ratio sum (ratios) len (ratios) if ratios else 0.0 avg_std sum sum (spatial_stdss) len (spatiial__ttd)) if apatial_stds
else 0.0 return -2 * (avg_ratio avg_std) (avg_ratio avg_std 1e-10)
39

---

Figure 18: zero-cost proxy3 discovered by APD in the NAS-Bench-201.
import torch
import torch. nn as nn
def (mode, inputs, targets);
activations []
def hook can (module, input, output):
activations. append (output detach () flatten (2))
hooks []
for layer in modules O)
if isinstance (layer, mn. Conv2d)
hooks. append (layer register forward_ hook (hook_cnn)
with torch no_grad
model (inputs)
for h in hooks: removee)
scores [] for act in activations:
act mean (dim=-1, keepdin=True)
channel_mean act.abs () mean (dim=(0, 2)) if baaanel_nean.eanr
> 1e-6:
cv = channel_mean s) / channel_ mean scores append itv.itmie
return -sum (scores) / len (scores) scores else 0.0
Figure 19: zero-cost proxyl discovered by APD in the TransNAS-Bent 101.
import torch
import torch functional F
def proxyl (model, inputs, targets): requires-grad.
with torch. enable_grad
output model(nnputs)
loss cross_eenroop (output, targets)
grad clean = torch. autograd grad (loss, inputs) [0]
output model (inputa 0.01)
loss cross_entroppy (output, targets) grad_noisy = torch. autograd grad (loss, inputs) [0]
inputs requires_grad. (False)
cos_sim wwc.oinnssimileeeee (grad_ flatten (), grad_noisy.
flatten (), dim-0)
return cos_sim.cpp ()
return cpore.cpu()
40

---

Figure 20: zero-cost proxy2 discovered by APD in the CranNNAS-Benchh
import torch
def proxy2(nodel) imputs, targets):
with torch. no-grad
clean_ model (inputs)
scaled_inputs inputs 1.01 Small fixed scaling ocaled_output model (scaled_inputs).
logit_diff torch norm (clean) (ctput scaled_output, p-2, dim-1) mman ()
return logit_ cpu () item
Figure 21: zero-cost proxy3 discovered by APD in the TransNAS-Benn See----Bec-O00- 101.
import torch
import torch.m. mn. functional as F
def (model inputs, targets):
with torch torch.no_gat ()
clean_output model model (inputs)
noise = torch. aarch.zeros.. zeroeeerossaaaa like (inputs) for in Vange(inputa, shape [2])
for in range noisel:, :, i, j] 0.01 ((i j) % 2
shape [3]):
1)
noisy_output = model (inputs + noise)
clean_probs F. (c
dim-1)
noisy_probs sof tma (noisy dim-1)
kl_div F. kl_ (clean_probs 0 noisy.proba. reduction= batchmean ')
kl_div. cpu item () return score.cpu()
Figure 22: zero-cost proxyl discovered by APD in the AutoFormer.
import torch
import nn. functional as F
def heuristic_8 (model, inputs, targets, loss_fn):
with torch. go_grad()
output model (inputs)
noise_levels [0.01, 0.05, 0.1]
kl. values [] clean_probs F. moftmaxx outpuu,, dim=1)
for level in noise levels_
noise sorch.randn_1ee (inputs) level noisy_output model noise)
noisy-probs sofofmaa (noiyy dim=1)
kl_div = F F. ()g(), noisy.probs,
kl_values append
acore = torch, prod (torch. stack (kl, (kl_values) ** (1/len kl.values)) return cpu item
41

---

Figure 23: zero-cost proxy2 discovered by APD in the AutoFormer.
import torch
import torch.nn nn functional as F
def proxy)(model, inputa, targeta, loss_fn):
with torch. norch.no.grad()
# Input perturbation component
noise = torch. randn_like (inputs) 0.01
perturbed_inputa = imputs noise
original_outputs model (inputi))
perturbed_outputs model (peellpperuroood.nnut.
diff torch. norm (original_ouutpute perturbed p-2, dim-1).mest
# divergence component original_softmet original. softmax (dim=1)
perturbed_softmax - perturbed_ outputs softmax (dim=1)
kl_div = F. F.kl_di (original_ soffinaa_sottaa log (), perturbed
reduction= batchmean')
return (diff xl_div) cpu () item ()
Figure 24: zero-cost proxy3 discovered by APD in the AutoFormer.
import torch
def proxy(mmodel, inputs, targets):
with torch. no_grad
alean_outpor model (inputs)
scaled_inputa inputs * 1.01 Small fixed scaling ocaled_output model (scaled_inputss)
logit_diff torch. norm (clean_outpal scaled_output, p-2, dim-1) mean ()
return logit_ diff. cpu () item()
sep_conv 3x3
c_(k)
_xx5
c_(k-2)
sap
3
donv_
sep_corv__e0
c_(k-1)
Sep_ 383
sep.conv 3x3
Cop_ooon_GGa 5x5
Figure 25: Normal cell searched by APD.
42

---

c_(k-11
skip connect
3
sep_cott
sep_ cony 3x3
c_(k)
dil conv 3x3
c_(k-2)
max _ool_3x33 3x3
conv
dil obnv
Skip conpeca
Figure 26: Reduction cell searched by APD.
Figure 27: Optimal structure found in the DARTS search space by APD.
(nnomapeetor sep sep___ppp_co_33' 0), ('sep_conv_5x5', 1),
O'sep_cony_3x3', 0), ('sep_conv_3r3', 1),
('sep_conv_5x5', 0), ('sep_conv_5x6: 0),
('sep_conv_3x3" 3), ('sep_conv_3x3- 0)1, normal_ (2, 6), 0), ('dil_conv_3x33, 0),
reduce=(("so__onv__3
('dil_cony_3x3) 0), ('skip_connect', 0),
("max_pool_3x3', 0), [dil__oov_5xx5 3)
('sep_cony_3x3', 2), ('skip_connect', 1)1, reduce conatraecn (2, 6))
Figure 28: Subnet discovered in the AutoFormer tiny search space by APD.
RETRAIN:
MLP :
3.5
3.5
3.5
3.5
4
4
3.5
3.5
4
4
4
DEPTH 12
240
43

---

Figure 29: Subnet discovered the AutoFormer small search space by APD.
RETRAIN: :
:
3.5
3.0
4.0
4.0
4.0
3.0
3.0
3.0
4.0
4.0
4.0
3.0
4.0
3.0
NUM HEADS:
7
7
5
DEPTH:
EMBED_ DIM: 448
44

---

Figure 30: Subnet discovered in the AutoFormer base search space by APD.
RETRAIN: :
MLP RATIO:
3.0
3.0
3.0
4.0
4.0
3.5
3.0
4.0
3.5
3.0
4.0
3.0
3.0
4.0
3.0
NUM HEADS:
10
9
10
10
9
9
9
10
10
DEPTH: 15
EMBED_DE DIM: 576
45


Reviews:
[{"review_id": "WmcXGyJ9SU", "reviewer": "Reviewer_n7rC", "summary": "The paper presents a novel Automatic Proxy Discovery (APD) framework driven by LLM, which tries to discover optimal zero-cost proxies (ZCPs). Novelty of the paper lies in the how an actor-critic model acts as an evolutionary decision-maker. The actor-critic model does that by receiving the fitness scores from the evaluator that evaluates the proxies, and then tries to learn a policy that would maximise this fitness score.", "questions": "I believe the above-mentioned points are easily fixable. A better citation method can be used. One tricky change would be to adjust the Related works in the paper. I believe a lot from section 4 and 5 can be reduced. I believe 3.1 and 3.2 are a bit redundant. Can you make it a single section out of both of them? I will improve my scores if these issues are resolved, and I believe that they are easily resolvable. My current score is 4, but I am willing to make it 5.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\nThe idea is quite novel, though the direction where we are applying some sort of search/learning algorithm over the LLM output is the current trend. I like how an actor-critic model evaluates the generated ZCPs. The paper has thorough experiments and ablation studies to demonstrate the importance of the pipeline. The explanation of the methodology is extensive.\n\n**Weaknesses**:\n\nThe paper has quite a few issues related to writing:\n\n- There are only citations with the name used by the authors. You have to use number-only citations when using them during the paragraph.\n- Related work is in the supplemental material, which is a big issue, in my opinion. Many readers want to read literature related to your work to know more about your field. \n- Related work in the supplemental material is poorly written. You have to cite the works that introduced LLMs in NAS, such as EvoPrompting [1] and LLMatic [2]. The work takes a lot of inspiration from both of the works, therefore the need to cite them. \n\n[1] Chen, A., Dohan, D. and So, D., 2023. Evoprompting: Language models for code-level neural architecture search. Advances in neural information processing systems, 36, pp.7787-7817.\n\n[2] Nasir, M.U., Earle, S., Togelius, J., James, S. and Cleghorn, C., 2024, July. LLMatic: neural architecture search via large language models and quality diversity optimization. In proceedings of the Genetic and Evolutionary Computation Conference (pp. 1110-1118).", "was_revised": true, "final_justification": "I have read other reviews which suggest that the paper should be rejected but I believe this is a good paper and should be added to the field.", "rebuttal": "Thank you for the helpful and insightful review, which is very helpful for us to further improve this paper. Next, we will answer your questions one by one, and we hope this will improve your acceptance of the paper.\n\n**Q1**: Concern about citations.\n\n **A1**: Many thanks for your comments! All citations have been thoroughly updated to number-only format (e.g., [1], [2]) in the revised manuscript.\n\n**Q2**: Concern about Related work.\n\n**A2**: Many thanks for your comments! First, we have added more related work in our paper, i.e., EvoPrompting [1] and LLMatic [2]; these works will enhance the readers' understanding of our work and the field. Second, we have revised the \"Related work\" to improve the clarity.\n\n**Q3**: I believe a lot from Section 4 and 5 can be reduced. I believe 3.1 and 3.2 are a bit redundant. Can you make it a single section out of both of them? I will improve my scores if these issues are resolved, and I believe that they are easily resolvable. \n\n**A3**: Many thanks for your comments! We sincerely thank the reviewer for this valuable suggestion to improve our paper's structure. Following this advice, we have merged the former Sections 3.1 and 3.2 into a single, cohesive section to eliminate the redundant description. To demonstrate specifically how this was done, we now present the conceptual overview (previously in Section 3.1) and the corresponding technical details (previously in Section 3.2) together for each component. For instance, the high-level description of the Fitness Evaluator is now immediately followed by its mathematical formulation. The new, merged text for this component is as follows:\n\nTo fulfill the goal of designing ZCPs for the training-free NAS, APD utilizes LLMs to automatically generate proxies by evolving both natural language descriptions and corresponding code. In addition, we propose an actor-critic RL controller sampling appropriate prompt strategies to guide the evolution, aiming to optimize the correlation between proxies and final model performance. The overall framework of APD is depicted in Fig. 4, which consisting of three main components as follows:\n\n**Proxy Candidate Generator.** LLM in APD serves as proxy candidate generator. Carefully structured prompts enable it to synthesize new ZCPs or refine existing ZCPs. \n\nLet $\\mathcal{P}$ denote the set of valid proxies ...\n\n**Fitness Evaluator.** The fitness evaluator swiftly quantifies each candidates ZCP by computing its Spearman correlation with ground-truth accuracies on the given NAS benchmark A (e.g., NAS-Bench-201). For a benchmark set $\\mathcal{B}= \\{(a_i, p_i)\\}_{i=1}^m$ of architectures and ground-truth accuracies under dataset D. The fitness assigned to f is \n\n$\\phi(f)=\\rho(f(a), p)-\\beta \\text{cost}(f)$\n\nwhere ...\n\n**RL Evolution Scheduler.** To render the proxy evolution strategy learnable and capable of converging efficiently toward optimal ZCPs, APD introduces an actor–critic module that serves as the evolutionary decision-maker. The actor–critic treats the fitness score returned by the evaluator as its reward and learns a policy that maximizes this signal, thereby jointly optimizing both the evolving set of ZCPs and the evolution strategy itself. \n\nIn a given search space $\\mathcal{A}$ ...\n\nTo accelerate convergence toward  high correlation proxies while maintaining effective exploration in search space $\\mathcal{F}$, APD introduces a light-weight Actor-Critic as evolution scheduler that governs every evolutionary step. At generation $t$, ...\n\nAs shown above, the conceptual explanation now flows directly into the corresponding technical details. We hope this revision can resolve the redundancy issue and provide a much clearer, more streamlined presentation of our methodology. We are grateful for the feedback that prompted this significant improvement.\n\nFor Section 4 and 5, we present the experiments in our paper. First, Section 5 is a LaTeX typo (\\section{Generalizability on OoD-ViT-NAS-Ti} - \\subsection{Generalizability on OoD-ViT-NAS-Ti}), it should belong to Section 4. Second, we promise to reduce the redundant parts in Section 4 and 5 in the final version.\n\n**Q4**: Concern about NeurIPS Paper Checklist\".\n\n**A4**: Many thanks for your comments! We have deleted the instruction block of \"NeurIPS Paper Checklist\" in the revised manuscript.\n\n[1] Chen, A., Dohan, D. and So, D., 2023. Evoprompting: Language models for code-level neural architecture search. Advances in neural information processing systems, 36, pp.7787-7817.\n\n[2] Nasir, M.U., Earle, S., Togelius, J., James, S. and Cleghorn, C., 2024, July. LLMatic: neural architecture search via large language models and quality diversity optimization. In proceedings of the Genetic and Evolutionary Computation Conference (pp. 1110-1118)."}, {"review_id": "2uf0RucZiU", "reviewer": "Reviewer_sUV8", "summary": "This paper proposes a new framework, APD (Automatic Proxy Discover), that uses LLM to automatically generate and optimize zero-cost proxy tasks in NAS (Neural Architecture Search). The method trains LLM with RL (reinforcement learning) to improve LLM’s ability to generate zero-cost proxy tasks. The experiments show that APD outperforms previous traditional methods in both performance and efficiency.", "questions": "1. How do you choose T_max? Can you determine when the search has sufficiently converged?\n2. How do you compare with other LLM-based NAS work? \n3. How stable are the discovered proxies if you slightly reword the prompts or switch LLM versions?\n4. Can you run APD under a limited budget of LLM API calls?\n5. How does APD perform with lightweight or open-source models (e.g., Qwen3-8B)?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n1. The paper is well-written and easy to follow.\n3.  The actor-critic part of this method is interesting; it fills the gap between NAS (domain-specific task) and general-domain LLM without requiring retraining or fine-tuning the LLM.\n4. The paper conducts comprehensive experiments, APD outperforms SOTA NAS methods on five search spaces in both final accuracy and proxy tasks ranking correlations.\n\n**Weakness**\n1. The paper didn’t compare or comment on previous efforts on using LLM for zero-shot proxy task generation, such as [1 – 5]. \n2. There is also a ZCP work [6] that was cited by ZiCo and seemed to outperform ZiCo. Why didn’t you compare your results with [6]?\n3. The paper didn’t discuss the cost of LLM API calls.\n4. APD is technically not training but searching. Actor-critic models are trained with a specific LLM on a specific search space. This means when it comes to a new NAS search space, the entire APD process needs to be run again. Thus, the cost of using APD is much more than using a traditional, manually designed zero-cost proxy. In conclusion, while APD proposes a novel method to automatically discover new proxy tasks, the total cost of NAS using APD may actually increase.\n5. Risk of Data Containment. The LLM may have knowledge of the recent SOTA zero-cost proxy tasks on mainstream search spaces and benchmarks. When it performs the proxy task discovery, the knowledge might help LLM generate better results. This leaves concerns when this method is deployed on unseen tasks.\n\n[1] Z. Yan, Y. Qin, X. S. Hu, and Y. Shi, “On the viability of using LLMs for SW/HW co-design: An example in designing CiM DNN accelerators,” arXiv preprint arXiv:2306.06923, 2023.\n[2] H. Wang, Y. Gao, X. Zheng, P. Zhang, H. Chen, and J. Bu, “Graph neural architecture search with gpt-4,” arXiv preprint arXiv:2310.01436, 2023.\n[3] W. Li, X. Su, S. You, F. Wang, C. Qian, and C. Xu, “Diffnas: Bootstrapping diffusion models by prompting for better architectures,” arXiv preprint arXiv:2310.04750, 2023.\n[4] M. Zheng, X. Su, S. You, F. Wang, C. Qian, C. Xu, and S. Albanie, “Can gpt-4 perform neural architecture search? ” arXiv preprint arXiv:2304.10970, 2023.\n[5] Qin, R., Hu, Y., Yan, Z., Xiong, J., Abbasi, A. and Shi, Y., 2024, January. Fl-nas: Towards fairness of nas for resource constrained devices via large language models. In 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC) (pp. 429-434). IEEE.\n[6] Li, Yuhong, et al. \"Extensible and efficient proxy for neural architecture search.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "was_revised": true, "final_justification": "I thank the authors for thoroughly answering all my questions. I'm happy to raise my overall rating to board line accept. Thanks.", "rebuttal": "Thank you for the helpful and insightful review, which is very helpful for us to further improve this paper. Next, we will answer your questions one by one, and we hope this will improve your acceptance of the paper.\n\n**Q1**: Comparison with task [1-5]. \n\n**A1**: Many thanks for your comments. First, we will cite the works [1-5] in our paper, although the task and benchmark of works [1, 2, 3, 5] are totally different with our APD, those works are important developments in NAS filed by introducing LLMs. Next, we provide the detailed comparison and analysis as follows:\nIn short, the cited works [1-5] primarily explore an approach where the LLM generates a candidate architecture, which is then evaluated through a training-based cycle. This is a valuable research direction, though its computational requirements differ from the zero-cost paradigm we focus on. \nWhile this is a valuable research direction, those works suffer from huge computational costs due to training-based evaluation. For instance, work [2] aims to search for GNN, for a generated network by LLM, the network must be trained on specific datasets until convergence. By contrast, our APD differs extremely from works [1-5]  a completely training-free evaluation. In addition, the tasks and benchmarks of works [1, 2, 3, 5] are totally different with our APD. This is challenging for making a fair comparison between works [1, 2, 3, 5] and our APD. For instance, work [1] focuses on SW/HW co-design, work [3] aims to design UNet for diffusion.\n\nIn addition, paper [4] appears to be the most suitable for a direct comparison, as it aligns closely with our task on a shared benchmark and helpfully provides public code. We provide a comparison on NASBench-201 below. As shown, we can find that the Top1 accuracy of our APD excels GENUS. \n|Method|CIFAR-10|CIFAR-100| ImageNet16-120|\n| :--------- | :----------: | :----------: | :------------: |\n| GENUS| 93.79±0.09\\%|70.96±0.33\\%|  44.96±1.02\\%|\n| APD (ours)|93.76±0.09\\%| 72.22±0.65\\% |45.03±0.76\\%|\n\n**Q2**: Comparison with work [6].  \n\n**A2**: Many thanks for your comments! We did not initially include [6] (Eproxy) due to its reliance on training-based evaluation, which differs from our APD (without any training). In addition, we provide a comparison on DARTS search space (ImageNet1k) with Eproxy below. As shown, we can conclude that our APD is superior to Eproxy [6].\n|Method| Top-1 Acc.(\\%) | Top-5 Acc.(\\%) |\n| :----------- | :-----------: | :-----------:|\n| Eproxy|75.6|91.9||\n| APD (ours)|76.9|94.0|\n\n**Q3**: Concern about the cost of LLM API calls.\n\n**A3**: Many thanks for your comments!  We agree that transparency on this front is crucial and are happy to provide the details. For a typical discovery process where $T_{max}$=100, the end-to-end time cost is approximately 1.18 hours on a single RTX 4090 GPU. We find that the vast majority of this duration roughly 80\\% is spent on the fitness evaluation. The actual time spent on LLM API calls, including network latency, constitutes only a small fraction of the total time. Moreover, throughout our entire experimental process, we made a total of  4,151 API calls, consuming 18,442,421 tokens in total. This averages to about 4,443 tokens per call. In a single search, we typically make 100 API calls, consuming approximately 44,430 tokens.\n\n**Q4**: Concern about search costs of APD.\n\n**A4**: Many thanks for your comments! APD is not limited to a specific search space and can generalize across multiple ones ( i.e., NAS-Bench-201\\&101, DARTS). When the discrepancy between spaces is small, the proxy found on the source (e.g., 201) transfers well to the target (e.g., 101, DARTS), as shown in Tables 3–4. For significantly different spaces (e.g., 201 vs. TransNAS, TransNAS vs. AutoFormer), the proxy may become suboptimal due to search space bias rather than limitations of APD. In such cases, we retrain the proxy for the new space, which leads to strong performance, validating our approach. Moreover, APD is highly efficient, requiring only about 1.18 hours on a single RTX 4090 GPU with 4 GB of VRAM.\n\n**Q5**: Risk of Data Containment.\n\n**A5**: Many thanks for your comments! We thank the reviewer for raising this very important and valid point about the risk of data containment. We agree that ensuring our method generalizes to truly unseen tasks, rather than simply recalling knowledge from the LLM's pre-training data, is crucial for validating our contribution. We believe several aspects of our findings help to address this concern. The proxies discovered by our APD framework are structurally novel and do not merely replicate existing methods like Synflow or SNIP, suggesting a genuine discovery process is taking place. More importantly, our method demonstrates strong performance on less mainstream and Out-of-Distribution benchmarks, such as the heterogeneous tasks in TransNAS-Bench-101-Micro and the distribution-shift settings in OoD-VIT-NAS. Success on these benchmarks, which are less likely to have been part of the LLM's training data, points toward true generalization.\n\n**Q6**: How do you choose $T_{max}$? Can you determine when the search has sufficiently converged? \n\n**A6**: Many thanks for your comments!  In our initial experiments, we set a large $T_{max} = 200$ and monitored the average proxy fitness. We observed that performance plateaued during the last 100 generations, indicating convergence. Therefore, we set $T_{max} = 100$ in our main experiments.\n\n**Q7**: How do you compare with other LLM-based NAS work?\n\n**A7**: Many thanks for your comments! See A1 and A2.\n\n**Q8**: Robustness of Discovered Proxies to Prompt Variations and LLM Versions.\n\n**A8**: Many thanks for your comments!  We agree that robustness to the choice of LLM and prompt variations is crucial for the reliability of any LLM-driven framework. The stability of our APD framework across different LLMs is demonstrated by the results in Figure 5 and Table 6. These results demonstrate the performance of our APD framework when integrated with seven different mainstream LLMs, including models from various providers like GPT-4o, Claude 3.7, and Deepseek V3. As shown, our method achieves consistently strong and stable performance across all these different models, indicating that the framework is not highly sensitive to the specific version of the LLM used. This demonstrates the robustness of our overall approach. \nRegarding the sensitivity to slight prompt rewording, our APD framework is designed to be robust to such minor variations for two main reasons. First, our prompts are structured templates that carry essential, non-negotiable information, such as the task definition, search space constraints, and parent proxies for evolution. While the natural language phrasing can be varied, the core informational content remains consistent, guiding the LLM's generation in a stable manner. More importantly, APD is not a one-shot generation system but a closed-loop evolutionary framework. Even if a reworded prompt produces a slightly different or suboptimal initial proxy, its quality is immediately assessed by the fitness evaluator. Over the course of many generations, the evolutionary process of selection, crossover, and mutation ensures that the system converges toward high-performing proxies. This iterative optimization is inherently self-correcting and robust to small perturbations in any single generation step, making the final discovered proxy the product of a stable search rather than a sensitive single prompt.\n\n**Q9**: APD Performance Under Limited LLM Budget.\n\n**A9**: Many thanks for your comments! Yes, APD can certainly be run under a limited budget of LLM API calls.\nThe total cost of the discovery process is primarily determined by some hyperparameters such as the population size and the $T_{max}$. To operate under a stricter budget, one can simply reduce either of these parameters to decrease tokens cost and the total number of required API calls. We have already explored this trade-off in the ablation studies presented in Appendix D.1 of our paper. As those results demonstrate, reducing the search budget leads to a performance trade-off, resulting in a modest and acceptable decrease in the final quality of the discovered proxy. We conducted an experiment to explore the relationship between population size and the corresponding API costs.\n| Population Size| Tokens per-Iteration|\n| :--: | :--: |\n|1|673|\n|2|1839|\n|3|2543|\n|5|4430|\n\n**Q10**: APD Performance Under lightweight LLMs.\n\n**A10**: Many thanks for your comments! That is an excellent point, as assessing the performance of APD with accessible, open-source models is indeed crucial for its broader adoption. Our framework is compatible with a wide range of LLMs, including more lightweight ones. We have found that while these models can successfully participate in the discovery process, the complex, multi-step reasoning required to generate highly novel and effective proxies tends to benefit from the capabilities of current state-of-the-art models. Consequently, the performance when using more lightweight models, while still functional, is generally not as strong as what we achieve with leading models like GPT-4o or Claude 3.7. This appears to be more a reflection of the current state of LLM reasoning capabilities on this complex task rather than a limitation of the APD framework itself. The performance when using a representative open-source model is as follows:\n\n|   Models   | NB201 (CIFAR-10) | NB201-201 (CIFAR-100) | NB-101 (CIFAR-10) | DARTS (CIFAR-10) |\n| :--------: | :--: | :---: | :--: | :--: |\n| Llama 3 8B|90.98 |63.86|88.81|96.95|\n| Qwen 3 8B|86.50|64.96|86.57|95.89|\n|  Gemma 7B  |88.61|65.21|86.38|97.27|"}, {"review_id": "CxOXQSRmYB", "reviewer": "Reviewer_eQXV", "summary": "This paper introduces the APD (Automatic Proxy Discovery) framework, which leverages large language models (LLMs) and actor-critic reinforcement learning to automatically generate zero-cost proxies (ZCPs) for training-free neural architecture search (NAS) on image datasets. \n\nThe method begins by prompting an LLM to produce proxy candidates (code + rationale) using three strategies, i.e., initialization, mutation, and crossover. An RL controller (based on the actor-critic based RL) iteratively improves prompt selection based on the reward that combines the correlation between proxy predictions and true performance with proxy evaluation cost. \n\nThe framework is validated across multiple NAS benchmarks (e.g., NAS-Bench-201, DARTS, TransNAS-Bench) and demonstrates strong results in terms of ranking correlation and searched architecture accuracy. APD outperforms prior handcrafted and learned ZCPs, and its proxy search converges in under an hour. The authors claim APD represents a paradigm shift in training-free NAS by replacing manual proxy engineering with an automated, LLM-driven discovery process.", "questions": "1.  Why an actor-critic RL framework is needed (but not a simpler strategy)?\n\n2. Is it helpful to mix the LLMs,  instead of using a single LLM?\n\n3. Are the proxies discovered by the LLM theoretically meaningful in terms of expressivity and transferability, as studied in works like \"Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective\"?\n\n4. What are the main factors influencing APD’s performance?", "limitations": "Yes, in Appendix E.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:** \n\n1. This paper proposes Automatic Proxy Discovery (APD), which uses LLMs to generate zero-cost proxies, which is a novel approach. \n2. The paper has relatively comprehensive evaluations and demonstrates strong empirical results on 5 benchmarks across 4 tasks: APD’s best discovered proxy achieves high ranking correlation with true performance.\n3. The paper is generally well-written, with clear figures.\n\n**Weaknesses:** \n\n1. In Section 5.1, the paper states that \"the performance improvement can be attributed to our proposed actor–critic reinforcement learning framework, which provides key reasoning and feedback for proxy generation.\"\n   However, actor–critic methods typically require carefully designed state representations, hyperparameter tuning, stable reward signals, and sufficient training data to converge reliably. \n   Is APD’s performance highly sensitive to the stability and training quality of the RL controller?\n   Moreover, the paper does not explore whether simpler strategies—such as random sampling, regularized evolution (REA), or aging evolution (AREA), as used in “Neural Architecture Search without Training” (Mellor et al., 2021)—could achieve comparable results with less complexity and better robustness. \n2. In Section 3.2, the paper states that  \"computed on a randomly sampled subset of B to provide an unbiased estimate of ranking fidelity.\" However, in practical NAS applications, the goal is to identify top-performing architectures,  not to preserve the global ranking across the entire search space. Some zero-cost proxies may exhibit low overall correlation yet perform well at ranking the top-K architectures. \n   Therefore, relying solely on random subsets for evaluation may inadvertently penalize such proxies and limit APD’s ability to discover ZCPs that are highly effective in identifying top candidates. \n   It would be valuable to analyze whether using top-K-aware evaluation criteria could lead to better proxy discovery.\n   Also, how is the value of B defined?\n3. In Algorithm 1, APD uses a fixed step budget T_{max}, but in practice, users typically set time- or cost-based budgets. \n   Since each step may vary in cost (e.g., LLM prompt length, ZCP evaluation), this step-based formulation may not align with real-world constraints. \n   Could APD adapt to time-aware or resource-bounded scenarios?\n4. The paper utilizes LLMs to design ZCPs and further incorporates an actor-critic reinforcement learning loop to guide proxy evolution. While this combination is conceptually interesting, the technical implementation of both components is relatively straightforward and lacks deeper methodological innovation. \n   Since the LLM is invoked directly with structured prompts, the primary technical contribution is the design of the reasoning engine that guides prompt generation and proxy evolution.\n   However, the reasoning engine directly uses standard actor-critic reinforcement learning with no much domain-specific adaptation, limiting the paper’s technical depth and contribution.", "was_revised": true, "final_justification": "I am amazed by the authors' determination and speed in generating new results  and lengthy rebuttal.  The paper, as it is, is still weak in contributions. Some justifications/arguments are not convincing; for example, on interpretability of the proposal (quoted below).  The technical contribution is rather engineering and problematic (please refer to the weaknesses).   I remain negative on the paper.\n\n\n\"We fully agree that interpretability is a crucial aspect of machine learning. However, we must also recognize a common and well-acknowledged phenomenon in the field: many highly successful algorithms still lack strong interpretability, yet this does not diminish their value or contribution.\n\nFor example, Large Language Models (LLMs) have revolutionized our understanding of machine intelligence through their remarkable capabilities in knowledge generation, and have achieved state-of-the-art performance on numerous benchmark tasks. But let us reflect: do models like GPT-4o or DeepSeek offer clear and reliable interpretability? Unfortunately, the answer is still no — even today, LLMs remain largely black boxes.\"", "rebuttal": "Thank you for the helpful and insightful review, which has helped us improve this paper further. Next, we will answer your questions one by one, and we hope this will improve your acceptance of the paper. \n\n**Q1**: Sensitivity of APD to RL controller stability and training quality.\n**A1**: Many thanks for your comments! We chose the actor-critic method primarily for its well-established balance of simplicity and effectiveness, which allowed us to rapidly validate our core framework. By using a standard implementation, we ensured a reliable foundation that does not inherently cause performance degradation. To specifically test the sensitivity, we also explored dynamically adjusting the actor-critic's configuration, and as shown in the ablation studies in Appendix D.1, we found that the impact on the final performance of our APD framework was minimal. Those results and findings have validated that our method is nonsensitive to the RL controller.\n\n**Q2**: Exploring simpler strategies, i.e., random sampling, REA, or AREA.\n**A2**: Many thanks for your comments! The question regarding the necessity of our Actor-Critic controller compared to simpler, yet powerful, evolutionary strategies like REA or AREA is an excellent and important point.\nWe agree that simpler strategies such as Regularized Evolution are strong baselines known for their robustness and effectiveness. The primary reason we chose an A2C framework is its theoretical advantage in handling the specific nature of our search problem. In our APD framework, the agent is not directly modifying an architecture, but rather learning a higher-level meta-policy to select the most effective prompt strategy based on the current state of the proxy population.\nAn A2C controller has the capability to learn complex, state-dependent behaviors. For instance, it can learn to favor exploration-heavy actions like 'Initialization' when it detects that the search has stagnated, or to favor exploitation-heavy 'Mutation' actions when high-performing candidates are found. This adaptive control over the search strategy is something that fixed-rule algorithms like REA, which always discard the oldest member, cannot achieve.\nTo validate this choice, we provide an ablation study comparing our A2C with these simpler strategies on NAS-Bench-201  in CIFAR-10. Those results have proved that A2C is better than simpler strategies. \n|Method|Test Acc|\n|:---:|:--:|\n|A2C (ours)|93.76|\n|REA|85.29|\n|AREA|85.97|\n|Random Sampling|86.88|\n\n**Q3**: Exploring Top-k-aware evaluation and definition of B.\n**A3**: Many thanks for your comments! Our method follows the same design principle as SWAP (ICLR 2024), using ranking correlation over a randomly sampled subset as the fitness criterion. This strategy promotes generalizable proxy discovery, as it encourages learning global performance trends rather than task-specific heuristics. In contrast, top-k accuracy can yield strong performance on a specific task but often lacks generalization, as it tends to overfit to a small set of high-performing architectures. The following experiments demonstrate this difference in transferability.\n|Method|NB-201(C10)|NB-201(ImageNet)|NB-101(C10)|DARTS(C10)|\n|:--:|:--:|:--:|:--:|:--:|\n|Top-k|93.83±0.23|43.87±0.93|91.39±0.28|97.37±0.05|\n|Correlation|93.76±0.09|45.03±0.76|93.49±0.34|97.63±0.13|\n\nFollowing prior work [1, 2], we define the set B as a randomly sampled subset of 200 architectures from the full benchmark. This subset is used to evaluate proxies based on their relative ranking rather than absolute correlation. To validate this design, we conducted an ablation study comparing proxy rankings across different subset sizes (50, 100, 200, 500). Results show that rankings derived from subsets closely match those from the full benchmark, confirming that a small, random subset is sufficient for reliable evaluation.\n|Subset Size|Kendall's Tau|\n| :---: | :---: |\n|50|0.811|\n|100|0.913|\n|200|0.991|\n|500|1.00|\n\n[1]\"Zero-cost proxies for lightweight NAS, ICLR 2021.\n[2] \"How powerful are performance predictors in neural architecture search?, NeurIPS 2021.\n\n**Q4**: Could APD adapt to time-aware or resource-bounded scenarios?\n**A4**: Many thanks for your comments! Thank you for this very thoughtful question regarding the practical application of APD under real-world budgets. That is an excellent point, as aligning the search process with time- or cost-based constraints is indeed an important consideration. Our APD framework is fundamentally designed from the ground up for resource-constrained scenarios, and its current efficiency is already exceptionally high. For instance, the entire discovery process completes in approximately 1.2 hours ($T_{max}$=100) on a single RTX 4090, requires only no more than 4 GB of VRAM, and involves a finite, one-time number of LLM API calls. This low cost stands in sharp contrast to the hundreds of GPU-hours typical of traditional NAS. We found our current fixed step budget to be effective for this. However, we believe the suggestion to incorporate a dynamic step mechanism is very insightful and could reduce costs further. We will explore dynamic step sizes in our future work to continue enhancing our framework's efficiency, and we appreciate this valuable insight.\n\n**Q5**: Concern about domain adaptation of standard actor-critic.\n**A5**: Many thanks for your comments! We appreciate the opportunity to elaborate on the non-trivial design choices behind our reasoning engine. Our decision to incorporate reinforcement learning is not arbitrary, but rather a direct result of our initial analysis, which showed that a naive, open-loop prompting approach fails due to a lack of a reasoning and feedback mechanism, as demonstrated by the \"naive\" example in our paper. To the best of our knowledge, introducing an RL agent to learn an adaptive policy for prompt evolution is a novel approach in this domain. Designing an effective bridge between the generative LLM and the NAS task is a significant challenge, as most general-purpose reasoning models are computationally prohibitive and not tailored for this feedback loop. We found RL to be a natural fit, but building an effective strategy is far from straightforward. The core difficulty lies in defining a stable learning process where the actions are abstract heuristic strategies and the reward is the resulting proxy's ranking correlation. We chose the A2C method specifically because it effectively handles this unique problem structure. The Actor learns a nuanced policy over these abstract actions, while the Critic provides a learned value baseline, which is crucial for stabilizing the search process given the unpredictable stochastic variations. Therefore, we hope to convey that the design and integration of the RL controller is a key technical contribution, requiring careful domain-specific adaptation rather than a straightforward application of a standard algorithm.\n\n**Q6**: Why an actor-critic RL framework?\n**A6**: Many thanks for your comments! Our design is motivated by the need to bridge the significant reasoning gap between the generative, black-box LLMs and the specific, feedback-driven requirements of the NAS task. A simple, fixed strategy like random sampling or basic evolution is unable to learn from and adapt to the complex interactions within this process. Reinforcement learning, by contrast, excels in such scenarios. The RL agent effectively simulates the search process, learning a sophisticated policy to intelligently guide the LLM based on task-specific feedback. It learns how to guide and evolve proxies to achieve better results, a reasoning and learning capability that traditional, fixed-rule methods simply do not possess.\n\n**Q7**: Mixing the LLMs.\n**A7**: Many thanks for your comments! We conducted a supplementary experiment where we used a mixture of GPT-4o and Claude 3.7 on NB-201(CIFAR-10). We use the same setting as the LLM ablation study (Table 6). As shown, we find that mixing the LLMs cannot excel the optimal single LLM (GPT-4o). The potential reason can be attributed to the similarity of LLMs.\n|LLM|Test Acc|\n|:-| :-: |\n|GPT-4o|81.10|\n|Claude 3.7|81.14|\n|GPT-4o+Claude 3.7|81.14|\n\n**Q8**: Discussion about the theoretical motivation of proxies discovered by the LLM.\n**A8**: Many thanks for your comments! Our method is fundamentally different from the theory-driven approach in TE-NAS, which relies on manual design and expert knowledge based on specific theoretical properties like NTK. In contrast, our approach uses LLMs to automatically generate heuristics, representing a new paradigm in Zero-Cost Proxy (ZCP) design. This automation not only reduces reliance on human-crafted rules but also achieves strong empirical performance. While we believe that the proxies discovered by APD may have underlying theoretical value, the black-box nature of LLMs currently makes it difficult to derive explicit theoretical interpretations. Exploring the theoretical grounding of LLM-designed proxies is an exciting direction we aim to pursue in future work.\n\n**Q9**: What are the main factors influencing APD’s performance?\n**A9**: Many thanks for your comments! As experimental results show in our paper, we find that the actor-critic RL Scheduler and Evolution are the main factors influencing APD’s performance. Specifically, In Table 7, we provide the ablation study of various components. As shown, the naive version without Evolution and Actor-Critic modules performs poorly (82.04\\% in CIFAR-10 and 47.62\\% in CIFAR-100). Introducing Evolution alone significantly boosts performance, yielding +6.49\\% and +13.54\\% improvements, respectively. In addition, we observe that there is a joint contribution between Actor-Critic and Evolution. Specifically, Actor-Critic and Evolution obtain 93.76\\% in CIFAR-10 and 72.22\\% in CIFAR-100. Those results validate that the actor-critic RL Scheduler and Evolution are the main factors influencing the effectiveness of our APD."}, {"review_id": "Uw0MsmbyIZ", "reviewer": "Reviewer_1iSx", "summary": "The paper discusses the limitations of zero-cost proxies and proposes to address them by introducing an LLM-driven method to automatically discover such proxies. The method, named Automatic Proxy Discovery (APD), can be viewed as an evolutionary loop, consisting of: 1) an LLM-driven Proxy Candidate Generator, 2) a Fitness Evaluator, which evaluates proxies on a subset of architectures using their ground-truth performances, and 3) an RL Evolution Scheduler, an actor-critic algorithm that guides the evolutionary step. The authors evaluated it on different search spaces and datasets.", "questions": "Is the provided implementation sufficient to reproduce the results?\n\nCould you address all my concerns regarding overfitting and reliance on the gt accuracies? How many unique architectures are used per search space? Are the 200 sampled once and reused, or redrawn each iteration? Do you use train or test accuracy?\n\nDo you apply a proxy learned on one benchmark to another? This could provide a meaningful measure of overfitting. It seems that such cross-search-space generalization is not tested and a new proxy is found for each benchmark, but this is unclear. Also, which proxy is used for the DARTS experiments, and how is the final architecture selected for a given proxy?\n\nTo be more convinced, I would also like to know the real impact of the actor-critic module. If I understand correctly, it only samples one action among three - one of which (initialization) is likely to be used only at the beginning. While there is an ablation study evaluating this module, I do not find it satisfying.\nIn Table 7, what exactly does naive+evolution refer to? Does it mean sampling each action with 1/3 probability? If I understand correctly, if the initialization action is chosen in the i-th iteration, all proxies gathered so far in the first i-1 iterations discarded, which could significantly affect the outcome of the ablation study.\nIt seems like one could easily construct a handcrafted policy for sampling actions - something smarter than uniform 1/3 probability - that could match the performance of the actor-critic scheduler. I would appreciate it if the authors could go into more detail, clear my doubts, and provide stronger evidence that this module actually contributes meaningful value.\n\nI am open to discussion and happy to be corrected or change my score if I misunderstood any aspects of the work.", "limitations": "Largely addressed; I mentioned some possibly unaddressed limitations of the work above.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and structured in a way that makes it easy to follow the main flow of the paper\n- I appreciate that the authors clearly highlight the limitations of a naive approach that relies only on an LLM. They also elaborate on these shortcomings in the appendix and discuss prompt engineering, which I consider valuable. Rather than incorporating LLMs into NAS in such a naive way, the authors wrap them in an evolutionary loop, which is an interesting design\n- I find the high-level idea of restricting the LLM's role to generating or mutating proxies to be compelling. While a simple LLM-only approach falls short, it intuitively makes sense that an LLM should perform reasonably well at this specific task of modifying proxies in each iteration. Overall, while it is difficult to effectively incorporate LLMs into NAS, the authors presented a creative and promising approach.\n\nWeaknesses:\n- Possible overfitting. Overfitting is a major issue in NAS, but this is not talked about at all in the paper. In each iteration, APD generates several new proxies, ranks them by the correlation with ground-truth (gt) performance, and drops the worst ones, which potentially makes the method just an overly-complex way to overfit the benchmark. There are in total 10 * 100 such iterations, which - depending on whether architectures are reused across iterations - can lead to either overfitting to a small set of architectures (especially that the authors use a single fixed seed) or excessive reliance on the gt performance (see the point below).\n- Vague reliance on gt accuracies. In the appendix, the authors state that “computing fitness requires baseline accuracies for roughly 2-3% of each benchmark”. However, it is unclear where this number comes from. There are 10 * 100 iterations, and in each iteration candidate proxies are evaluated on 200 architectures, potentially yielding the need for gt performance of 200k architectures if the batch of architectures gets resampled each time. Unfortunately, it is impossible to verify since these details are not included in the paper, and the code that corresponds to the evaluation of proxies is simply missing. A fixed set of architectures can be treated as a best-case scenario, but I am sceptical as this is not backed by any evidence.  Also, it is unclear if these are train or test accuracies, or how this number scales with the size of the search space. I do not feel like these issues were given enough attention.\n- Poor reproducibility. As described above, important details are not included in the paper. The implementation seems to be incomplete, as some modules (such evaluation of candidate proxies) are missing. Also, the requirements can be installed only on Windows OS.\n- While the paper is overall well-written, the phrasing is sometimes catchy and inaccurate. Calling the method a zero-cost proxy (ZCP) is misleading, as it still relies on many gt accuracies. Similarly, the paper mentions chain-of-thought a few times, but the only overlap is just a standard actor-critic loop. In particular, there is no reward-trained LLM reasoning.", "was_revised": true, "final_justification": "I provided such justification for raising the score in my discussion with the authors.", "rebuttal": "Thank you for the helpful and insightful review, which has helped us improve this paper further. Next, we will answer your questions one by one, and we hope this will improve your acceptance of the paper.\n\n**Q1**: Possible overfitting \n**A1**: Many thanks for your comments! Our method does not suffer from overfitting, and we have provided clear empirical and conceptual evidence to support this. First, while APD searches over a small subset (200) from the source search space (NAS-Bench-201), the discovered proxy generalizes well to unseen search spaces (NAS-Bench-101, DARTS) and datasets (CIFAR-100, ImageNet), as demonstrated in Tables 3–4 and Appendix D. If overfitting were occurring, APD cannot obtain optimal performance cross unseen search spaces. Second, regarding the broader claim that “overfitting is a common issue in NAS,” we want to clarify: overfitting in traditional NAS typically arises during the full training of architectures after the search stage. In contrast, APD is a training-free method. It does not involve gradient-based updates or full model training, which fundamentally eliminates the main source of overfitting in standard NAS pipelines. Hence, both in practice and by design, our method is robust to overfitting.\n\n**Q2**: Vague reliance on gt accuracies\n**A2**: Many thanks for your comments! To clarify, following prior works published at ICLR [1] and NeurIPS [2], we define the set B as a single, fixed subset of 200 architectures randomly sampled once from the full benchmark using a fixed random seed. This subset remains unchanged throughout the entire search process, and only the test accuracies of these architectures are used for all fitness evaluations. No new architectures are re-sampled or introduced during the search.\n\nOur evaluation focuses on the relative ranking of proxies within this subset rather than their absolute correlation scores. To validate the reliability of this design, we conducted an ablation study using 30 discovered proxies. We calculated each proxy’s correlation scores on subsets of varying sizes (50, 100, 200, and 500 architectures) and compared the resulting rankings to those derived from the full benchmark via Kendall’s Tau. The results show consistently high rank correlation across all subset sizes, confirming that a small, randomly sampled subset is sufficient for reliable evaluation while effectively balancing accuracy and computational cost. These experimental details have been added to the final version to improve reproducibility and transparency.\n|Subset Size|Kendall's Tau|\n| :---: | :---: |\n|50|0.811|\n|100|0.913|\n|200|0.991|\n|500|1.00|\n\n[1] Zero-cost proxies for lightweight NAS, ICLR, 2021.\n[2] How powerful are performance predictors in neural architecture search?， NeurIPS 2021.\n\n**Q3**: Poor reproducibility.\n**A3**: Many thanks for your comments! As stated in \"A2\", we first validate and improve the experimental details. In addition, we provide the main code for the evaluation as follows:\n\n```python\ndef evaluate(args, f_score):\n    # load archs and dataset\n    ...\n    loader = iter(train_loader)\n    inputs, targets = next(loader)\n    eval_len = len(arch_info)\n    results = []\n    index_arr = np.arange(len(arch_info))\n    np.random.seed(args.seed)\n    np.random.shuffle(index_arr)\n    for index, i in enumerate(arch_info.values[index_arr[:eval_len]]):\n        start_t = time.perf_counter()\n        network = model_201.Network(16, 4, i[0], 10)\n        network = network.to(device)\n        swap_score = []\n        for _ in range(args.repeats):\n            network = network.apply(network_weight_gaussian_init)\n            score = f_score(network, inputs.to(device), targets.to(device))\n            swap_score.append(score)\n        total_eval_time += time.perf_counter() - start_t\n        results.append([i[0], np.mean(swap_score), i[1]])\n    results = pd.DataFrame(results, columns=['genotype', 'swap_score', 'acc'])\n    avg_time = total_eval_time / len(results)\n    return stats.spearmanr(results.swap_score, results.acc)[0] + avg_time * args.beta\n```\n\nFor concern of Windows, our method is compatible with both Windows and Linux operating systems. The win-inet-pton package is included only for Windows systems with GPU support to handle certain network utilities. When running on Linux, this package is not required and can be safely removed without affecting the core functionality or performance of our framework.\n\n**Q4**: Inaccurate statements.\n**A4**: Many thanks for your comments! First, we agree with the reviewer that zero-cost proxy (ZCP) methods are not entirely \"zero cost\" as they rely on a limited set of ground-truth (gt) accuracies. Our definition of ZCP follows established conventions from prior works such as ZiCo and AZ-NAS, where ZCPs are evaluated without additional training. Nonetheless, we acknowledge the terminology could be misleading and will revise it in the final version for clarity.\n\nSecond, regarding the use of Chain-of-Thought (CoT), we appreciate the opportunity to clarify. As discussed in [1], reward-trained LLM reasoning typically uses training accuracy as a reward to guide LLM generation. However, this approach is often ineffective due to the semantic gap between language models and NAS-specific tasks—accuracy is a sparse and indirect signal, which we also observed in our naive baseline.\n\nTo overcome this limitation, we introduce an actor-critic RL controller that learns to schedule generation strategies (e.g., initialization, mutation, crossover) in a way that promotes iterative, feedback-driven improvements. While we do not perform token-level CoT prompting or reasoning within the LLM, our controller-driven proxy discovery mechanism mimics multi-step reasoning across iterations. To reflect this more accurately, we now define APD as: “A controller-guided, Chain-of-Thought-inspired scheduling framework for evolving zero-cost proxies.” \n\nWe will update the manuscript to use this revised phrasing, avoiding confusion with conventional CoT paradigms while preserving the intuition of multi-step guided reasoning.\n\n[1] Can gpt-4 perform neural architecture search?, arXiv 2023. \n\n**Q5**: Concern about overfitting and GT \n**A5**: Many thanks for your comments! For overfitting and GT, see Q1 and Q2. In addition, we use a fixed set of 200 architectures—sampled once per benchmark—and reuse it throughout the search. Evaluation is consistently based on their test accuracy.\n\n**Q6**: Impact of the Actor-Critic Scheduler\n**A6**: Many thanks for your comments! We feel sorry for not explaining the ablation study in Table 7 with sufficient clarity. To clarify, the \"naive + evolution\" setting is a specific baseline designed to test a simple, greedy evolutionary strategy. It refers to an iterative process where we continuously use the simple prompt from Appendix A to have the LLM generate new ZCPs. In each generation, only the single best-performing ZCP from the current population is retained. This is distinct from sampling the three actions with uniform probability. We would also like to clarify that in our full APD framework, the 'initialization' action does not discard the existing population but rather introduces new, diverse proxies into it to escape local optima without resetting progress.\nThe core contribution of the Actor-Critic module is that it provides a principled, learning-based solution to bridge the significant reasoning gap between the generative, black-box LLM and the NAS tasks, where simpler strategies struggle with reliability. The agent must learn a policy over abstract actions based on a noisy and unstable feedback signal. This learned, adaptive control over the search is the meaningful value that a static or simple handcrafted policy cannot provide, leading to a more robust and effective discovery process.\n\nRegarding whether handcrafted policies can replace the RL controller, once RL is used, the naive 1/3 uniform sampling strategy is not applied. To verify the RL policy’s advantage, we tested two additional handcrafted strategies: the fixed strategy refers to using a constant action sampling ratio of 1:2:1. The dynamic strategy is divided into two stages: in the first stage, the probabilities of Initialization and Crossover are higher, while in the later stage, the probability of Mutation becomes dominant. Experimental results show that our RL method consistently outperforms these handcrafted strategies, demonstrating its superior adaptive control of the search.\n\n|Method|CIFAR-10(NB-201)|\n|:---:|:--:|\n|Naive|82.04|\n|Naive+EC|88.53|\n|Naive+EC+fixed|88.97|\n|Naive+EC+dynamic|89.13|\n|Naive+EC+Actor-Critic|93.76|"}]

Author rebuttal:

Meta-review: This paper received mixed ratings from the reviewers: one acceptance, two borderline acceptances, and one borderline rejection. Overall, the feedback tends to be positive, largely due to the thorough and exhaustive rebuttal efforts that addressed most of the initial concerns. The remaining issues primarily come from reviewer eQXV, who requested clarifications on implementation details and further discussions. Although the reviewer appreciated the detailed and iterative rebuttals, they still consider the contribution to be somewhat weak.

The AC has read the paper and finds the idea of applying reinforcement learning on LLM for NAS to be an interesting and promising direction for this line of research. The AC agrees that the writing can be improved in the final version by incorporating all reviewer feedback to produce a stronger, more polished paper. Taking all factors into account, the AC recommends accepting this paper, mainly based on the novelty and potential impact of the idea.
Decision: Accept (poster)