RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy
Authors: Zipeng Ji
Guanghui Zhu
Chunfeng Yuan
Yihua Huang
Venue: ICML 2025 poster (2025)
Keywords: Automated Machine Learning
Neural Architecture Search
Large Language Models
Primary area: deep_learning->algorithms
TL;DR: A novel framework that combines the text- and code-level comprehension capabilities of LLMs with a Reflective Zero-Cost evaluation strategy for NAS

Abstract: LLM-to-NAS is a promising field at the intersection of Large Language Models (LLMs) and Neural Architecture Search (NAS), as recent research has explored the potential of architecture generation leveraging LLMs on multiple search spaces. However, the existing LLM-to-NAS methods face the challenges of limited search spaces, time-cost search efficiency, and uncompetitive performance across standard NAS benchmarks and multiple downstream tasks. In this work, we propose the Reflective Zero-cost NAS (RZ-NAS) method that can search NAS architectures with humanoid reflections and training-free metrics to elicit the power of LLMs. We rethink LLMs’ roles in NAS in current work and design a structured, prompt-based to comprehensively understand the search tasks and architectures from both text and code levels. By integrating LLM reflection modules, we use LLM-generated feedback to provide linguistic guidance within architecture optimization. RZ-NAS enables effective search within both micro and macro search spaces without extensive time cost, achieving SOTA performance across multiple downstream tasks.

RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy
Zipeng Ji
Guanghui Zhu
Chunfeng Yuan
Yihua Huang'
Abstract
vital and useful tool. Recently, the combination of Large Language Models (LLMs) with NAS represents a cutting- edge development in automated machine learning, seeking to alleviate the difficulties of manual designs and explore novel architectures on diverse NAS tasks. However, most of the existing work remains in the exploratory phase, where LLMs generate neural architectures directly through tex- tual prompts (Zhao et al., 2023; Yu et al., 2023; Wei et al., 2023). This approach suffers from two key drawbacks: (1) Reproducibility Stochastic LLM responses hinder consis- tent results. (2) Interpretability: Text-based prompts lack clarity on the design rationale, making optimization and trust difficult. Moreover, LLM-to-NA. methods that focus on generating architecture code (Lehman et al., 2024) can only support tiny search spaces and simplified networks, thus exhibiting poorer performance compared to established NAS algorithms on standard NAS benchmarks (Chen et al., 2023). Furthermore, current LLM-to-NAS algorithms rely on iterative or evolutionary methods, facing high computa- tional costs, since each architecture requires full training for evaluation. To address this, aim to design a novel LLM- to-NAS algorithm that (1) enhances LLMs' understanding of NAS architectures from both text-level and code-level, (2) addresses the time-cost issue in existing L.MM-to-NAS methods, and (3) achieves better performance and scalability for broader search spaces and standard benchmarks.
LLM-to-NAS is promising field at the inter- section of Large Language Models (LLMs) and Neural Architecture Search (NAS), as recent re- search has explored the potential of architecture generation leveraging LLMs on multiple search spaces. However, the existing LLM-to-NAS meth- ods face the challenges of limited search spaces, time-cost search efficiency, and uncompetitive performance across standard NAS benchmarks and multiple downstream tasks. In this work, we propose Reflective Zero-cost NAS (RZ-NAS) that can search NAS architectures with humanoid re- flections and training-fio free metries to elicit the power of LLMs. We rethink LLMs' roles in NAS in current work and design structured, prompt- based to comprehensively understand the search task and architectures from both text and code levels. By integrating L.M reflection modules, we use LLM--genrated feedback provide lin- guistic guidance within architecture optimization. RZ-NAS enables effective search within both mi- сro and macro search spaces without extensive time cost, achieving SOTA performance across multiple downstream tasks.
In this paper, we introduce a novel framework that com- bines the text- and code-level comprehension capabilities of LLMs with Reflective Zero-Cost evaluation strategy for NAS (RZ-NAS)) To integrate the text- and code-level understanding abilities of LLMs, we develop structured prompts to precisely define NAS tasks. These prompts in- clude: a high-level role, detailed instructions, an in-context example, and the key reflective module. Moreover, we uti- lize Zero-Cost proxies instead of training architectures to reduce computational resources and time cost while main- taining competitive performance (Abdelfattah al., 2021; Sun et al.. 2023). The reflective module guides the LLM to reflect on mutation performance and generates targeted suggestions for further iteration improvements.
1. Introduction
Neural Architecture Search (NAS) has rapidly advanced, transforming neural network design by automating the search for optimal architectures (Zoph & Le, 2017; Liu et al., 2019; Termritthikun et al., 2021; Jiang et al., 2023; Zhu et al., 2022). With increasing computational complex- ity and demand for high-performance neural networks in tasks such as image recognition (Real al., 2019) and ob- ject detection (Sun et al., 2022), NAS has proven to be
'State Key Laboratory for Novel Software Technology. Nanjing University, Nanjing, China. Correspondence to: Guanghui Zhu <zgh@njuu eju..dd.en>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
RZ-NAS is available at https: / http://gttttbccooo Pasa bb/RR=-AS.

---

RZ-NAS:
Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy
Search Space
Search Efficiency on CIFARIO (GPU days)
Support Zero-Cost
LLM's Role NAS
Algorithm
Text-Level Understanding
Code-Level
Micro
Macro
NAS
Understanding
GPT-NAS et al., 2023)
X
1.5
X
EvoPrompting (Chen et al., 2023)
/
/
X
GENIUS (Zhao ul., 2023)
X
/
1.0
X
X
LLMatic (Nasir al., 2024)
/
X
X
FL-NAS (Qin et al., 2024)
x
/
X V
X
RZ-NAS (Ours)
/
/
0.03
Table Comparison of key capabilities across existing L.LM--t-NAS work. Texe-Level Understanding indicates providing the textual description of NAS to LLMs, whereas Code-Level Understanding refers to supplying NAS code to LLMSs. / means that the property is present the algorithm, while indicates its absence. denotes that the source code isn't available, the search cost is not mentioned.
Building on this, we adopt the evolutionary NAS strategy, optimizing the random mutation process guided by the text- and code-level understanding of LLM. In Table marize the improvements of our method compared to five existing LLM-to-NAS studies. Through the reflective Zero- Cost evaluation strategy, RZ-NAS can achieve better per- formance than the original proxies. In addition, it can even outperform traditional NAS methods while maintaining a low search cost. The contributions of this work are summa- rized as follows:
sum of the clement-wise product of weights and gradients to reflect the sensitivity of a network. Jacob (Lopes et al., 2021) leverages the Jacobian matrix between the loss and multiple input samples to quantify the capacity. MAE-DET (Sun et al., 2022) leverages the differentiable entropy to evaluate the amount of information the model extracts. The most recent Zero-Cost NAS strategies (Abdelfattah et al., 2021; Gao et al., 2022) use the evolutionary algorithm as the search framework, mimicking the evolutionary process by iteratively selecting, mutation and recombining candidate architectures based proxy scores. Recently, benchmarks such as WAS-Bench-SSite-22 have raised concerns about using them as full replacements for accuracy. These insights motivate the careful selection and combination of proxies in practice.
sum-
(Krishnakumar et al., 2022),
We design a reflective Zero-Cost NAS strategy, coupling architecture generation with humanoid reflections and training-free metrics to elicit the power of -.-.M.t-NNS.
We rethink the role of LLMs in NAS and design struc- tured, prompt-based approach by combining the text- and code-level understanding of LLMs for architecture muta- tion in evolutionary NAS.
2.2. LLM for Evolutionary Algorithm
The LLM-based evolutionary algorithm is still at a very early age. Pioneering work (Lehman et al., 2024) finds that LLMs designed for code generation can significantly en- hance the effectiveness of the mutation process in Genetic Programming (GP). Building this, recent studies explore LLMs' role in selection within evolutionary algorithms, re- ducing reliance on black-box functions. (Hao et al., 2024) encodes architectures as natural language via prompt engi- neering and uses LLMs to predict proxy scores, reducing computational costs. LLaMEA (van Stein Bäck, 2024) leverages LLM to automatically generate meta-heuristic algorithms based on a given of criteria and task defini- tions. Coomera-Pareder et al., 2023) indicates that LLMs can achieve state-of-ther the-thr-rrt performance in the cap set prob- lem in finite dinte--dmessioall spaces, revealing the potential of LLMs to contribute to mathematical optimization problems.
We show that RZ-NAS outperforms the SOTA LLM-to-
NAS and Zero-Cost NAS methods across various datasets and search spaces.
2. Related Work
2.1. Zero-Cost Neural Architecture Search
Zero-Cost NAS means "NAS without training", which has emerged promising approach that addresses the expen- sive computation cost in traditional NAS methods (Xiang et al., 2023; Javaheripi et al., 2022). Using proxy metrics with minimal computational overhead, Zero-Cost NAS al- lows evaluating many architectures without the full training cost. Some existing Zero-Cost proxies are based on the mod- els' gradient, such Gradnorm (Abdelfattah et al., 2021), Synflow (Tanaka al., 2020), and SNIP (Lee et al., 2019), others are built on different perspectives to evaluate models' capacity and expressivity. Gradnorm (Abdelfattah et al., 2021) measures the L2 norm of the gradients of network parameters for a random loss, estimating how well the gradi- ents propagate. Synflow (Tanaka et al., 2020) computes the
2.3. LLM for Neural Architecture Search
Recent advancements in large language models have sparked remarkable results in evolutionary algorithms for neural architecture search, presenting a novel avenue for
2

---

RZ-NAS: Enhancing L.M-guided Neural Architecture Search via Reflective Zero-Cost Strategy
automating the design of neural networks. GPT-NAS (Yu et al., 2023) utilizes GPT model predict components of neural architecture based on an encoding scheme tradition- ally employed for evolving architectures within evolutionary algorithms. Similarly, GENIUS (Zhao et al., 2023) replaces the architecture suggestion step with GPT-4, prompting it iteratively to propose architectures, evaluate their perfor- mance, and provide feedback for suggesting improvements. EvoPrompting (Chen et al., 2023) leverages LLMs to under- stand the code of crossover and mutation operators within evolutionary algorithms, focusing small search spaces and simplified architectural constructions. LLMatic (Nasir et al., 2024) designs candidate operation pools and utilizes LLM to update operations in the architecture.
problem can be expressed as follows.
a" = arg
(I)
3.2. Overall Search Framework
As shown in Figure 1, RZ-NAS is an iterative NAS optimiza- tion framework that combines LLMs, Zero-Cost proxy eval- uation, and the L.M reflection mechanism. It leverages the understanding of LLM on the specified task, search space, model architecture, and metrics to mutate architectures and evaluate them using Zero-Cost proxies, refining the search for optimized designs through reflective feedback.
In Figure 1, we precisely extract the roles of LLM, which includes two functions: mutation generator for selected architecture and a reflection module for yielding better muta- tions. To help the LLM understand the mutation process, we provide the genotype of the selected architecture in string format and three system-related descriptions with corre- sponding textual and code representations. The search space description introduces each operation in the defined space. The network construction description and the Zero-Cost proxy description explain how the network is built and how the proxy is computed, respectively, each using code and a brief textual explanation. The above inputs are integrated into prompt template and fed into the LLM to guide the mutation process, as detailed in Figure 2. Specifically, the architecture textual genotype is incorporated into the user prompt to request new mutations, while the three system- related descriptions are embedded in the system prompt, enabling the LLM to comprehensively understand the task context. During each iteration, the three system-related descriptions remain fixed, providing consistent context for the specific NAS task, while the architecture genotype is updated based on the mutations generated by the LLM.
2.4. Self-Reflections of LLMs
The self-reflection mechanism in LLMs refers to the model's ability to review the outputs or reasoning process to iden- tify and correct errors. While the theoretical support for the effectiveness of reflection mechanism remains unclear, in the context of LLMs, reinforcement learning with hu- man feedback (RLHF) enables output refinement through feedback, simulating the process of reflection (Lee et al., 2024). Reflexion (Shinn et al., 2023) proposes a frame- work to strengthen LLM agents through language feedback. introduces a
loop, where Actor generates a trajectory, Evaluator computes rewards, and
provides natural language feedback to guide subsequent decisions. ReEvo (Ye et al., 2024) com- bines evolutionary computation with human-like reflection to generate heuristic algorithms. Moreover, the reflection mechanism greatly improves model performance on tasks such as multiple-choie reasoning (Renze & Guven, 2024).
3. THE PROPOSED METHODOLOGY
After the LLM completes the mutation process, the muated architecture is first validated and then evaluated using pre- defined Zero-Cost proxies. Then, the LLM reflects on the mutated operation and its performance through the reflection module. The reflection module compares the architecture before and after the mutation and prompts the LLM to gen- erate better solutions based on the current mutation.
In this section, we first discuss the role of LLM in Zero- Cost NAS and formulate systematic framework that can be applied to multiple Zero-Cost NAS proxies. Then, we integrate LLM into Zero-Cost NAS methods for better and more intelligent search.
3.1. Problem Formulation
Population Construction. RZ-NAS maintains a dynamic architecture population throughout the optimization process. The initial population is randomly generated and the Zero- Cost score is calculated for each architecture. RZ-NAS supports both macro and micro search space. For the micro search space, the genotype format follows the approach in DARTS (Liu et al., 2019), where operations are organized within cell as directed acyelic graph (DAG) and stacked to form supernet. In the macro search space, the operations are connected sequentially and the architecture consists of vanilla convolutional layers, where each layer includes a
Let S and D denote the search space S and the dataset, respectively. Our goal is to find the optimal architecture a" from the candidate architecture set A in S via computing Zero-Cost score during LLM-guided architecture mutation. The mutated architecture is evaluated via Zero-Cost proxy object function O. Given task T,, search space S, dataset D, and model M, the search process returns the architecture with the maximum Zero-Cost proxy score. The optimization
3

---

RZ-NAS: Enhancing L.M--uided Neural Architecture Search via Reflective Zero-Cost Strategy
Text-level Code-level Descriptions
LLM- guided Evaluation Strategy
LLM Prompt Template
Architecture Population
H+
Search Space Description
System Prompt
Role
Muated Arch
clach Operation Class deser
cidentify system's 'nree's roles
cdef _J vdel forwardx
Instructions
<Explain detalls
Construction Description
Validate Architecture
In-contect Example
chanwonCremmuatton Descriptiss
<lim_start) uuser
sinput archtecture example
LLM
Compute Zero cost Proxy Score
<linustart] dinnslall-avviille aavistart
Selected Arch
Zero-cost Proxy Description
courpur of mussted architecturez
caspiam the mutation processo <[im_end]]>>
Ceet-osu Resriptians
Proxy
LLM Reflection Module
User Prompt
You are provided with [erchillllscora))
Text based Genotype
Mutate below architecture
suggest des gning mutation with better performance
llarchiteturrei:
"Connit u...2t Attak 3K00102,20 120,1,400.....
Update Population
Figure 1. The pipeline of RZ-NAS. The workflow starts initializing population of architectures. One architecture is randomly selected, and its text-based description along with system information including search space, network construction, and selected Zero-Cost proxy, is the LLM prompt template. The LLM generates a mutated architecture, which is then validated and scored using the proxy. The mutation, along with its score and possible errors, is fod back into the LLM to guide better mutations in subsequent iterations.
convolutional operator followed by ReLU activation (Serra et al., 2018; Hanin & Rolnick, 2019). Once the population is initialized, we transform each architecture into string format as shown in the text-based genotype in Figure
gramming tasks (Liu al., 2023; van Stein & Bäck, 2024), showcasing their ability to embed domain knowledge and intelligently explore the search space. Unlike prior stud- ies (Zhao et al., 2023) that treat LLM as a black box by replacing the entire NAS strategy with simplistic prompts, our approach leverages LLM more strategically within the mutation process.
Architecture Mutation. RZ-NAS understands both text- and code-level descriptions guide the LLM in mutating the architecture genotype during each iteration. Rather than directly mutating architecture code, the LLM performs text- based mutation, which is more feasible since code-level modifications may span multiple files and classes. The LLM's first role to guide the mutation of selected archi- tecture by modifying operators formatted textual geno- type. These mutated operators are defined in the search space and replace the original ones in the architecture geno- type. As result, the output of the mutation process is textual genotype, which is then transformed into executable architecture code for performance evaluation. The code and semantic descriptions used the four input components will be detailed Section 3.3.
Architecture Validation. After LLM generates the muta- tion, RZ-NAS validates its construction for correctness (e.g.. checking if the layer length is within the allowed limits). also adheres to the FLOPs budgets of Zero-Cost NAS to search for optimal networks within various budgets. The invalid architecture will not be added the population.
Compute Zero-Cost Proxies. RZ-NAS provides versa- tile Zero-Cost evaluation strategy that supports different Zero-Cost proxies. As discussed in Section 2.1. Zero-Cor Cost proxies initially originate from gradient- based methods such as Grasp (Wang et al., 2020) and later evolve into more di- verse proxies grounded in different theoretical frameworks, such as
Mutation in NAS typically relies on random perturbations to explore the search space. However, this often requires extensive customization and domain-specific knowledge to ensure meaningful and effective changes (Lehman et al., 2024). LLMs with implicit knowledge of architecture de- sign can guide architecture mutation more intelligently. Re- cent research in evolutionary computation demonstrates that LLMs can outperform traditional mutation in genetic pro-
(Lin et al., 2021). As a general search framework, RZ-NAS can incorporate different Zero-Cost proxies. The Zero-Cost proxy computation module in Fig- ure provides code block and proxy description for model understanding. The user prompt includes a JSON list with the current architecture, proxy type, and pre-mun are-mutation score. The L.LM is guided by structured prompts, I/O formats, and in-context examples. Reflections from prior outputs further

---

RZ-NAS: Enhancing LLMM-uided Neural Architecture Search via Reflective Zero-Cost Strategy
Algorithm 1 LLL--uideed Mutation and Zero-Cost Evalua- tion Strategy
Zero-Cost scores. The architecture with the lowest score is then removed to maintain a fixed population size. After T' iterations, the architecture with the highest score is re- turned. More details on the search workflow can be found in Algorithm 1.
Search space S, inference budget B, maximal depth L, total number of iterations T. evolutionary population size N, initial structure Fo, the selected Zero-Cost proxy ze OUTPUT: The architecture with highest proxy score.
INPUT:
3.3. Prompt Template Design
1: Initialize population P = (Fo)-
To achieve LLM-guider architecture mutation, we design prompts with structured context and a specified output for- mat. Figure 2 shows the system-level and task-specific prompt template inspired by Copilot (Denny et al., 2023) and multiple templates from OpenAI Library 2. The left side outlines the full template structure, while the right displays four input text-level and code-level descriptions. The prompt template includes three components: system prompts defining LLM's role and task descriptions, illustra- tive in-context examples, and specific user instructions.
2: for t 1,...... do
3: Randomly select F& E P.
4: Fe < GENTRATINGMITTATTONNNN Fr,
5: if VALIDATE(FF) True then
6: if P. meets the inference budget B then
7: Compute Zero-Cost score zc(Fi). Append F. to P. if [|| > E then
8:
9:
10:
Remove the architecture with the smallest Zero-Cost score from P.
11:
end if
The instruction emphasizes the expert role of LLM in Zero-Cost NAS and Python program- ming. The role definition should be clear and concise to generalize across domains without relying on specific scenarios (Zhang et al., 2024), guiding the LLM to pro- duce scientifically relevant and meaningful architectures. Next, we provide concise description of the search space, Language models help us understand the structure of each operation by detailing textual properties and construction code. For each operation in the search space, the defi- nition, parameters, and forward() function are explained in detail in Appendix A.3. Furthermore, to enhance ex- pertise optimizing mutation, RZ-NAS introduces the system reflection module, which reflects and gives hints for the improved mutation design in the next step.
System Prompt.
12: end if
13:
end if
14:
Get exception e if F; is invalid.
15:
(EFLECTION(FF,244
16: end for
17: Return F", the architecture with the highest score in P.
refine the process. To avoid hallucinations, Zero-Cost scores are computed by code, not LLM.
LLM Reflection Module. We introduce an innovative LLM reflection module that evaluates each mutated architecture and its computed score, providing suggestions and insights for improved designs in the next iteration. The reflection mechanism includes two modules: internal reflection mod- ule in the system prompt (e.g., Reflect to generate better mutation) and external L.M reflection module (e.g., Gen- crate suggestion for better mutation). The external module takes as input the architecture before and after mutation in the current iteration, the Zero-Cost proxy score, and any encountered exceptions input, and outputs dynamic, structured reflection suggestions to guide future mutations. These two modules form a closed-loop feedback system: the external module provides explicit optimization directions, while the internal module enables implicit semantic reason- ing. Previous research on self-reflection in LLMs (Renze & Guven, 2024; Ye et al., 2024) has demonstrated significant improvements in performance, mitigated hallucination, and smoother fitness landscapes by employing self-reflection problem-solving and heuristic tasks. A detailed example of reflection is shown in Appendix A.2.
During the network construction process, we define the input/output formats and mutation constraints, such as kernel size and input channels, through textual descrip- tions. In addition, in the code block, we illustrate the architecture construction process. In Figure 2, we pro- vide the input code where the architecture is formatted as an operation string and instantiated within the function create arch list from str() to build the architecture within the Network class.
Figure also shows the architecture mutation logic. Mul- tiple Zero-Cost proxy computation codes are integrated to ensure system compatibility with various evaluation methods. Based on the type parameter in the user prompt, the system selects the appropriate Zero-Cost proxy.
In-context example. The in-context example provides task-specific instructions, including mutation example and explanation of the reasoning steps that lead the output. Moreover, we integrate the step-to-step reflection
Population Update. All architectures including both the original and the mutated candidates are first sorted by their
htttpa: pttppa://paaaffmm...m openai..ea.......mm www.aicccomeeeeee docs/ examples ls
5

---

RZ-NAS: Enhancing L.M-guided Neural Architecture Search via Reflective Zero-Cost Strategy
System Prompt
</im_ siin_star/ssystem
- class ConviXeeeLLU" This class defines. black structure with several layers
Define Role and
expeet understanding the sere-cost methods tast price rutation process archi rechurecture with better rerc-rost prosy score.
optinal.
def forward self, >):
outpec =
Understand Search Space and Operations
for black in set block, List:
sutput block(ostput)
Define Space
I should understard all operations in the smarth space, All operations are defired be lon...
return cutput
Space Description
System Reflection
generate
acolutionary mutation. better architactures.
The input architecture is defined steiig variable naned "arch_str"
should reflect hints
class Netxork(mn.Module)" def linit (self):
Network Construction
The insut architccture is defined steing variable naned "leck_str' The format is: copersions parat)) coperation2 parkl,, paraa).
self.arch_list
Network Construction
def farlard(self, x):
create_arch fistn_-ist.roossstrrrrre
output + for bech_ete, the_ the_eterent enunerate(self..nm wwr.heee...rch.sssa..
Architecture
I should understard the key cote of architectures and mutate the operation fron the search space 115t
Mutation
Compute Zero-Cost Proxy The type of perb-rost praxy is crerd-cost proxpp. The corputing code is below.
Compute Zero-Cost Proxy
gradients ...
prasy description
def corpute, www.ggraddr..... score():
</imLand/>>
Code Block>
norm2_ torch.no, no_grrd())
for p paransters()
hasattr(p,
In-context Example
nore2_sur
p.grad
torch.sorr(p.graa.
uset
grad_noce float(torch, (torch sqrt(somm_mmmuuon
Mutate the architacture.
{(architecture_inget
</m end/>
Im sortsttrt/sssiiii
cthinkingscesplan the nutation process></ thinking
coutput
architecture>
("arch" sarchitectuurr genotypes,
< <|m_and>>
"type" sczero-cost typao, :c2er-oo2tt scores]
User Prompt
</insstar|>suser
architecture.
(*arch" (farchitecture]], "type":ctypex, "score": :cccores)


Figure Overview of architccture mutation prompt. Left: the structure of the whole prompt template. Right: four text-level and code-level input descriptions.
mechanism which is highlighted with XML-like tags to guide LLM in reasoning. The example can be tailored for different downstream tasks. Previous research (Min et al., 2022; Liu et al., 2024) has done extensive experiments to show the effectiveness of contextual learning. The in-context examples store successful cases as key-value pairs, which help the induction heads of the transformer identify successful architectural patterns stored in the context examples (Olsson et al., 2022). Our extensive experimental results show that the example code is crit- ical for LLM reasoning, leading to more consistent and correct architectures without syntax exceptions and en- suring good performance. To avoid the issue of prompt leakage, we add restrictions in the system prompt pre- vent prompt leakage. Because there could be cases where in-context examples output mutation results dur- ing our experiments. To address this, we include "do not disclose examples" in the system prompt. The Zero-Cost
NAS method in the in-context example should align with the method selected by the user in the following prompt.
User Prompt. This part defines the format and content of user input. The user input the architecture be mutated, which is randomly selected from the population. The random selection stragegy aligns with existing Zero-Cost methods such as Zen-NAS (Lin et al., 2021). The input is provided in JSON format, containing three keys: "arch", "type", and "score". Users must specify the Zero-Cost method to be applied. The "score" field initialized with the computed Zero-Cost score of the selected architecture.
The whole prompt design is presented in Appendix A.3. After L.L.-guided mutation, the mutated architecture is subjected to a syntax correctness check and verified for construction errors to prevent potential exceptions. an exception occurs, the process skips invalid architecture, and the exception message is sent to the L.M reflection module
6

---

RZ-NAS: Enhancing L.MMguidee Neural Architecture Search via Reflective Zero-Cost Strategy
Table The and validation accuracy on NAS-Bench-201. Top: traditional differentiable NAS strategies. Middle: previous I.I
NAS methods. Bottom: comparison of each Zero-Cos method's performance against our improved results. Each method is run times. The best results between the LLM- based and Zero- Zero-Cost methods are shown in bold.
CIFAR-10
CIFAR-100
ImageNet-16-11
Methods
valid
test
valid
test
valid
test
DARTS(2nd) (I.in et al, 2019)
39.77 + 0.00
54.30 0.00
15.03 + 0.00
15.61 + 0.00
16.43 + 0.00
16.32 +0.00
SNAS (Xie et 2019)
91.10 1.04
92.77 = 0.83
69.69 + 2.39
69.34
+ 1.98
42.84 1.79
43.16 1:66::20 2.64
PC-DARTS (Xu et al., 2020)
0.15
93.41 = 0.30
67.12 + 0.39
67.48
+ 0.89
40.83
0.08
41.31 - 0.22
RLNAS (Zhang et al., 2021)
89.91 0.00
93.35 = 0.00
70.98 + 0.00
70.71
0.00
16.86
0.00
13.70
DrNAS (Chen al., 2021)
91.55 0.00
94.36 0.00
73.49 0.00
73.51
0.00
46.37
0.00
46.34 0.00
B-NAS (Ye 2022)
91.55 0.00
94.36- 0.00
73.49 1:
73.51
0.00
0.00 46.37
46.34 :: 0.00
GENIUS
91.07 0.20
93.79
0.09
70.96 + 0.33
70.91
0.72
45.29 0.81
44.96 1.02
LLMatic
91.42 0.13
94.26 0.10
71.41 + 1.44
71.62 1.73
44.98 0.87
45.87 0.96
GraSP
89.69 1.39
89.34
2.16
61.11 +
60.89 3.88
24.17 + 10.38
22.99 +: 11.01
Gras(GraSP)
90.11 1.00
92.79
0.80
69.69 + 2.39
69.31 1.98
42 81 1.79
43. 16 2.64
Gradnorm
89.69 + 1.39
89.27
2.10
G1.11 + 4.17
61.06
4.02
24.17 + 10.38
24.10 + 11.36
Gradooonn)
90.76 + 1.77
93.99 3.44
71.03 + 2.34
71.61
+ 2.20
42. 13 1.40
42.61 1.38
Synflow
91.03 + 01.44
0.00 93.48
71.36 + 1.50
71.35
1.51
44.87 + 0.00
45.12 + 0.00
Ours(Syuflow)
91.45 0.42
73.23 1: 0.14
73.21 2.112
46.37 0.11
46.16 0.30
Zen-NAS
90.20 1:200:::0 0.00
93.76 - 0.00
70.21 + 0.71
70.67 + 0.62
40.78 + 0.00
OursiZen-NAS)
01.03:0..11
93.18
0.00
71.36 + 1.51
71.35
+ 1.51
41.77 0.00
45.12
89.94 01.00
93.35 0.00
70.98 + 0.00
7U.71
0.00
46.39 0.03
46.18 0.04
OursiZiCo)
91.45 0.10
91.24 0.12
73.35 1 0.14
0.21 73.30
0.24 46.53
46.24 :: 0.23
for the error analysis, enabling the LLM generate valid architectures in the next iterations.
Table 3. The correlation coefficients between various Zero-Cost proxies and test accuracy on NAS-Bench-201 (KT and SPR repre- sent Kendall's and Spearman's Φ, respectively). The best results are shown in bold.
4. Experiment
NAS-Bench-201
We evaluate RZ-NAS on multiple widely adopted Zero-Cost NAS proxies for different downstream tasks. The details of Zero-Cost proxies are provided in Appendix A. We define the mutation space based on the set of architecture opera- tors specified in the system prompt. The search procedure runs for 1500 evolutionary iterations. The population size is set to 100 for NAS-Bench-201 and CIFAR-10, 10, and 256 for -00, ImageNet, and COCO. All populations are initialized from scratch using random sampling. The search spaces differ by task: for MAS-Bench-220, CIFAR-10, and CIFAR-100, use the micro cell- search space; for ImageNet, we adopt the MobileNet macro search space, con- sistent with prior works like Zico and Zen-NAS. For COCO object detection, we stack operators to build the backbone following the same configuration used in MAE-DET (Sun et al., 2022). In all experiments, we use the GPT4o model to generate mutations. We also perform an ablation study with different LLMs in Appendix A.4.2. We sample the temperature of the model from [0.2, 0.4, 0.6, 0.8, 1.0] to encourage output diversity. The other settings are identical to different Zero-Cost proxies. In RZ-NAS, the number of input tokens and output tokens is in the range of 2300-2600 and 1 150-200, respectively. We perform 550 iterations for one proxy in one search space per proxy. Therefore, the total cost per proxy is around $75.
CIFAR-10
CIFAR-100
Image
Method
KT
SPR
KT
SPR
KT
SPR
GraSP
0.37
0.54
0.36
0.51
0.40
0.56
Ours(GraSP)
0.43
0.57
0.41
0.55
0.44
0.59
Synflow
0.54
0.73
0.57
0.76
0.56
0.75
(urs(Synflow)
0.56
0.73
0.60
0.79
0.58
0.78
Zen Zen-Score
0.29
0.38
0.28
0.36
0.29
0.40
Ours(Zen-Score)
0.37
0.54
0.36
0.51
0.40
0.56
ZiCo
0.61
0.80
0.61
0.81
0.60
0.79
Ours(ZiCo)
0.63
0.82
0.63
0.84
0.64
0.81
4.1. Performance Comparison
4.1. 1.1.1. NAS AAS-BENCH-201 SEARCH SPACE
We compare the accuracy performance on the NAS-Bench 201 Benchmark shown in Table 2. We train the network with the best score for each proxy for three runs under its training setting. RZ-NAS can significantly improve the per- formance of all Zero-Cost proxies. We further compare the performance with existing LLM-to-NAS methods available in published results or open-source implementations. Ta- ble 2 demonstrates the superior performance of RZ-NAS. We also compute Kendall's J and Spearman's 's o correlation coefficients between the Zero-Cost proxy and test accuracy to evaluate whether these proxies accurately rank architec- tures in Table 3. Our method improves the correlation scores of all Zero-Cost proxies across the three datasets, indicating that our method can help Zero-Cost proxies have a greater
7

---

RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy
Table Performance comparison on CIFAR-10 and CIFAR-100 on the DARTS search space. For the "Method" column, "MS" represents multi-shot NAS: "OS" short for one-shot NAS: "ZC" short for Zero-Cost NAS. The best results are shown in bold.
aho Incortter Duanle auraaeetreeetee whle wno Teetal Cercrption
Vic h-comen Da
Dute Drsetdio
Tedka Oeerrgiion
Method
Test Error [%]
Method
CodtGGU days)
CIFAR-10
CIFAR-100
PNAS ILiu al., 2018)
3.41 0.00
19.53
ENAS (Pham et 2018) 2018)
2.89
1943
RI.
0.5
DARTS(2ad) (L.in ot al. 2019)
2.76 + 0.05
20.58 + 0.44
0S
(a) NAS-Bench-201
(b) DARTS
SNAS Oiie al. 2019)
P.DARIS (Liu al., 2018]
2.5
17.46
0S
0.3
R-DARTS (Chenn et ol., 2020)
2.90 +: 0.21
18.34
08
1/6
Figure Ablation results on different search spaces.
DARTS-PT
2021)
0.06
B- DADRTS (Yo et al., 2022)
2.53 0.08
17.43
08
0.4
OLES (Jiang ot ol., 2023)
2 0.11
17.33
OS
0.4
GraPP
0.33
27.8 0.17
lights the strength of RZ-NAS in balancing performance and computational cost, making it particularly advantageous in scenarios where fast architecture search is essential, es- pecially in resource- resource-constrainee constrained environments. Moreover, for given Zero-Cost method, our proposed search strat- egy built on this Zero-Cost method can always achieve an improvement in accuracy across different FLOP budgets.
Ours SP)PP)
6.07
26.8
7C
0.02
Gradnomt
5.7 0.006
25.7 1.008
ZC
0.01
Grddio)))
0.22
24.6 0.116
Synflow
6.1+0.11
0.112
7C
0.02
Ours Syntlow)
0.07
21.3 0:20
ZC
0.02
Zeo: NAS
1.50 0.04
19.9 0.111
Ouns NAS)
17.77
ZiCoo
2.45 0.111
17.78
+ 0.13
ZC
0.05
OusaZiCo)
2.41 + 0.13
17:48 + 0.08
7C
0.03
Table 5. Top-1 test error on ImageNet under various FLOP budgets. For the "Method" column, "MS" means multi-shot NAS; "OS" is short for one-shot NAS; Scaling represents network scaling methods; "ZC" is short for Zero-Cost NAS. Q: The test error is reported from (Li al., 2023). The best results shown in bold.
ability to select better candidates.
4.1.2. DARTS SEARCH SPACE
We also evaluate RZ-NAS on CIFAR-10 and CIFAR-100 within the DARTS search space, two of the most widely adopted datasets in the NAS research, which were not previ- ously applied in existing LLM-to-NAS studies. As shown in Table 4, we compare RZ-NAS with traditional image clas- sification models, NAS methods, and previous Zero-Cost methods. We adhere to the search space of each Zero-Cost method without introducing additional operations. Follow- ing the computation of each Zero-Cost proxy, our method significantly outperforms previous Zero-Core methods and demonstrates competitive results compared to SOTA NAS methods. Among all Zero-Cost proxies, ZiCo and Zen- NAS emerge as the top- pep-performig runners-up. followed by Synflow.
Rudger(inacimal #F1.0P5)
Approach
Tap-1(()
Method Cos(GPU) days)
EffcienNa
23,9
MS
3500
NasNet-Ba
27.2
1800
DONNA
22.0
QS
25
450M
Zen-NA
27.9
2C
0.
ZC
Z.Co
22.0
ZC
0.4
OusiZZCoo
21.0
ZC
0.4
Emeiemner
21.9
Scaling
3800
DARTS
26.7
05
PC DARTS
24.2
OS
3.8
DONNA
21.6
25
600M
OyrseSy
Synflow
20.0
20.9
ZC
7C
0.25
0.35
Zen-NAS
20.9
0.4
(uns(Zan-NAS)
20.0
7C
0.4
ZoCo
20.6
ZC
0.4
19.9
0.4
19.9
3500
SARPDDRRS
24.0
Scaling
1000M
KAS
19.9
2C
0.5
Zen NAS)
18.9
0.5
4.1.3. MOBILENET SEARCH SPACE
ZiCo
19.5
zC
0.5
Ouro
18.7
2C
0.5
We further adopt widely used Mobile search space, where architectures are constructed by stacking mul- tiple Inverted Bottleneck Blocks with SE modules (Howard et al., 2017; Sandler et al., 2018). RZ-NAS is applied to search for architectures under varying FLOPs budgets (450M, 600M, and 1000M). As shown in Table 5, RZ-NAS using ZiCo demonstrates significant improvements over all prior NAS methods, solidifying its effectiveness in the search for low-cost, high-accuracy neural architectures. For example, with a FL.PPs budget of approximately 450M, our method achieves a remarkable 79.0% Top-1 test accuracy on ImageNet, which is higher than the leading NAS approach DONNA (Moons et al., 2021) and over 62 times faster in search cost. This dramatic improvement in efficiency high-
4.2. Ablation Study
1.2.1. EXPLANATION OF REMOVING DIFFRRENT COMPONENTS
In Figure 3, we evaluate RZ-NAS without in-context ex- ample, reflection module, code description, and textual de- scription on NAS-Bench-22 and DARTS search spaces. This investigation aims to uncover the contribution of each prompt element to the overall effectiveness of RZ-NAS in accurately navigating the search space and identifying high- performing architectures. More detailed explanations and more ablation studies can be found in Appendix A.4.1.
8

---

RZ-NAS:
Enhancing L.M-guudee Neural Architecture Search via Reflective Zero-Cost Strategy
Table Performance comparison CIFAR-10 using NAS-Bench-201 search space. RZ-NAS with the random selection strategy shows
competitive results compared to the highest Zero Cost score-based selection stragety.
Method
GraSP
Zen-NAS
ZiCo
Test Error %
RZ-NAS
5.4 0.07
2.50 0.17
2.41 0.13
RZ-NAS (Highest Proxy Score)
5.38 = 0.12
2.52 = 0.15
2.42 = 0.20
Table Perfromance comparison between the original prompt template and two other different prompts phrasing on CIFAR-10. 10. By keeping the in-context example and code block fixed in the prompt template, use GPT-4o to rephrase the semantic prompt. Different prompt phrasings can influence the outcomes, yet they still demonstrate superiority over the original Zero- Cost methods.
Method
GraSP
Gradnorm
Synflow
Zen-NAS
ZiCo
Original Zero-Cost Proxy
6.0 = 0.13
5.7 0.05
5.9 = 0.11
2.55 0.04
2.45 + 0.11
Test Error
RZ-NAS
5.4 0.07
5.2 0.12
4.3 + 0.07
2.50 0.17
2.41 0.13
%
RZ-NAS (Prompt Rephrase 1)
5.5 0.18
5,2 + 0.08
4.3 + 0.17
2.51 + 0.08
2.40 + 0.20
RZ-NAS (Prompt Rephrase 2)
0.10
5.3 0.07
4.7 = 0.09
2.54 + 0.14
2.44 0.12
4.2.2. PERFORMANCE COMPARISON WITH DIFFERENT ARCHITECTURE SELECTION STRATEGIES
Resthet
In each iteration of RZ-NAS, the architecture selected for mutation is randomly chosen from the population, rather than selecting the architecture with the highest Zero-Cost score. Table shows that the random selection strategy achieves competitive performance. The random selection strategy maintains population diversity and avoids prema- ture convergence. L.M-guided mutations steer low-scoring candidates toward better solutions. Unlike traditional algo- rithms, which may lead to inefficiency, LLM-gguded mu- tations intelligently explore architectures, maintaining di- versity, and accelerating convergence. This aligns with previous LLM-to-NAS work such as ELM (Lehman et al., 2024) and EvoPrompting (Chen al., 2023).
(a) RetinaNet
(b) FCOS
Figure 4. Object detection with RetinaNet and FCOS.
Normalization. Figure 4(a) and Figure 4(b) present the per- formance comparison across different heads (RetinaNet and FCOS). Notably, RZ-NAS improves the COCO mAP while maintaining similar FLOPs.
4.2.3. PERFORMANCE COMPARISON WITH REPIRASING PROMPT
We further evaluate the effectiveness of the structured prompt template. GPT4o is used to rephrase prompts. Ta- ble 7 shows consistent performance with different rephras- ing changes on CIFAR-10, demonstrating the robustness of the structured prompt template in RZ-NAS. Previous research confirms that L.MSs are insensitive phrasing changes in structured tasks when examples, I/O formats, and specific context remain consistent (Raffel et al., 2020).
5. Conclusion
In this work, we proposed a LLM-to-NAS framework via reflective Zero-Cost strategy. We designed a structured, prompt-based approach that combines text- and code-level understanding, enabling the L.MM to generate robust archi- tectures tailored to diverse search spaces and tasks. By cou- pling architecture generation with reflections and training- free metrics, RZ-NAS effectively leverages the reasoning capabilities of LLMs to enhance NAS performance. Ex- perimental results demonstrate that RZ-NAS outperforms slate-of-the-e art LLM-to-NAS and Zero-Cost NAS methods across multiple datasets and search spaces, achieving com- petitive superior performance with significantly reduced search costs. In future work, we will integrate more cost- efficient open-source models into our framework to improve cost-efficiency. These findings highlight the versatility and scalability of LLMs as powerful tools for efficient and high- performance architecture design.
4.3. Object Detection
RZ-NAS can also be applied to search efficient backbones for object detection tasks, leveraging the Zero-Cost detec- tion strategy MAE-DET (Sun et al., 2022). Following the comparison setting, our method is designed to align with traditional ResNet and the MAE-DET strategy with similar FLOPs and parameter counts. For training, we use ResNet- like backbones on the COCO dataset (Lin et al.. 2014), incorporating multi-scale training and Synchronized Batch

---

RZ-NAS: Enhancing LLMMguided Neural Architecture Search via Reflective Zero-Cost Strategy
Acknowledgements
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. MobileNets: Efficient convolutional neural net- works for mobile vision applications. arXiv preprint arXiv: Neee:/7040466,, 2017.
This work was supported by the Frontier Technology R&D Program of Jiangsu Province (#BF2024005), Nanjing Sci- ence and Technology Program (#202304016), and the Col- laborative Innovation Center of Novel Software Technology and Industrialization, Jiangsu, China.
Javaheripi, M., de Rosa, G. H., Mukherjee, S., Shah, S., Religa, T. L., Mendes, C. C., Bubeck, S., Koushanfar, F., and Dey, D. Lite Seaeo:omeeere:: training-fiee free neural architecture search for efficient language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022.
Impact Statement
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Jiang, S., Ji, Z.., Zhu, G., Yuan, C., and Huang. Y. Operation- level early stopping for robustifying differentiable NAS. In Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023.
References
Abdelfattah. M. S., Mchrotra, A., Lukasz Dudziak, and Lane, N. D. Zero-cost proxies for lightweight NAS. In Proceedings of the International Conference on Learning Representations, 2021.
Krishnakumar, A., White, C.,, Zela, A., Tu, R., Safari, M., and Hutter, F. NAS-bench-suute-zero accelerating re- search on zero cost proxies. In Proceedings of the 36th International Conference on Neural Information Process- ing Systems, 2022.
Chen, A., Dohan, D., and So, D. EvoPrompting: Language models for code-level neural architecture search. Pro- ceedings of the 37th International Conference on Neural Information Processing Systems, volume 36, pp. 7787- 2023.
Lee, K., Hwang, D., Park, S., Jang, Y., and Lee, M. Re- inforcement learning from reflective feedback (RLRF): Aligning and improving Ilms via fine-grained self- reflection. arXiv preprint arXiv:2403. 14238, 2024.
Chen, X., Xie, L., Wu, J., and Tian, Q. Progressive DARTS: Bridging the optimization gap for nas in the wild. Inter- national Journal of Computer Vision, pp. 1-18, 2020.
Lee, N., Ajanthan, T., and Torr, H. S. SNIP: single- shot network pruning based connection sensitivity. In Proceedings of the 7th International Conference Learning Representations, 2019.
Chen, X., Wang, R., Cheng, M., Tang, X., and Hsieh, C.-J. DrNAS: Dirichlet neural architecture search. In Pro- ceedings of the International Conference on Learning Representations, 2021.
Lehman. J., Gordon, J., Jain, S., Ndousse, K., Ych, C., and Stanley, K. O. Evolution Through Large Models, pp. 331-366 Springer Nature Singapore, 2024.
Denny, P., Kumar, V., and Giacaman, N. Conversing with
Li, G., Yang. Y., Bhardwaj. K., and Marculescu, R. Zico: Zero-shot nas via inverse coefficient of variation on gra- dients. In Proceedings of the Eleventh International Con- ference on Learning Representations, 2023.
copilot: Exploring prompt engineering for solving esl problems using natural language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, pp. 1 1136-1142 2023.
Lin, M., Wang, P., Sun, Z., Chen, H., Sun, X., Qian, Q.. Li, H., and Jin, R. 7en-NAS: A zero-shot nas for high- performance deep image recognition. In Proceedings of the International Conference on Computer Vision, 2021.
G., Xu, H., Shi, H., Ren, X., Yu, P. L.. Liang, X., Jiang, X., and Li, Z. Autobent-zero: Evolving bert backbone from scratch. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10663 10663-10671, 2022.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision, pp. 740-755, 2014.
Hanin, B. and Rolnick, D. Complexity of Linear Regions in Deep Networks. In Proceedings of the 36th Interna- tional Conference on Machine Learning, pp. 2596-2604. PMLR, 2019.
Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., and Murphy, K. Progressive neural architecture search. In Proceedings of the European Conference on Computer Vision. pp. 19-35, 2018
Hao, H., Zhang, X., and Zhou, A. Large language models as surrogate models in evolutionary algorithms: A pre- liminary study. Swarm and Evolutionary Computation, 91:101741, 2024.
10

---

RZ-NAS:
Enhancing L.MMguided Neural Architecture Search via Reflective Zero-Cost Strategy
Liu, F,, Tong, X., Yuan, M., and Zhang, Q. Algorithm evolution using large language model. arXiv preprint arXiv:2311.15249 2023.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21
Liu, H., Simonyan, K., and Yang, Y. DARTS: Differentiable architecture search. Proceedings of the 7th Interna- tional Conference on Learning Representations, 2019.
(1):5485-55511 2020.
Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. Regu- larized evolution for image classifier architecture search. Proceedings of the Thirty Third AAAI Conference on Artificial Intelligncce, pp. 4780-4789, 2019.
I Liu, S., Chen, C., Qu, X., Tang, K., and Ong, Y.-S. Large language models as evolutionary optimizers. In Proceed- ings of the IEEE Congress on Evolutionary Computation, pp. 1-8, 2024.
Renze, M. and Guven, E. Self-reflection in Ilm agents:
Effects problem-solving performance. arXiv preprint arXiv:2405.06682, 2024.
Lopes. V., Alirezazadeh, S., and Alexandre, L. A. EPE-NAS: Efficient performance estimation without training for neural architecture search. In Proceedings of the 30th International Conference on Artificial Neural Networks, pp. 552-563, 2021.
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S.. Wang, P., Fawzi, O., Kohli, P., Fawzi, A., Gro- chow, J., Lodi, A., Mouret, J.-B., Ringer, T., and Yu, T. Mathematical discoveries from program search with large language models. Nature, 625:468-477 2023.
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M.,, Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048- 11064, 2022.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. MobileNetV2: Inverted Residuals and Lin- ear Bottlenecks. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 4510-4520 2018
Moons, B., Noorzad, P., Skliar, A., Mariani, G., Mehta, D., Lott, C., and Blankevoort, T. Distilling optimal neural Rapid search in diverse spaces. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12229-12238, 2021.
networks:
Serra, T., Tjandraatmadja, C., and Ramalingam, S. Bound- ing and counting linear regions of deep neural networks. Proceedings of the International Conference on Ma- chine Learning, pp. 4558-4566, 2018
Nasir, M. U., Earle, S., Togelius, J., James, S., and Cleghom, C. MLtati:: Neural architecture search via large lan- guage models and quality diversity optimization. In Pro- ceedings of the Genetic and Evolutionary Computation Conference, volume 89, pp. 1110-1118, 2024.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: language agents with verbal reinforce- ment learning. In Proceedings of the 37th International Conference Neural Information Processing Systems, pp. 8634-8652, 2023.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann. B., Askell, A., Bai, Y., Chen, A., Conerly. T., Drain, D., Ganguli, D., HatfieldDDodss, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J.. Lovitt, L., Ndousse, K. Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, In- context learning and induction heads. arXiv preprint arXiv:2209. / 1895, 2022.
Sun, Z., Lin, M., Sun, X., Tan, 7.,, Li, H., and Jin, R. Mae- det: Revisiting maximum entropy principle in zero-shot nas for efficient object detection. In Proceedings of the In- ternational Conference on Machine Learning, pp. 208 10- 20826, 2022.
Sun, Z., Sun, Y., Yang. L., Lu, S., Mei, J., Zhao, W., and Hu, Y. Unleashing the power of gradient signal-to-noie ratio for zero-shot nas. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 5740- 5750, 2023.
Pham, H., Guan, M., Zoph, B., Lc. Q. and Dean, J. Effi- cient neural architecture search via parameters sharing. In Proceedings of the 35th International Conference on Machine Learning, pp. 4095 0004,410,77 2018.
Qin, R., Hu, Y., Yan, Z., Xiong, J., Abbasi, A., and Shi, Y. FL-NAS: Towards fairness of nas for resource con- strained devices via large language models. In Proceed- ings the 29th Asia and South Pacific Design Automation Conference, pp. 429 434, 2024.
Tanaka, H., Kunin, D., Yamins, D. L. K., and Ganguli, S. Pruning neural networks without any data by iteratively conserving synaptic flow. In Proceedings of the 34th International Conference on Neural Information Process- ing Systems, pp. 6377, 2020.

---

RZ-NAS: Enhancing L.MM-guided Neural Architecture Search via Reflective Zero-Cost Strategy
Termritthikun, C., Jamtsho, Y., leamsaard, J., Muneesawang, P., and Lee, I. Eeea-net: An early exit evolutionary neural architecture search. Engineering Applications of Artificial Intelligence, 104:104397, 2021.
Zhang, L., Ergen, T., Logeswaran, L.., Lee, M., and Jur- gens, D. SPRIG: Improving large language model per- formance by system prompt optimization. arXiv preprint arXiv:2410. (tth:2410.14826, 2024.
van Stein, N. and Bäck, T. Llamea: A large language
Zhang, X.,
Hou, P., Zhang, X., and Sun, J. Neural archi- tecture search with random labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10907-10916, 2021.
model evolutionary algorithm for automatically generat- ing metaheuristics. IEEE Transactions on Evolutionary Computation, -932):3313345,2 2024.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang. J., Dong, Z.,, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 2023.
Wang, C., Zhang, G., and Grosse, R. Picking winning tickets before training by preserving gradient flow. Proceedings of the International Conference on Learning Representations, 2020.
Zhu, G., Wang, W. Xu, Z., Cheng. F., Qiu, M., Yuan, C., and Huang, PSP: Progressive space pruning for effi- cient graph neural architecture search. In Proceedings of the IEEE 38th International Conference on Data Engi- neering, pp. 2168-2181, 2022.
Wang, Cheng, M., Chen, X., Tang, X., and Hsieh, C.-J. Rethinking architecture selection in differentiable nas. In Proceedings of the International Conference on Learning Representations, 2021.
Wei, L., He, Z., Zhao, H., and Yao, Q. Unleashing the power of graph learning through Ilm masdased autonomous agents. arXiv preprint arXiv:2309.04550 2023.
Zoph, and Le, V. Neural architecture search with rein- forcement learning. In Proceedings of the International Conference on Learning Representations, 2017.
Xiang. L., Dudziak, L., Abdelfattah, M. S., Chau, T., Lane, N. D., and Wen, H. Zero-cost operation scoring in dif- ferentiable architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10453-10463, 2023.
Xie, S., Zheng, H., Liu, C.,, and Lin, L. SNAS: stochastic neural architecture search. In Proceedings of the Interna- tional Conference on Learning Representations, 2019.
Xu, Y., Xie, L., Zhang. X., Chen, X., Qi, G.-J,, Tian, Q.. and Xiong, H. PC-DARTS: Partial channel connections for eeficie-effiiennt architecture search. In Proceedings of the International Conference on Learning Representa- tions, 2020.
Ye, H., Wang, J., Cao, 7.,, Berto, F,, Hua, C., Kim, H., Park, J., and Song, G. ReEvo: Large language models as hyper-heuristies with reflective evolution. Proceedings of the International Conference on Neural Information Processing Systems, 2024.
Ye, P., Li, B., Li, Y., Chen, T., Fan, J., and Ouyang, W. b-darts: Beta-decay regularization for differentiable chitecture search. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10874-10883, 2022.
Yu, C., Liu, X., Wang, Y., Liu, Y., Feng, W., Deng, X., Tang, C., and Lv, J. GPT-NAS: Evolutionary neural architecture search with the generative pre-trained model. arXiv preprint arXiv:2305.05333 2023.
12

---

RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy
A. Appendix
A.1. Illustration Zoro-cost Proxyies
We illustrate the mentioned zero-cost proxies in our experiment.
GraSP (Wang et al., 2020). It attempts to approximate the change in gradient norm (instead of loss), which denoted as,
S(0) = -H 9 L
0
(2)
00
where L is the loss function of neural network with parameters 0, H is the Hessian.
Gradnorm (Abdelfattah et al., 2021). Gradnorm sums the Fuclidean norm of the gradients after a single mini-batch of training data, which can be denoted as,
N
SGralNorm (0)
VoL(T,, ys: 0)
(3)
where L( 0) the loss function parameterized by 0, and Ve represents the gradient with respect to 0.
Synflow (Tanaka et al., 2020). Synflow simply computes the loss, which the product of all the parameters in the network.
0L @ 0
S(
(4)
00
ZenNAS (Lin et al., 2021). ZenNAS approximates the gradient w.r.t feature maps and measures the complexity of the neural network. It computes
6, -.= oig//
(5)
where oij is the mini-batch standard deviation statistic of the j-th channel of the i-th BN layer.
ZiCo (Li et al., 2023). ZiCo is the SOTA Zero-Cost NAS method that can consistently surpass many conventional NAS strategies. Zico has relatively complicated computation method that considers both absolute mean and standard deviation,
Σ log
E L(X, uy θ| WEOL VVar(VV Mi θ)I)
(6)
where O denotes the initial parameters of the given network, O denotes the parameters of the network, and represents each element in θ, X; and y, are the ith input batch and corresponding labels from the training set.
MAE-DET (Sun et al., 2022). Beyond image classification, MAE-DET extends to the field of object detection and instance segmentation. DeepMAD regards the neural network as an information system and calculates the differential entropy H (x)) of the output feature map,
((x) 1 log (ar((((())))) +
Σ log
(7)
where i.P is the variance of the last feature map and of is constant.
A.2. Example of Reflection
We provide example of the iterative reflection under the NAS-Bench-22 search space and the GradNorm Zero-Cost proxy.
The input of the randomly selected architecture:
("arch":* mhor_come 1x1"Di Iskip_ laski__cconcccccpppse connect "D 11 connect " sk skippconneett
skip conpect "1"", "1", "type": "Gradnorm", "score":3.9)
13

---

RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy
Output after LLM mutation:
("arch":" Inor _noo__oov________ 1x1 1 Inno__cong _oov__xxl 1x11xx10 skisssconneessee + | none" sk skip_connect "2,", "type": "Gradnerm", "score":1.2 exception:"") "":
Output of the LLM reflection module:
of the axip_conppee connect operations with 3x3 the cell. Convolution operations like 3x3 Cony can capture more spatial features compared to skip connections, potentially improving the architecture's pezformence." Our architecture Genotype might exceed the limit of MXX_LAYERS LAYERS. The exception will output Error: out of the 1imit of MAX MAX_MAYRRS."
"Ixy replacing
increase the expresaiveness of
Output of reflection module with exception:
"The "MAX LAVERS" error indicates that the architecture exceeds the allowed number of layers. You reduce the number of layers the architecture For example, consider reducing of the ResK1K5K1 layers to simpler operation like ConvK1ENRE10 or reduce the number of channels to stay within the limit."
A.3. Prompts
We present example below to illustrate how design the prompt.
A.3.1. SYSTEM PROMPT
Define Role and Task:
I am an expert in understanding the zero-cost NAS methads and evolutionary algorithms of neural architecture search.
T am good at Python programming evoluti orary algorithms.
# My task is to guide the mutation process of the architecture to get an optimal architecture with better zero-cost proxy score. should understand given dataset, search space and network inaeritttion,, and mutate the selected arch itecture textual genotype to generate better models.
I should only generate valid Python code.
Define Search Space (Here we take an operation in the search space as an example):
should generste an optimal architecture by mutating an operation in the search space. A11 operations defined below.
vvclass SuperResk1K331vv. This class defines a residual block with convolutional layers having 1x3x1. It designed for applications that zequize smaller kernel sizea to ext local features while maintaining a residual connection for improved gradient flow.
kernel
The iaitt___eelf in_channels, out_channels, stride, bottleneck channnels,,a sub_layers)" parameters are:
in in__hannels': Number of input channels.
"out_channels's Number of
channels.
"stride": Stride length for the convolational layers.
"battleneck, "aattleencccccmmeess" Number of channels the bottleneck layer reduced
dimensi ona ity..
"sub_ Ssub_layers" Number of sub-layers the block.
Punctiio "forward" code:
""python
def (srward
output -
block self.bl belf.block... list:
output block (output)
return output
System Reflection
the expert in the domain of evolutionary mutation. should reflect and give hints design better architectures.
Network Construction
14

---

RZ-NAS: Enhancing L.M-guidde Neural Architecture Search via Reflective Zero-Cost Strategy
Input architecture is defined string variable named "block_str'. acoperat copppa ion2 bl oock str is
format 1s:
operationl paraa)
para),,, For example, the 80, "SuperConvK3BRRMIl (3, 64, ScperResk (64, 168, 16, 3) ScperrRess R3K1
para2)) 2...aaa).....ppe......... koperat ionk para2)a,
The parameter "bottlenec" "mncck_channes"" ard "out chaanehs **must not exceed+A 2048 and *Amust be multiple of 8, The "sub_las sub_layers" should be least 1.
The output channela of the previous block should match the input channels of the next block. pyttoon
de creste__looe
block_list _ist__ffreeee from_sstrt__
while len(s) 0:
is fl.oonnd class False
tae_block_ carc__clas__aaae __e_block_class_name
blocks_dict keys
Imp _ip_idx s.find('
imp_idx is L.foound_blcckclaae
@(0:tmp_idx) the bee_blocc_class_nee
= True
the_block_olass _alllaam_ ael._neelocss_diir__eeae (tteee___tthhehhhkkiikk
the_block, remaining_s bte_bloc_cllas sromte_ vo_create-to_creat ** kwargs)
if the_block is None:
(ttpe//atttapeettteeeeeeeeeeeeee remaining_s
if len (s) > 0 and s(0] == '/
return biock_list, s[1:]
break pass end pasa end block_olass
assert is ss_found pass return block_list class Maste:Net [assexsee(PPaineeePPlaaaeee
bistkllokkkttt_.._o__ def init _ntt__iitttt_____ (seli,):
self.block_lis def forward(self, output - for block_id, the_block in enumerate (self. bbll
(bl pcr_str)
output block (output)
oatput F.adaptive_avg pod_e_llllloodddoeooooooe pool2 oatput_ssze-_]
output torch. flatten (output, output = self.fe. fc_ inar (output)
return output
Architecture Mutation:
should anderstand the key code architectures mutate one operation the architecture by replacing it with another operation defined in the search space list. can mutate the operation type and paraneters of oparations.
Compute Zero-Cost NAS Proxy:
The type zero-cost proxy is gradnorm. The computing code is below.
"''ython
import os, sys, time
hys.path.appente (os, pats.ppahh... di rname (os path. pathhdirzae daame (oa. ( 11 import torch
from torch inport nn import numpy np
def Nettwork_weihhte wtt.:/eeettt_sssssaiiiiiiiiii (net: an.oodlle)
with torch.
for m in met.modulles):: if isinstance conv2d)
nn. inii. ioii. if basattr "bias') and m.bias is None:
inb.init.zerss (m.blas)
elif isinstance (m, Bathblncc nn nn. mm.rrpppoom)) (tw.tt.ooos___m..ee._eeee nn. (eoiit.eero__mm.bb (m.bias)
elif isinstance
In.
nn. iiiiiii... www.int.normaliitt in.
if hasattz (m, 'blas') and m.bias is None:
mn.init. zeros_ (m.bias)
else:
continue return net
15

---

RZ-NAS: Enhancing L.MM-gudde Neural Architecture Search via Reflective Zero-Cost Strategy
import torch. nn. functional as def ropy
target):
target be one-hot format!!
prop_ prob_logit F.log_ softmmx (logit, dim=i)
loss (target prob_logite suum (dim-1) mean return loss
def compuse_nas_sccne (gpu, model, resolution, batch_size)
model.crain()
gradd_rr_____ (True)
model. zedel.zer__grad
if qpu is None:
torch. model model.cuda (gpu)
(gpu)
wwwpwork_weehh__aaaae_e (aanchhsssie---atttossiseee 3, resolution, resolution))
(model)
input =
if gpu is not Nene:
input input. aopput (qpu)
output = model (input)
y_true torch. (size- (batze aize, output. shape [1]1, device-tozch.device ('cuda: [) .ormat
(gpu))) 1e-10
y_true y_true torch.sum(y_truun dim=1, keepdim=True)
num num_classes shape torch.randint (low-0, high-num_classes, size- (izee(batch_siee)
oce_hot_y F.one_hot (y, cim_clases) () if is not None: one_hot_y
cuda
loss cross_ eross_enntroyy (output, one m)
loss. backward)))
norm2_ aum 0
torch.no g torch.no_grad():
for p in model.parameten () ():
if hasattr (p, 'grad') and p.grad not None: norm2 sum +- torch.norm (r.grad)
grad_norm float (torch. (norm2_sum)
return grad_norm
A.3.2. IN-CONTEXT EXAMPLE
Define Role and Task:
Here is the example:
iminsstrrt / Jsuser
Mutate the below architecture. "*"jaon
("arch": " 1, 1) (32, 120, 40) ResX1K5X1 (120, 176,2, 32)", "type": "synflow", "score":0.91
<lim_end|>
<|im_star: pta.sisst/sssistantt
**'aaon
32,1, ResKiKšK1 (32, 120, 40) ResK1K5K1 (120, 2,),2,32) "type": "aynflow",
"score":1.9)
<Thinking>
BNRELU" and get the optimal architecture by improving the zero-cost NAS score from 0.9 to 1.9. Other mutation methods cannot get such high score. </thinking>
mutate "ConvX3BNRELL"" into "ConvX1BNRELJ""
<lim_nd]]>
A.3.3. USER PROMPT
Define Role and Task:
Now given the new architecture belox with new user: <l stimssssrr/seee Jauser
16

---

RZ-NAS: Enhancing L.MMguuddd Neural Architecture Search via Reflective Zero-Cost Strategy
Mutate the below architecture. :..j
""type" "type"":<type>" " "score" <score>)
<l <mm_nnd]>>
[mm_start] p>assistant
jaon
A.4. Ablation Study
1. EXPLANATION OF REMOVING DIFFERENT COMPONENTS
The Role of In-context Example We create a "w/o In-context Example" variant by removing the in-contextual example module from the LLM imput prompt and conducting the same search process on MAS--eecc--201. We found that this setting performs worse with increased exceptions from 2% to 7% and worse accuracy. This demonstrates the critical role that in-context examples play in guiding the LLM to make informed predictions and decisions by providing it with prior examples of successful architectures. Without these examples, the model struggles to establish meaningful patterns in the search space, leading to suboptimal architecture selections and a higher likelihood of generating invalid output.
Reflection Module The "w/o Reflection Module" variant is created to reflect on previous architecture performance and error exceptions. The absence of reflection leads a performance drop and increased exceptions. Without the reflection module, we observe significant decline in performance and noticeable increase in exceptions from 2% to 5% during the search process. This suggests that the absence of reflection leads to a lack of iterative improvement, as the system fails to learn from past successes and failures.
The reflection module not only improves accuracy, but also reduces the exception rate by allowing the model to adjust its search strategy dynamically. For instance, when encountering frequent invalid architectures, the reflection module helps the system identify problematic patterns in the search and generate solutions that better adhere the constraints of the search space. Furthermore, reflection enables the system adaptively refine its exploration of the search space, prioritizing areas with higher potential while discarding unpromising regions. Without this module, the RZ-NAS framework essentially operates in a static, trial-andeerro manner, resulting in lower search efficiency and less competitive performance compared to the full RZ-NAS setup.
Code Deseription We create a "w/o Code Description" variant by removing the three code descriptions in the system prompt that detail the NAS process and architecture construction. This results in a significant decrease in RZ-NAS performance, highlighting the importance of these descriptions in providing contextual guidance the LLM. These descriptions help the LLM understand the intricacies of the NAS process and ensure that its outputs align with the search objectives. Without these descriptions, the model struggles to interpret the search space effectively, leading to less accurate and less efficient architecture searches.
Textual Description We also create a "w/o Textual Description" variant by removing textual explanations from the system prompt and relying solely on instructions the user prompt and code blocks. The results show that textual descriptions play a crucial role in aligning the LLM's understanding of the problem and its objectives. Without them, the code blocks alone are insufficient, especially given the complexity of the search space and network construction in RZ-NAS. Unlike simpler and smaller search spaces in prior studies, RZ-NAS involves more sophisticated setup, requiring detailed textual descriptions to provide the necessary context for effective search guidance. This finding highlights the need for a balance between textual and code-based inputs to maximize the effectiveness of the LLM in NAS tasks.
A. A.4.2. PERFORMANCE COMPARISON ON DIFFERENT LLMS
We evaluate RZ-NAS with different LLMs under the TAS-Bench-201 search space. This comparison allows us assess how the choice of LLM affects the overall effectiveness of our approach. To ensure a comprehensive comparison, select three widely used LLMs that have garnered significant attention in the research community. In addition to GPT4-o, which has established itself as a leading model in various applications, we include L.LaMA 3.1 3, which displays impressive code generation capabilities. Furthermore, we incorporate Claude 3.5 4. Known for its strong performance in understanding and
h mwta. comaaacco//aaoo wtt.cc/m//llo/e "https://claude" http///llg//eaaaaeee/ 1 3-
17

---

RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy
Table Accuray Comparison under the NAS-Bench-201 search space with different L.L.s. The hest results shown bold.
CIFAR-10
CIFAR-100
ImageNet-16-11
Method
GPT4-o
Llama 3.1
Claude3.5
GPT4-o
Llama 3.1
Claude3.5
GPT4-0
Llama 3.1
Claude3.5
GraSP)
92.79 0.80
91.67 = 1.02
89.97 + 0.21
69.34 1.98
67.45 +: 0.79
66.31 = 0.33
43.16 + 2.64
12.45 +: 0.39
41.23 0.18
Ours (Gradnorm)
3.441
93.4120.30
93.21 0.13
2.220
67.48 0.89
61.12
42.61 1.38
41.48 +: 0.39
41.83 0.28
SynflSy))
93.51 0.74
0.44
93.08 + 2.42
73.21 0.12
0..3
71.59 0.32
46.34 0.32
0.14
45.22 1.10
Ours(Zen-NAS)
93.48 0.00
93.34 + 0.27
92.75 +: 0.81
71.35
70.68
+: 0.60
60.91 # 0.88
45.12 +: 0.00
41.44 + 0.10
41.43 0.40
Ours(ZiCo)
0.12
93.31 = 1440
91.55 + 0.72
0.221
71.37 + 2.13
71.23 1.11
46.18 = 0.33
15.37 0.12
44.37 0.30
Table 10. Performance comparison between the temperature sampling method and the fixed temperature setting on CIFARIO under the MAS-Bench-200 scarch space.
GraSP
Gradnorm
Synllow
Zen-NAS
ZiCo
Temperature
valid
test
valid
test
valid
test
valid
test
valid
test
0
89.91=0.11 II
12.11=1.24
90.65:00.41
03.12=0.63
90.90=0.33
92.89-03 0.56
90.37=0.31
0..44:0.22
90.65-0.22
93.68+0.32
0.2
90.08:40.45
92.55=0.53
90.74:0.65
93.44=0.94
91,47=0.58
93.47=0.86
90.52=0045
92.81=0.255
90.94=0.20
93.89=0.25
0.4
90.10:1..4
92.79:00446
90.76::1.10
93.91=1.07
90.89=0.32
92.85=0.44
90.83=0.30
92.82=0.47
91.22=0.14
94.11::022
49.9240.44
92.34=+0.77
1.144:1.21
93.02=0.89
91.01=0.29
93.18-061
91.03=0.87
93.46=0.61
91.15=0.12
94.04+0.14
0.8
89.98=00.74
92.20=1.08
1.,48:+1.32
93.20=1.72
01.10=0.27
93.02-0.53
90,77=0.55
92.41=0.44
90.93=0.63
93.62+0.41
1.0
19.45=11.10
89.33
90.10::1..9
93.10=22.01
90.91=0.85
92. 02.89-0.76
90.63=20.66
92.97:0083
89.96=0.13
93.48:0.40
Uniform sampling
90.11=1.00
92.79=0.80
10.76::1.77
93,99#3444
91.45:0442
93.47=0.74
91.02:00.44
93.48:00.00
91.45=0.10
94.24::012
generating natural language, Claude 3.5 has also shown potential in code-related tasks. The results are shown in Table 9, our findings reveal that RZ-NAS delivers similar results across different LLMs, while the GPT4-o model consistently outperforms the other LLMs in terms of accuracy.
A.4.3. PERFORMANCE COMPARISON UNDER DIFFERENT TEMPERATURE STRATEGIES
In RZ-NAS, the temperature of the large language model is sampled uniformly from the domain, not fixed value. The comparison with different fixed temperatures is shown in Table 10. The results demonstrate the superiority of the temperature sampling approach in RZ-NAS.
Accuracy nat Grasitorm and Cradiiia
The Accurncy Zan
R2-4A5iZen
20Cs and 42--455
Deatt vest
Time por (ontwokt
Soarrh (seva168))
Searrn Time (SRTOWR))
(a) GraSP
(b) Zen-NAS
(c) ZiCo
Figure 5. Visualize the test accuracy of three original Zero-Cost strategies our LLM L.M-gudded methods on CIFARIO under the NAS- Bench-201 search space over time. We record best test accuracy every 50 seconds in the population.
A.5. Visualization of Test Accuracy Comparison Between Original Zero-Cost Strategies and RZ-NAS
We visualize the test accuracy comparison between original Zero-Cost proxies and RZ-NAS over time in Figure 5. The CIFARIO 10 dataset and the NAS-Bench-201 search space are used. The results show that our approach achieves higher test accuracy during the training process compared to the original Zero-Cost strategies.
18


Reviews:
[{"review_id": "GPrQm7ZWDr", "reviewer": "Reviewer_oqYW", "summary": "The paper presents an LLM-guided NAS framework that leverages Zero-Cost proxies for evaluating neural architectures without full training. The approach employs a reflective strategy in which the LLM mutates candidate architectures and, via a reflection module that uses training-free proxy scores (e.g., GraSP, Synflow, Gradnorm), provides feedback to guide subsequent mutations. The method is evaluated across multiple NAS benchmarks and claims to achieve competitive or superior performance while significantly reducing search cost.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your insightful comments and detailed suggestions! We hope our responses address your concerns and enhance your view of our work!\n\n**1. Detailed explanation about reflection mechanism**\n\nThe reflection mechanism includes two modules:  internal system reflection in the system prompt (e.g., `Reflect to generate better mutation`) and  external LLM reflection module (e.g., `Ask for suggestion...`). The external module takes the current iteration's architecture, zc proxy score, and potential exceptions as input, and outputs dynamic, structured reflection suggestions. These two modules form a closed-loop feedback system: the external module provides explicit optimization directions, while the internal module enables implicit semantic reasoning. Their efficacy is validated in ablation studies.\n\nThe role of `line 15` in Algorithm 1 refers to the external LLM reflection module. Next, we provide an example of the iterative reflection under the NAS-Bench-201 search space and the GradNorm zc proxy.\n\n***Input of the randomly selected architecture***\n\n`{\"arch\":\"|nor_conv_1x1~0|+|skip_connect~0|skip_connect~1|+|skip_connect~0|skip_connect~1|skip_connect~2|\",\"type\":\"Gradnorm\",\"score\":3.5}`\n\n***Output after LLM mutation*** \n\n`{\"arch\":\"|nor_conv_1x1~0|+|nor_conv_1x1~0|skip_connect~1|+|nor_conv_3x3~0|none~1|skip_connect~2|\",\"type\":\"Gradnorm\",\"score\":4.2, exception:\"\"}`\n\n**&Output of the LLM reflection module***\n\n`\"Try replacing one of the skip_connect operations with 3x3 Conv to increase the expressiveness of the cell. Convolution operations like 3x3 Conv can capture more spatial features compared to skip connections, potentially improving the architecture's performance.\"`\n\nOur architecture `Genotype` might exceed the limit of `MAX_LAYERS`.  The exception will output as `\"Error: out of the limit of MAX_LAYERS.\"`\n\n***Output of reflection module with exception***\n\n`\"The \"MAX_LAYERS\" error indicates that the architecture exceeds the allowed number of layers. You can reduce the number of layers in the architecture. For example, consider reducing one of the ResK1K5K1 layers to a simpler operation like ConvK1BNRELU or reduce the number of channels to stay within the limit.\"`\n\n**2. Initial architecture F0**\n\nThe initial population is randomly generated and the zc sore is calculated for each architecture in the candidate pool. This process aligns with the evolutionary strategy algorithms used in zero-cost NAS (e.g., Zen-NAS, Zico), and we have not modified this process.\n\n**3. Issue with Equation 1**\n\nThank you for highlighting the issue with Equation 1. The revised equation is:\n\n$a^∗=argmax_{a \\in A} O(a,T,S,D,M)$\n\nHere, $a \\in A$ is the architecture being evaluated, $O(a,T,S,D,M)$ is the Zero-Cost proxy score for architecture $a$, given task  T, search space S, dataset D, and model M.  The expectation was unnecessary and has been removed, as our method deterministically evaluates architectures using $O$. We will update the equation and surrounding text in the revised manuscript to reflect this correction.\n\n**4. Related work needs more detail**\n\nThank the reviewer for helping improve the paper's clarity and readability. We agree that adding clearer explanation of Zero-Cost NAS and the definition of zc proxies in the main text is necessary. In the revised version, we will include more detailed descriptions of them. Moreover, we will give more explanation about Hao et al.'s (2024) you mentioned, which encodes architectures as natural language via prompt engineering and using LLMs to predict proxy scores, reducing computational costs.\n\n**4. Some statements should be clearly explained**\n\nThanks for the valuable suggestions. The in-context examples store successful cases as key-value pairs, which help the induction heads of transformer to identify successful architectural patterns stored in the context examples. We will further elaborate these statements in the revised version.\n\n**5.  More insights on prompt leakage**\n\nWe add restrictions in the system prompt to prevent prompt leakage. Because there could be cases where in-context examples are output as mutation results during our experiments. To address this, we included \"do not disclose examples\" in the system prompt. This approach has been studied in prompt engineering and open-source prompt libraries to address the issue of prompt leakage[1].\n\n[1] https://docs.anthropic.com/en/docs/test-and-evaluate/strengthen-guardrails/reduce-prompt-leak?utm_source=chatgpt.com\n\n**6. The search space S**\n\nThe search space S isn’t task-specific. In NAS, it adapts to datasets and evaluation operators. For fair comparison, we use the same search space as prior zero-cost NAS methods. Validating NAS algorithms across different search spaces proves their generalization. Therefore, S can be more generally defined as the search space, as it may change based on the evaluation setup, but we ensure consistency with prior benchmarks.", "claims_and_evidence": "- Although empirical gains and computational benefits are reported, further evidence is needed on how the architectures evolve over time using the LLM reflection module. In particular, more experiments showing the utility of this module in guiding the mutation process would strengthen the claims.  \n- It is also unclear how the initial architecture (F0) is chosen, raising the possibility that a very good prior might skew the results.", "methods_and_evaluation_criteria": "- The methods section is not easy to follow. The description of the mutation process and reflection module is difficult to follow and requires multiple readings. More streamlined and organized presentation would help readers grasp the key details.  \n- Equation 1 seems to have an issue; taking an arg max over architectures does not make sense given that the expectation is over all architectures. \n- Definitions of the Zero-Cost proxies (e.g., GraSP, Synflow, Gradnorm, Zen-NAS, ZiCo) are relegated to the appendix; integrating these into the main text would improve clarity.", "theoretical_claims": "No theoretical claims", "experimental_designs_or_analyses": "- A more detailed qualitative analysis is needed to illustrate how the architectures evolve over time—from the initial seed (F0) to the final designs. For example, visualizations showing the progression from the initial design (F0) through subsequent iterations could provide valuable insights into the diversity of the architectures chosen and effectiveness of the proposed method.\n- More details are required on the reflection module: for instance, the paper should explain how the LLM uses the scores and the architectures to output a reflection, and clarify the role of \"line 15\" in Algorithm 1.  \n- The selection of the initial architecture F0 is not clearly discussed. It is important to know whether F0 is randomly chosen or based on a prior, as a very good initial architecture could bias the results.", "relation_to_broader_scientific_literature": "This work introduces a novel reflective module to guide the mutation process.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths \n- Easy inference-based method using LLMs for NAS leveraging Zero-Cost proxies (e.g., GraSP, Synflow, Gradnorm).  \n\nWeaknesses \n- Poor writing in several sections makes the paper hard to follow. Insufficient detail in the related works section and definitions of key concepts, which are either vague or relegated to the appendix. \n- Lack of qualitative evaluations and clarity on how the reflection module improves mutation.", "other_comments_or_suggestions": "- Related Works section needs more detail. A clearer explanation of Zero-Cost NAS would help, particularly how proxy metrics estimate model performance without full training. The statement \"They mimic natural selection by iteratively selecting, mutating, and recombining candidate architectures based on proxy scores.\" is vague—specify the types of proxies used and their correlation with final performance. Similarly, Section 2.2 is superficial. For example, \"Hao et al. (2024) demonstrates how LLMs serve as surrogate models, significantly reducing computational costs.\" lacks details—briefly explain the mechanism rather than just stating the benefit. Adding these refinements would improve clarity.\n- Some statements are not clearly explained. For example, the paper mentions, \"Some researchers proposed the hypothesis that induction heads might constitute the mechanism for the actual majority of all in-context learning in large transformer models.\" This claim needs further elaboration and context.   \n- “To avoid the issue of prompt leakage, restrictions should be specified in the system prompt of not disclosing any information defined in the example.” Could the authors provide more insights on this? \n- The search space ‘S’ is chosen tailored to a downstream task such as the MobileNet search space. It is not clear whether ‘S’ can be defined more generally.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No"}}, {"review_id": "5kXI0KkRC2", "reviewer": "Reviewer_TuuS", "summary": "This paper proposes Reflective Zero-cost NAS (RZ-NAS), a Large-Language-Model-to-Neural-Architecture-Search (LLM-to-NAS) method that integrates evolutionary search with LLM guidance and zero-cost (zc) proxies. The approach leverages LLM reflection to iteratively refine architecture mutations, improving search efficiency. Empirical results demonstrate that RZ-NAS performs well on both micro and macro search spaces, and can outperform selecting architectures based solely on zc proxy scores.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for recognizing our contributions and detailed feedback. Hoping our responses address your concerns.\n\n**URL** https://anonymous.4open.science/r/rebuttal-ECC4/icml_rebuttal.png\n\n**1. Reflection mechanism**\n\nThank you for this comment. Due to the character limit, detailed responses can refer to the first response to Reviewer oqYW.\n\n**2. ZC Proxy Integration**\nThe system prompt defines the model's task to optimize architectures for higher ZC proxy scores during mutation. The compute ZC proxy module provides a code block and proxy description for model understanding. The user prompt includes a JSON list with the current architecture, proxy type, and pre-mutation score. The LLM is guided by structured prompts, I/O formats, and in-context examples. Reflections from prior outputs further refine the process. To avoid hallucinations, calculations are computed by code not LLM.\n\n**3. Experiments**\n\n**Mutation action space:** The action space is defined in the operators of system prompt.\n\n**number of architectures evaluated:**  We perform 1500 evolution rounds, far fewer than Zico’s 100k iterations, selecting one architecture per iteration.\n\n**Population size:**  For NasBench201 and Cifar10, the population size is 100. For Cifar100, Imagenet and  COCO, the populatioin size is 256.\n\n**Error rate** We compare five zero-cost proxies in Tables 1 and 2. For ImageNet, we follow Zico’s setup with 450M, 600M, and 1000M FLOPs budgets, selecting the best proxies for display.\n\n**Initialization**  We initialize the random population from scratch. \n\n**VALIDATE** This function checks LLM-generated architectures for valid operators, channels, and structure. Invalid architectures are excluded, and exceptions are passed to the reflection module for analysis.\n\n**Query costs**  Each iteration outputs a mutated architecture, taking 0.8-2.0 seconds. With fewer epochs than zero-cost proxies, our method costs about 40 minutes on CIFAR10, roughly comparable to Zico’s 0.03 GPU days.\n\n**Implementation of baselines** We evaluated 1500 architectures which are different from previous zx proxies.\n\n**Search spaces and benchmarks**: Micro: For NASben201, Cifar10, and Cifar100, we used a cell-based search space. Macro:  For the Imagenet task, we used the Mobilenet search space, consistent with previous zero-cost proxy methods (e.g., Zico, Zen_Nas). For object detection, we stack operators to construct the backbone, following the same search space and network construction as Mae-DET.\n\n**Temperature**: The temperature is sampled uniformly from the domain not a fixed value. The comparison with different fixed temperatures is in Table 1 of **URL**.\n\n**4. Reflection literature**\nDue to the character limit, detailed responses can refer to the first response to Reviewer yPSd.\n\n**5. References**\nWe agree that [1] you mentioned provides context for our setup and limitations discussed in [2, 3] are highly relevant. We’ll add them in the revision.\n\n**6. LLM expense**\nOur token is around 2500 tokens. On dataset Cifar10, we spend about $75 for one zc proxy training using GPT4o. This expense is acceptable given GPT-4o’s superior reasoning capabilities Furthermore, more cost-effective open-source models with competitive performance have emerged since our submission. We will integrate them into future work to improve cost-efficiency.\n\n**7. Dependency on robust zc proxies**\nWe did experiements onTransNASBench-101-Mirco which show the robustness of our method in Table 2 of **URL**.\n\n**8. Reliance on pormpt engineering**\nWe used GPT4o to rephrase prompt. We show RZ-NAS’s robustness after rephrasing on Cifar10(in Table 3 of **URL**). Previous research confirms LLMs are insensitive to phrasing changes in structured tasks when examples, I/O formats and specific context remain consistent[4].\n\n[4]Exploring the Limits of Transfer Learning with a Unified Text-to-Text Framework.\n\n**9. Typos and notational issues**\nThanks for pointing out issue. S defines set of operations and A is the set of all possible architectures generated from S. We revise the equation in the third question of oqYW.\n\n**10. Visualize acc and zc scores**\nWe visualize and show our effectiveness on nasbench201 in Figure 1 of **URL**.\n\n**11. Random selection**\nOur random selection differs from traditional evolutionary strategies but is based on two key ideas, showing competitive results in Table 4 of **URL**.\nFirst, we follow selection design in previous zero-cost NAS (e.g., Zennas) which use random selection to boosts search space coverage. Our framework supports multiple zc proxies, aligning with existing methods. Second, random sampling maintains population diversity and avoid premature convergence. LLM-guided mutations steer low-scoring candidates toward better solutions. Unlike traditional algorithms which may lead to inefficiency, LLM mutations intelligently explore architectures, maintaining diversity and accelerating convergence. This aligns with prior LLM-in-NAS works like ELM and EvoPrompting.", "claims_and_evidence": "The claims are mostly supported by the reported results, which show that:  \n1) **zc proxies serve as valid signals for LLM reflection** and offer a much lighter alternative to accuracy-based evaluation, which requires full neural architecture training.  \n2) **LLM guidance improves evolutionary search**, providing meaningful refinements beyond random mutation and simply selecting the highest zc-scoring architecture.  \n\nHowever, the paper has **serious issues regarding clarity and depth of discussion**, making it difficult to fully verify the reported results. This lack of transparency weakens the strength of the claims. I will elaborate on these issues in the later sections.", "methods_and_evaluation_criteria": "The overall method and evaluation criteria are mostly reasonable and align with the goals of improving LLM-to-NAS efficiency with zc proxies. The proposed method is clear from a high-level perspective,  and the evaluation criteria of accuracy and correlation are standard in NAS. However, the clarity issues make the methods still to be abstract and black-box:\n\n- Reflection Mechanism Detail:\nThe paper introduces a reflective module meant to help the LLM improve its mutation process based on feedback (e.g., zc scores and exceptions). However, it only provides a high-level description of this component. For instance, while it's clear that the LLM uses “humanoid reflections” to guide future mutations, the paper doesn’t detail how the reflection is processed, what specific signals trigger a change in strategy, or how this feedback loop is quantitatively integrated into the mutation process.\n\n- ZC Proxy Integration:\nThe framework leverages several zc proxies to evaluate architectures without full training. However, it is unclear how LLM understands the information of these zc proxies based on text and code guides, how LLM considers the existing zc scores to decide the next mutation, and what changes exist when different types of zc proxies are provided to LLM. The whole process is a black box and represented by a black-box `GENERATINGMUTATION` function in Algorithm 1. While it is understandable that interpreting the decision of an LLM can be hard, it is expected to provide at least **some levels of interpretation** of an LLM's decision.\n\nFrom another perspective, the paper even didn't provide **any output** of LLM except the performance of the final selected architectures. Only the input prompts are clearly outlined. **It is expected for authors to provide at least a few interactions with LLMs, including the full conversations, and how LLM responds to the given zc scores or exceptions**. Given that the codes are not provided, it is expected for the authors to provide far more details of the algorithm than the current version.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design suffers from even more severe clarity issues than the methodology. Below is a non-exhaustive list of important details that are expected but missing:  \n\n- The **mutation action space** provided to the LLM.  \n- The **number of architectures evaluated** during the search process.  \n- The **evolutionary population size**.  \n- The **error rate when combining with each zc proxy**, currently only briefly mentioned in the appendix of the ablation studies.  \n- **Initialization details**—does the search start from scratch, or are some architectures collected before the first mutation?  \n- How the **`VALIDATE` function in Algorithm 1** is implemented.  \n- The **query costs of LLM**—are they significantly higher than local zc proxy computations?  \n- The paper states that **five different temperature settings** were tested for the LLM, but only one result is reported—where are the other four?  \n- **Implementation details of the baselines**—how were they run, and were zc proxy results obtained from the same number of evaluated architectures?  \n- **Details on search spaces and benchmarks**—the paper even does not specify which are **micro vs. macro search spaces**.  \n\nGiven that so many critical details are missing, the lack of clarity makes the paper feel **incomplete**.", "relation_to_broader_scientific_literature": "This paper contributes to the broader scientific literature by combining two emerging directions in NAS: **LLM-to-NAS** and **zero-cost NAS**. By integrating these two approaches, RZ-NAS seeks to improve search efficiency while maintaining competitive performance.\n\nHowever, the clarity issues in the paper make it difficult to fully understand its **precise innovation in combining these two fields**. While it is clear that the method involves LLM-guided mutations and zero-cost proxy evaluations, the **lack of details on how these components interact** makes it hard to determine whether RZ-NAS introduces a fundamentally new strategy or primarily refines existing ideas. The missing implementation details further limit the ability to compare this work to prior research effectively.", "essential_references_not_discussed": "To the best of my knowledge, no missing references are \"essential\" to be included in this paper, although more existing zc proxies [1] and the discussion about those zc proxies are often inconsistent and unreliable [2,3] (given that the paper uses it to replace the accuracy) can be considered to include.\n\n[1] NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies. NeurIPS Datasets and Benchmarks Track 2022.\n[2] A Deeper Look at Zero-Cost Proxies for Lightweight NAS. ICLR Blog Track 2022.\n[3] Robustifying and Boosting Training-Free Neural Architecture Search. ICLR 2024.", "other_strengths_and_weaknesses": "## Strengths\n- While the idea of the combination of LLM and zc proxies is simple and intuitive, this paper is the first to provide a concrete implementation.  \n- The explanation of the input prompt design is detailed.\n- Some ablation studies are provided to help understand the effects of different components.  \n\n ## Weaknesses\n- Severe Clarity Issues. As explained above.\n- Lack of In-Depth Analysis of LLM Behavior.  As explained above.\n- Dependency on LLM Quality and Ignore LLM Costs. Table 6 shows that the performance of RZ-NAS heavily depends on the quality of LLMs, which can introduce high query costs. This may raise the question of whether this is worthwhile in terms of the cost-performance tradeoff, or would it be more economical to simply let zc proxies search for more architectures? The paper does not account for LLM inference costs, making comparisons with pure zc-proxy baselines unfair.\n- Dependency on Robust zc Proxies. The paper implicitly assumes the zc proxies are always safe and robust to be used as reward signals to LLM,  but this is not always true (see Essential References Not Discussed for reference). Authors are expected to conduct experiments on datasets on benchmarks where zc proxies generally perform poorly (e.g., TransNAS-Bench-101-micro, TransNAS-Bench-101-macro).\n- Reliance on Correct Prompt Engineering. The results hinge on carefully designed prompts (system instructions, reflection steps, code examples). Poorly formed prompts could yield LLM hallucinations or invalid code. The paper provides some template details, but real-world usage might require ongoing prompt tuning. If the authors can demonstrate that RZ-NAS is not highly sensitive to prompt wording, I am willing to remove this concern.", "other_comments_or_suggestions": "I would suggest the authors significantly improve the clarity and fill in the necessary details to make RZ-NAS **more complete**.\n\nTo better understand the **contribution of LLM during the search process** (and also, the performance over different search budgets), rather than just its effect on the final selected architecture, the authors may consider **visualizing the accuracy and zc scores of each searched architecture over time**. A comparison with purely zc proxy-based baselines would help assess how LLM-guided mutations impact search dynamics. This could be effectively presented as a **line graph on NAS-Bench-201**, where ground-truth accuracies are available.\n\n### Non-Exhaustive List of Typos and Notational Issues\n- Section 3.1 Problem Formulation:\n  - The search space is first defined as  $S$, but later, architectures are denoted as  $a^* \\in A$, leading to inconsistency.\n  - The zc score is initially defined as $ O(a_k)$, but in Equation (1), the function is written with four parameters, which are not explained.\n\n- Algorithm 1:\n  - The maximal depth $L$ is defined but never used.\n  - The evolutionary population size is first denoted as $N$ but appears as $E$ on line 9.\n  - On line 15, $z$ is undefined—does this refer to the zc score obtained on line 7?\n\nThe authors are encouraged to carefully revise their manuscript to improve consistency and readability before submission.", "questions_for_authors": "I understand that many clarity issues have already been raised in the above sections. However, I have one additional question regarding the decision rationale in Algorithm 1.\n\nIn **line 3**, the architecture selected for mutation is **randomly chosen** from the population, rather than selecting the existing highest zc score architecture. In most evolutionary algorithms for NAS, selecting the architecture with the highest proxy score is a more common strategy, as it prioritizes refining the most promising candidates.\n\nCould the authors clarify the rationale behind this design choice? Additionally, what would be the performance impact if **line 3 were modified to always select the architecture with the highest zc score** instead of choosing randomly?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kzadTYb50D", "reviewer": "Reviewer_yPSd", "summary": "This paper rethinks the role of LLMs in NAS. It doesn't directly apply LLM to NAS architecture optimization but optimize the process of random population generation utilizing the code- and text-level understanding of LLM. The novelty lies in the proposed reflective strategy that achieves architecture generation with humanoid reflections and training-free metrics to address some key challenges in current LLM-to-NAS works and achieves good results on multiple search spaces and benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the positive feedbacks and your recognition of our contributions! We will respond to your valuable comments and detailed suggestions!\n\n**1. More literature review on current reflection techniques**\n\nIn the revised version, we will include a more complete literature review of existing reflection research, along with expanded theoretical insights in Section 2 (\"Related Work\").\n\nThe self-reflection mechanism in large language models (LLMs) refers to the model's ability to review its own outputs or reasoning process to identify and correct errors. *Reflexion*[1] first proposed a framework to strengthen LLM agents through language feedback. It introduced a \"Generate-Evaluate-Reflect\" loop, where an Actor generates a trajectory, an Evaluator computes rewards, and Self-Reflection provides natural language feedback to guide subsequent decisions. Reflexion outperformed the previous GPT-4 performance across multiple coding benchmark tests. In the field of evolution, *ReEvo*[2] combined evolutionary computation with human-like reflection to generate heuristic algorithms. Moreover, the reflection mechanism greatly improved model performance on tasks like multiple-choice reasoning[3].\n\n**2. Detail the LLM-based reflection module**\n\nThank you for this comment. Reviewer oqYW pointed out the similar question. Due to the character limit, detailed responses include the reflection mechanism and the input/output of the LLM reflection module, can refer to the first response to Reviewer oqYW.\n\n**3. Compare the reflection module of RZ-NAS to previous work**\n\nUnlike previous work (e.g., *Reflexion*) that used feedback mechanisms to improve closed tasks like code generation and question answering, RZ-NAS deals with a more open-ended search task. **Thus, we are the first to apply the reflection mechanism to NAS-related work, creating two types of reflection**: one is the internal reflection mechanism in LLM's system prompt, which guides the LLM to discover potentially better solutions via the iteration process; the second is the external reflection module, providing reflection suggestions for each mutation.\n\n**4. More discussion on the theoretical analysis of the reflection module**\n\nIn terms of theory, while the theoretical support for the reflection mechanism effectiveness remains somewhat of a black box, there are related directions that can provide insight into it. For example, in the context of LLMs, the combination of reinforcement learning with human feedback (RLHF) allows the model to adjust based on feedback during the generation of outputs, thus simulating a process of reflection and improvement[4]. Our feedback model is similar to simulate human-like reflection, providing mutation evaluations of the model's behavior during the iterative process, which can correct potential errors in the generated architecture and accelerate the learning of architecture generation strategy through reflection.\n\n**5. Applied the method to other NAS methods beyond image-related tasks?**\n\nYes. We further conducted experiments based on Graph NAS. Since the zc proxy is designed for image-based evaluation, in Graph NAS scenario, we can still use our proposed LLM-in-NAS framework with the evaluation metric changed to node classification accuracy. The evoluationary algorithm(EA)-based Graph NAS method combined with the LLM-in-NAS framework can get superior results on widely-used graph datasets.\n\n| Method               | Search Strategy | Cora            | CiteSeer        | PubMed          | CS               | Physics          |\n| -------------------- | --------------- | --------------- | --------------- | --------------- | ---------------- | ---------------- |\n| GCN                  | Manual          | 83.1 ± 0.40     | 70.3 ± 0.70     | 79.0 ± 0.50     | 80.8 ± 0.10      | 96.38 ± 0.07     |\n| GAT                  | Manual          | 83.0 ± 0.70     | 72.8 ± 0.70     | 79.2 ± 0.30     | 81.3 ± 0.30      | 96.37 ± 0.20     |\n| GraphNAS             | RL              | 84.2 ± 1.00     | 73.1 ± 0.90     | 79.6 ± 0.40     | 92.54 ± 0.09     | 93.87 ± 0.06     |\n| AutoGraph            | EA              | 83.8 ± 0.50     | 74.4 ± 0.60     | 79.2 ± 0.30     | 92.54 ± 0.09     | 94.02 ± 0.12     |\n| **Ours (AutoGraph)** | **EA**          | **84.8 ± 0.20** | **75.0 ± 0.40** | **80.6 ± 0.28** | **93.14 ± 0.19** | **94.62 ± 0.10** |\n| DFG-NAS              | EA              | 85.2 ± 0.20     | 75.3 ± 0.30     | 81.1 ± 0.30     | 93.44 ± 0.06     | 94.50 ± 0.21     |\n| **Ours (DFG-NAS)**   | **EA**          | **85.8 ± 0.30** | **75.7 ± 0.70** | **83.1 ± 0.30** | **94.14 ± 0.06** | **94.95 ± 0.19** |\n\n[1] Reflexion: Language Agents with Verbal Reinforcement Learning\n\n[2] ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution\n\n[3] Self-Reflection in LLM Agents: Effects on Problem-Solving Performance\n\n[4] Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflection", "claims_and_evidence": "The paper claims the reflective module in LLM-to-NAS searching methods, and gives empirical evidence to illustrate the superiority of the reflective module and its generalizability on multiple tasks. It compares the efficiency of the search process with and without the reflective module, showing that the reflective module leads to a significant reduction in the computational resources required to find optimal neural architectures.", "methods_and_evaluation_criteria": "Overall, the evaluation approach is solid, which can be seen from the following aspects.\n1.  Comparing the method against established NAS proxies on relevant benchmarks like NAS-Bench-201, CIFAR-10, CIFAR-100, and ImageNet.\n2.  Comprehensive ablation experiments.\n3. Inclusion of diverse tasks, such as image classification and object detection.\n4. The use of GPT4-o for mutation generation and temperature sampling ensures diverse output. \n5.  Consistency with prior work, such as ZiCo and Zen NAS, ensures fair comparison.", "theoretical_claims": "Yes. The paper primarily focuses on empirical results and provides insights through experimental evaluations. While there are no explicit theoretical claims for assessment, it would be beneficial to explain more about the theoretical foundations of the proposed method.", "experimental_designs_or_analyses": "Yes. The paper displays the results on image classification and detection tasks. The author conducted experiments base on widely adopted NAS benchmarks (CIFAR-10, CIFAR-100, and ImageNet) and search spaces (DARTS, NAS-Bench-201, MobileNet) and compared with existing NAS and NAS-to-LLM methods.", "relation_to_broader_scientific_literature": "This paper builds on existing research in Zero-Cost NAS and LLMs, offering a novel integration of both to optimize the architecture search process. The authors tried to address existing challenges in current LLM-to-NAS methods, such as slow search efficiency and underutilization of LLM capabilities. By leveraging evolutionary algorithms and Zero-Cost proxies, the paper provides a more comprehensive and effective framework for diverse NAS tasks. The approach combines the strengths of Zero-Cost proxies with LLMs' text and code-level understanding, improving the efficiency of architecture exploration. The paper would benefit from a more extensive literature review, particularly on the current reflection techniques from experimental and theoretical aspects, as the reflection technique has been developed in prompt engineering and code generation [1,2].\n[1] ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution, NeurIPS 2024.\n[2] Large Language Model-Aided Evolutionary Search for Constrained Multiobjective Optimization.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "S1. Make a novel structured, general framework for applying LLM to NAS tasks. The integration of LLMs with Zero-Cost NAS proxies addresses several challenges in traditional NAS and LLM-to-NAS methods. The innovative combination of these two techniques creates a more efficient framework for architecture search.\nS2. The author introduces a comprehensive framework that combines LLMs’ text and code-level understanding. The paper compares this method with past LLM-to-NAS and traditional NAS approaches from multiple perspectives, demonstrating its versatility and effectiveness.\nS3. The experiments are extensive and in detail. The authors test the effectiveness of RZ-NAS across multiple NAS tasks and ablate the importance of using the reflection modules and necessity of other components.\n\nW1. The paper utilizes the zero-cost NAS strategy which may limit to image classification or detection tasks. For other downstream tasks like graph node classification task that without zero-cost proxy computation, does the proposed LLM-to-NAS strategy work without training-free computation? \nW2. It is suggested to explain the reflection module in LLM-based strategies in more detail. Plus, the reflection module in previous research (e.g. evolutionary algorithm, NAS-related work) could be expanded.", "other_comments_or_suggestions": "I would appreciate the authors' response on my main concerns listed above.", "questions_for_authors": "1. Could the proposed method be applied to other NAS methods beyond image-related tasks?\n2. As mentioned in the Relation To Broader Scientific Literature, the authors should provide more explanations of current reflection modules. Reflection module appears to be used in language model development. What is the primary distinction compared to RZ-NAS? Further analysis and discussion on this would be valuable.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l7XjU1m6cN", "reviewer": "Reviewer_wR18", "summary": "This paper studies LLMs in NAS from a novel perspective. It begins by addressing the challenges in existing LLM-in-NAS strategies and explores the potential of combining LLM’s code- and text-level comprehension with zero-cost computation proxies. The proposed approach introduces the reflective module within LLM-based zero-cost NAS to improve the process of generating random populations. This method allows for architecture generation using humanoid reflections and eliminates training-related time costs. Extensive experimental results show the effectiveness across multiple search spaces and benchmarks.\n\n## update after rebuttal\nThe authors have addressed my concerns, and l would like to retain my positive score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and constructive comments! We truly value your kind words about our contributions. We hope our responses address your concerns and enhance your view of our work! If there are any additional comments or questions, you can post them in the next stage.\n\n**1. More theoretical explanations or literature reviews on current reflection modules**\n\nThank you for your insightful suggestions. Reviewer yPSd pointed out the similar question. Due to the character limit, detailed responses include literature reviews and theoretical explanations of the LLM reflection module, can refer to the 1st and 4th responses to Reviewer yPSd.\n\n**2. Discussion the case without Zero-Cost proxies**\n\nThe reason why we use zero-cost proxies is due of the heavy time cost of NAS search, which requires full training cycles to evaluate candidate architectures. Actually, our proposed LLM-in-NAS framework is designed to be flexible, which can also use validation accuracy (instead of proxies).\n\n**3. Why not to directly modify the code of the architecture guided by LLM**\n\nThe main reasons why not to directly modify the architecture code are two-fold.\n\n***Complex Search Spaces***\nPrevious works directly modify architecture code with LLM in simple search spaces  (e.g., EvoPrompting’s chain CNNs on MNIST) , while our task is based on standard NAS search space (e.g. DARTS , NASBench),  involving multi-module combinations and cross-file dependencies (e.g. the coupling between the Genotype and Network classes). \n\n***Genotype Representation***\nFor both the macro and micro search spaces, we use genotype string instead. This align with previous NAS frameworks like DARTS and ENAS that also use genotypes. Genotype ensures that the network code remains unchanged, avoiding structural bias from LLM-generated code. Additionally, we can limit LLM errors to fixable issues and filter out invalid structures via regex-validated syntax.\n\n**4. How to ensure generated architectures are valid and functional**\n\nWe provide the input and output of the architecture in JSON format in the in-context examples. Both of our search spaces (micro and macro) use formatted input and output. Previous LLM-related work has shown that enforcing formatted input and output in in-context examples significantly improves the results [1]. Additionally, since the candidate operators in the search space are fixed, the LLM is constrained to selecting operators that exist within the search space during the mutation process. This is specifically explained in the `Define Search Space`section of system prompt in the Appendix—where it states, \"I should generate an optimal architecture by mutating an operation from this file.\"*—as all operators are defined in the* operator.py *file*.\n\n[1] Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements\n\n**5. Any issues where the generated architectures introduce bugs?**\n\nYes, we do encounter issues where the LLM’s output has problems. For example, sometimes the output may have missing operators, causing the architecture to be invalid in the validation module. However, such cases are very rare. In about 100 iterations, we typically encounter around 2 such architectures. Since we impose strict input and output format constraints, the LLM is able to understand the architecture's format. When an architecture is invalid, the exception for that iteration is passed to the reflection module as a parameter, and the architecture is not added to the population. In the next iteration, we randomly select a new architecture for mutation, reflect on the error from the previous invalid architecture, and guide a better mutation.\n\n**6. Clarify the process used to restrict the generation of architectures**\n\nAfter the LLM generates an architecture, we connect it to a validation module that checks the architecture. This module verifies if the operators belong to the search space, checks the input and output channels. For he micro search space, it checks the number of cells and edges. For the macro search space, it checks the number of layers. If the architecture is invalid, it is not added to the population. The validation module then outputs an exception, which is passed to the reflection module. The reflection module analyzes the error and provides a reflection result. A random architecture is then selected from the population for the next iteration, starting the mutation process guided by the LLM.\n\n**7. The NAS-Bench-201 accuracy results in the table are unclear.**\n\nSorry for the confusion. We have rechecked the data and confirmed that the results in bold indicate the best method compared to the existing LLM-based and zero-cost methods. The revised version will present the accuracy results clearly and include an explanation in the footnote. We have also revalidated all experimental data and confirmed the consistency of the results.", "claims_and_evidence": "The author focuses on LLM-based zero-cost search methods and presents experimental comparisons to highlight their superiority over traditional NAS and LLM-in-NAS methods. It also explores various zero-cost proxies, demonstrating the generalizability of the approach across multiple benchmarks. Additionally, the paper performs an ablation study, confirming the effectiveness of each component.", "methods_and_evaluation_criteria": "The analysis of the issue is thorough, and the explanation of each step of the proposed method is detailed and well-reasoned. In Section 3, the author illustrated the interaction pipeline, the construction of the framework as well as each module of prompt, with a detailed illustration of how each module in the algorithm is built. The evaluation criteria are robust. In Section 4, the author conducted extensive experiments and provided in-depth analysis, comparing the proposed method not only with traditional NAS methods but also with LLM-in-NAS approaches.", "theoretical_claims": "Yes, the author provides theoretical support in Sections 2 and 3.", "experimental_designs_or_analyses": "Yes. The author compares different existing methods, including traditional NAS, training-free NAS, and LLM-in-NAS methods, across multiple tasks (e.g., image classification and object detection) on different benchmarks like CIFAR-10, CIFAR-100, and ImageNet. Additionally, it conducts ablation experiments to demonstrate the effectiveness of different modules within the framework.", "relation_to_broader_scientific_literature": "This paper introduces related work in Zero-Cost NAS and LLMs, comparing the most relevant previous studies across key aspects. In the literature review, the authors provide a comprehensive analysis of previous works on LLM-in-NAS, evolutionary NAS algorithms, LLM-in-evolutionary methods, and zero-cost proxies. While previous studies mainly focused on enhancing NAS performance through LLM’s text-level or code-level understanding, this paper tempts to overcome these limitations by exploiting the LLM’s understanding capabilities. By comparing factors such as search efficiency and the role of LLMs in NAS, the authors address the existing challenges in current LLM-to-NAS methods.", "essential_references_not_discussed": "Key related works are all discussed.", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper is thoughtfully structured and presents a clear narrative. It begins by establishing the background and outlining the problem being addressed. The writing is direct, making it easy to follow the progression of the work.\n\nThe innovation of the proposed method and its difference from others is clearly shown in Table 1.\n\nThe methodology is detailed, offering in-depth explanations of the RZ-NAS framework and its various components. This allows for a clear understanding of how the framework operates and its individual elements.\n\nThe author highlights the significant improvements brought by combining LLMs with Zero-Cost NAS proxies and shows the broad applicability of the proposed framework. It demonstrating the framework’s flexibility and its potential for generalization across various Zero-Cost strategies.\n\nWeaknesses:\n\nThe authors should provide more theoretical explanations or literature reviews on current reflection modules. How are these modules utilized in other LLM-driven algorithms? What sets them apart from the reflection mechanism in RZ-NAS? Additionally, it would be helpful to explain why the reflection module works from a theoretical or mechanistic perspective.\n\nWhile the paper provides ablation analysis to highlight the importance of mutation parts using LLMs, the strategy of Zero-Cost NAS also plays a critical role in the proposed method. The authors could enhance the paper by providing additional performance analysis without Zero-Cost proxies, along with a discussion on potential directions for further improving the method’s performance.\n\nRZ-NAS is based on the framework of evolutionary algorithms.  As LLM-driven GE [1], and other similar methods, they utilize LLM to directly understand and modify evolutionary code. Why did you choose not to directly modify the code of the architecture guided by LLM? It would be better to give more explanations for better understanding the contribution of RZ-NAS.\n\n[1] LLM Guided Evolution - The Automation of Models Advancing Models, GECCO '24", "other_comments_or_suggestions": "see Strengths And Weaknesses", "questions_for_authors": "How do you ensure that the generated architecture can be converted into valid and qualified architectures? Have you encountered any issues where the generated architectures introduce bugs? Could you clarify the process used to restrict the generation of architectures to ensure that they are compilable and meet the necessary criteria?\n\nThe result in the table of the test and valid accuracy on NAS-Bench-201 appear a bit unclear. As the results in bold indicate the best compared to the existing LLM-based and zero-cost methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: A recent line of works considers using large language models to perform neural architecture search (NAS). Like many NAS approaches, these can be expensive; in addition, the LLM-based approach can be inflexible as well. The authors set out to minimize these costs. The idea is to build on zero-cost proxies, a tool from traditional NAS that lowers the cost of evaluating a strategy. A second contribution is to use an LLM-guided feedback mechanism for improvement. This approach performs well on a number of settings. 

The reviewers brought up several areas for improvement, particularly on the clarity of the writing. The authors have sketched out these changes and satisfied everyone’s concerns. 

This is a nice paper overall; it has a mixture of good ideas with good technical execution. For this reason, I believe it is worth accepting.
Decision: Accept (poster)